Moving to Google+

See my Google+ profile. New stuff will be posted there from now on.

As to why, it's just easier to manage. No worries about updating software and making the site work with new browser technologies and mobile is someone elses job. :)

So I tried out facebook ads

One of the marketing platforms for Chili Charger we decided to use was facebook. Since I have never used it, my account is pretty empty. Turns out that is a major problem. But not so fast.

I created a brand page, uploaded an app icon, header and URLs to the play store. Created an ad account, added payment information... BooM! To protect me from them evil hackers my account was locked. Safety precaution. Of course.

But fear not, follow these easy instructions to regain access to your account. So I uploaded a picture of me as requested. That was on a Friday.

Saturday: no answer, account still being verified.

Sunday: no answer, investigation is ongoing.

Monday! I'll leave this one blank.

Tuesday: no answer, but if I now log in my submitted information does not need to verified anymore. Instead the account has been disabled. WTF? Still no answer whatsoever. Only remaining method now is to submit an appeal.

Wednesday is here. No answer, of course not. But when I randomly try to login, it suddenly works! Not a single answer via mail, nothing in the support inbox. No further information on why the whole account was disabled.

I have never, ever been treated worse by any other customer service. I mean, I was about to become a paying customer using their ad system. But apparently they don't need any more customers.

Bottom line if one wants to use facebook ads: create and account and wait half a year. During that time add random friends, but no more than 2 a week! Post a cat picture every Friday, write a blog about every rainy Sunday and add totally random musings every Tuesday. Maybe after that you will evade the automatic account disabling that probably consists of a chain of half a million conditions and nobody has a clue anymore how the code works.

Snø and Cyklus

Our first two Android apps are now on the app store. Interesting experience to develop something from zero to app store without anyone knowing everything better. And it's so much easier to know everything better if you are not involved in the whole thing!

Snø is a weather app with pictures from all around the world. It's probably a bit hard to market, because it fits in neither category. Not a full blown weather app and not a full blown social media picture sharing thingy. Of course it also shows the lunar phases! Apparently everything I do includes that functionality.

And Kianga made Cyklus, a 24-hour watch face with weather forecast. It shares the weather data backend with Snø. Will be interesting to see which one will generate more queries... and costs us more money. Probably Snø in the long run since it's free.

So download it and make the cat happy!

Hetzner root server full disk encryption with cryptsetup LUKS

Updated 29 Dec 2015

This howto turns a Hetzner default Debian root server installation into a fully encrypted server with LUKS that can be unlocked remotely with the dropbear ssh server. If the server is rebooted, seized or a KVM switch is plugged into the wrong server, the data cannot be accessed. Likewise, your data will be lost forever* if you forget the passphrase. This works with Debian Jessie and Wheezy images. But read the note on Jessie below.

(* a long time)

Please note the date of this post. If it is older than a year, the defaults of the Hetzner install image might have changed. Paths and devices might be named differently. I assume the following (Hetzner default) disk layout. If you have configured your server differently, change the device names accordingly.

md0: Raid1 consisting of sda1, sdb1 as swap md1: Raid1 consisting of sda2, sdb2 as /boot md2: Raid1 consisting of sda3, sdb3 as the root filesystem

The server will not be reinstalled from scratch. All data is preserved (read below). You can encrypt an existing server without any config changes to the services it provides.

Debian Jessie

If you're using Jessie and onward, the init system will be systemd by default. Because of an unresolved bug with crypto luks + systemd + certain crypttab options, including keyscript, you'll have two options with this constellation. There are, of course, other more hackish approaches, but it is up to you to evaluate their safety.

Without systemd

Install the package sysvinit-core before you reboot. This removes systemd-sysv, which makes the system default to systemd as PID 1. Your system will continue to use the old sysvinit. Nothing else will change. Everything that worked in Wheezy will continue to do so.

With systemd

First copy /sbin/init to /sbin/init.sysv This will give an easy way back to sysvinit if thins go haywire. Install systemd-sysv to switch the default to systemd. Add a kernel parameter luks=no to the kernel command line in /etc/default/grub and run update-grub && update-initramfs -u. This will now tell systemd not to unlock the disk at boot, the old scripts in initramfs will take care of that.

There is one important note however: no other disk will be unlocked at boot time, only root. So if you have any other entry in crypttab, it will not be available to auto-mounting at boot and systemd will fail to bring up that mount point. That includes the encrypted swap further down in this guide. You will have to mount the rest of the paritions manually after each boot. If it's just swap, you could also remove it completely, 32GB or even 64GB in the current servers should be enough for almost every workload.

If the system now fails to boot, wait at least 1m30s, that's how long systemd tried to mount the missing devices until it gave up on my test system, pass the following kernel parameter on boot: init=/sbin/init.sysv. You can either do that with a LARA (KVM) console or edit your config in the rescue system. That will switch to sysvinit for this one boot.

Additional packages that must be present on your current system

Install these in your current installation before you proceed.

apt-get install cryptsetup apt-get install busybox dropbear

You should have cryptsetup installed before you install dropbear. There might be some strange side effects with the initramfs config otherwise. I have received such a report. At least it doesn't hurt.


There is a complete description on how to generate the keys and unlock the disk in the cryptsetup documentation. It is installed in /usr/share/doc/cryptsetup/README.remote.gz. Configure dropbear following the readme before you proceed.

If you have configured it and had the private/public SSH keys generated automatically, do not forget to copy the private key or install your public key according to the readme.

Reboot into rescue system

Create a backup if you're working with important data!

Log into the robot and reboot the server into the (64bit) rescue system. At the time of this writing every tool was available in the rescue system.

Save your old system

The old system is moved aside while the encrypted disk is prepared. If you have more data than what fits onto the root filesystem, you need to move large data directories either to the backup FTP or restore them from a backup afterwards. In the rescue system you will have a root filesystem ramdisk which is half the size of the server's memory. So if you have a 32GB root server you will have ~16GB available.

mkdir /oldroot mount /dev/md2 /mnt rsync -a /mnt/ /oldroot/ umount /mnt

Your system is now mirrored into /oldroot.

Create the encrypted disk

Let's create the encrypted filesystem. This will destroy all data on your server! You might want to use a diceware password so you don't have to enter any special characters while booting the system. Example on xkcd. But don't use an online generator or the horse staple. :) Or for example: dd if=/dev/random bs=1 count=18 | openssl base64

The following command will create a LUKS device using the aes-xts-plain64 cipher with 512 bits key size and a number of key iterations that will take roughly 5 seconds to compute on your hardware. Meaning it will take 5 seconds to unlock the disk. Since servers are rarely rebooted, it does not hurt.

cryptsetup --cipher aes-xts-plain64 -s 512 --iter-time 5000 luksFormat /dev/md2

You can verify that everything went fine with the luksDump command. It should produce something like the following output:

# cryptsetup luksDump /dev/md2 LUKS header information for /dev/md2 Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 512 MK digest: de ad be ef MK salt: de ad be ef de ea be ef MK iterations: 83750 UUID: 5f76b99e-0418-4fb5-a731-201004ede948 Key Slot 0: ENABLED Iterations: 336005 Salt: de ea be ef de ea be ef Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED

Once the LUKS device is created, activate it. This will ask for the passphrase you've just used.

cryptsetup luksOpen /dev/md2 root

The following step is optional but recommended. It fills the whole disk with random data and overwrites anything on the partition that may be recoverable. Even though it reads from /dev/zero, you are writing to the high level LUKS device which writes random encrypted noise to the disk. This might take a while. 2-4 hours with the current disk sizes. You can skip this step and create a large file on the filesystem after everything is finished with the same effect.

dd if=/dev/zero of=/dev/mapper/root bs=1M

Create a filesystem on the encrypted device. I will use ext4 with default options and set check interval to 6 months, error behavior to remount-ro and max mount count to 50.

mke2fs -t ext4 /dev/mapper/root tune2fs -i 6m -e remount-ro -c 50 /dev/mapper/root

You can now restore your data.

Reinstall the system

mkdir /newroot mount /dev/mapper/root /newroot rsync -a /oldroot/ /newroot/

Before we can changeroot into the new system we need some special file systems and the boot file system inside. These are needed so grub and initramfs reinstallation work.

mount /dev/md1 /newroot/boot mount --bind /dev /newroot/dev mount --bind /sys /newroot/sys mount --bind /proc /newroot/proc chroot /newroot

The fstab needs to be changed to reflect the new root file system. Edit the file and replace / (root) with the following:

/etc/fstab: /dev/mapper/root / ext4 defaults 0 2

To be able to boot from the encrypted file system we need a crypttab.

/etc/crypttab: # <target name> <source device> <key file> <options> root /dev/md2 none luks

Now all is set. We need to tell grub the new root device and regenerate the initramfs to include the encryption stuff. Run the following commands. They should automatically pick up /dev/mapper/root as the new root device.

update-initramfs -u update-grub grub-install /dev/sda grub-install /dev/sdb


IPv6 will not be running after a reboot. The IPv4 networking will be brought up in the preboot ssh server. Once Debian boots the real system, configuring the network interfaces will fail, because eth0 was already configured with ip earlier and the network will be in a somewhat inconsistent state. I fixed this be forcing the network down and up at the end of the boot sequence. Add the following two lines to your /etc/rc.local:

/sbin/ifdown --force eth0 /sbin/ifup --force eth0


You can now leave the chroot, reboot the server and hope for the best. :)

logout umount /newroot/boot umount /newroot/proc umount /newroot/sys umount /newroot/dev umount /newroot sync reboot

The system will now boot into the dropbear ssh server which you can connect to and enter the passphrase to unlock the disk. The system will then continue to boot. Once the server answers pings again, ssh into it. To enter the passphrase use the following command:

echo -n "YOUR SUPER SECRET PASSPHRASE" >/lib/cryptsetup/passfifo

The system will now disconnect your ssh. If not you may have used the wrong passphrase. My terminal was kind of broken afterwards and I had to restart the shell. You now should be able to connect normally.

If you have skipped the optional step (writing the disk full) you should do it now. The easiest way is to create a large file with dd.

dd if=/dev/zero of=foofile bs=1M

You should stop any databases or other software that does not like "No space left on device" before writing the disk full. <Insert Mysql bashing here>

Encrypted swap

The swap can also be encrypted with a random key at boot time. This works automagically. Add the following line into your /etc/crypttab. This sets up an encrypted swap device with a random key as /dev/mapper/swap everytime the system boots.

swap /dev/md0 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

And change the swap entry in your fstab into the following:

/dev/mapper/swap none swap sw 0 0


Don't panic. After all you have backups!

If you cannot log in via dropbear at all, because you have forgotten to copy the generated private key to your system or did not put your public key onto the server, it can be fixed easily. Just boot back into the rescue system. You can edit the config and regenerate the initramfs with the same commands as used above.

If login does work, first check the passphrase. You never know, you might just have a typo. If it still does not work, reboot the system into the rescue mode. Try unlocking the disk manually.

cryptsetup luksOpen /dev/md2 root

If that does not work you might have mistyped it while creating the LUKS device. Either try some common typos or you will have to start over.

If you can successfully unlock the disk, mount it and check if you have a problem with the dropbear configuration. See the readme mentioned in the section "Preparation" for any errors you might have in your configuration.

What has changed now?

The encryption is completely transparent. The only user-visible things that have changed are the following devices:

/dev/md0 is now /dev/mapper/swap /dev/md2 is now /dev/mapper/root

Everything else, including disaster recovery from failed hard drives has not changed. The disks are still part of the underlying MD device and do not know anything about the encryption. The boot partition is still unencrypted.

How secure is it?

The encryption itself should be bullet proof as long as the passphrase is complex enough of course. But even if it is "i hate passwords" you should be fairly safe since we used 5 seconds worth of iterations. On Hetzner's EX40 with an i7-4770 it results in 1.5 million iterations and that is (was? see post date!) a very fast CPU.

The boot partition is unencrypted. If someone gained access to your robot account and boots the server into the rescue system he can, of course, modify the initramfs and install software that will compromise your system if you login. The robot logs these reboot requests so if you didn't request it, be very wary and do not log in. The encrypted data is most likely not compromised or modified. We used aes-xts-plain64 which does not suffer from an issue like the default cbc-essiv mode. See for example the description of the malleability attack.

You could probably make a copy of your boot partition which you can restore in case of an unexplained reboot or checksums of the files which you can compare while in rescue mode.

If you need more security than that, only systems with TPM can provide a secure boot path that prevents tampering.


If you want, you can use the checklist below while you set up the server. All important steps are recorded, so you're unlikely to miss one.

Linux USB + Roccat Arvo keyboard = panic

Boom. I like boom… mostly. This must be the first, genuine, reproducible kernel panic I have had in a very long time! Some days ago I bought a Roccat Arvo gaming keyboard after my Logitech illuminated keyboard decided that it doesn't need the 'G' key anymore. Apparently a tiny plastic part broke that held the key in place. On ebay some sellers, more like scammers, sell "three key caps" for this exakt model. Turns out you get three random key caps. Hm, 105 keys, one broke, I get three random replacements. Do the math yourself. :)

Anyway, connecting the Arvo produced the following on my laptop:

input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.0/0003:1E7D:30D4.000 2/input/input23 arvo 0003:1E7D:30D4.0002: input,hidraw1: USB HID v11.10 Keyboard [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input0 arvo 0003:1E7D:30D4.0003: usb_submit_urb(ctrl) failed: -1 arvo 0003:1E7D:30D4.0003: timeout initializing reports input: ROCCAT ROCCAT Arvo as /devices/pci0000:00/0000:00:12.0/usb3/3-2/3-2:1.1/0003:1E7D:30D4.0003/input/input24 arvo 0003:1E7D:30D4.0003: input,hidraw2: USB HID v11.10 Device [ROCCAT ROCCAT Arvo] on usb-0000:00:12.0-2/input1 arvo 0003:1E7D:30D4.0003: couldn't init struct arvo_device arvo 0003:1E7D:30D4.0003: couldn't install keyboard arvo: probe of 0003:1E7D:30D4.0003 failed with error -110

Seems some combinations of USB controllers and hid devices are bugged. Using the mode switch for the keyboard's numpad and then pressing some keys on the numpad result in an instant kernel panic in io_watchdog_func() [ohci_hcd].

On another system it works perfectly fine, without the USB error messages if it is plugged-in. At least it seems like the underlying bug is already known:,

The solution!

What I tried is absurd and stupidly simple. Put an USB hub between computer and keyboard. All problems gone! At least until the bug is fixed which seems to be affecting a wide range of kernels. Happened with Debian's 3.16 - 4.0.5.

The real solution

This bug seems to have been fixed in kernel 4.1.4. USB: OHCI: Fix race between ED unlink and URB submission, commit 7d8021c967648accd1b78e5e1ddaad655cd2c61f upstream. Of course even Sid has 4.1.3 at the moment.


When you think you know all… most… well, a lot of Mysql's strange ways to fuck up your day. There is always something new to learn.

A regular SQL dump that has been generated with mysqldump on a totally ordinary Linux system. And when mysqldump just works on a database of several gigabytes something must be strange.

# mysql --one-database DB < db.sql ERROR 1054 (42S22) at line 1199: Unknown column 'NUDL' in 'field list'

What the noodle? Inspecting the SQL dump reveals the following INSERT:


Apparently the value should be NULL, but somehow mysqldump managed to mangle that into NUDL.

The Internet is for por... spam!

It took less than a week, if not only three days, for spammers to discover the new wolfpaper greeting cards I added to the archive. One day I noticed spam being posted and after a couple of days another one with very meaningful URLs like (not an actual URL, I hope) and completely invalid email. I added a protection from the pattern that was used.

And then there was silence. 

A legitimate card was created some time later so I was reminded that everything actually works. Out of curiosity I checked the server's log to see how many junk I had (not) missed. My guess was one maybe two every other day. Well, not quite. 

In the last week's access log there were about one hundred post actions. Every access from the spam script (it's too stupid to be human but you never know!) consists of two posts. So about 50 attempts in one week. That may not sound very much, but at the time this started the feature was live merely a week and nobody mentioned it on any site, board or weblog. 

I would really like to know how the spammers, or better "spam script" as I still pretend they cannot be human, discovered it so quickly. 

File formats that do not rock: CS...X!?

Comma separated XML – or also known as “parse this!”

<data> funny;column;description;goes into this field oh;and;here;is;some;other; column description dataset;1;2;3 <moredata> some id;123;43;653;314;fubar some id;325;31;434;143;blah some id;343;fu;---;;bar </moredata> <evenmoredata> interesting;description;for;some;columns;below or above blah;fasel;kek;blubb;10;432441 foo;bar;baz;<3;2352;23 </evenmoredata> </data>


  1. Find the dataset lines matching the definitions.
  2. Parse a single document containing 68000 such datasets.
  3. Write all of the data into an SQL database using sensible data types (text for each column does not count!) succeeding on the first try.

The Just in Time Concept

If you have come here in search of knowledge regarding JIT concepts as in “conecepts of the Just in Time principle” you have, quite non-obviously I admit, come to the wrong place. This article presents a whole new (business point-of-)view of the Just in Time Concept… as in “a concept just in time”.

The Just in Time Project – project manager point view

The problem with detailed and overly complicated project plans is… well they take their time. And we all know, time is money! And if the time you spend on your project plan is as little as possible, you can save a lot of money, right? Right?

Right! It'll all come “just in time”.

Well, ok there are some side effects. It generally goes like this: the project plan is a one-liner “make it work, make it look good and make it ASAP”. Not having any big picture the staff makes it work and look decent. And then the management (read: not project management) takes a look at it and, well, decides that a concept is needed!

But if all of this works in the first try, then you indeed saved a lot of money. Worth a shot, isn't it?

The developers' point of view

Before we begin, imagine for a moment the following development environment: a project devided in 3 principle parts with one developer responsible for one of each. Now the just in time concept's main dogma is again to save money. And so everyone starts right away, diving into their work like there is no tomorrow.

As the project evolves into a somewhat bigger gibberish mass from whence it started, person A needs to touch areas of person B, but since there is no conecpt for the whole thing (remember, we're saving time and money), person A goes ahead and changed stuff person B is working on. Which in turn pisses of confuses person B since in his little world of project part B things might probably work a little bit different. But never mind, B can change it back the way he wants it to work and break a part A was working on, after all he doesn't need to care for part A since there is no concept. ;)

To be continued…

Meisenmonitor reissue

older stuff →

Index: 1 - 2