[c2k8]: Hackathon Summary Part 8

c2k8 General Hackathon (Part 8) - June 7-15, 2008, Edmonton, Alberta, Canada I often find it ironic and a little sad that one of the most important security applications on the Internet is so poorly funded. For anyone that manages machines remotely and is concerned about security, it is perhaps the most important application to rely on for security and stability. I'm sure you know which application I am referring to, but do you know that it is the de facto world standard in its class on the Internet?

Read on to find out why and more:

I wonder if there is any other application on the Internet, proprietary or Open Source, that comes close to the market penetration that OpenSSH has earned compared to competing alternatives. Damien Miller (djm@), a name synonymous to OpenSSH, and I (mtu@) have just completed another OpenSSH scan. We scanned 2²⁵ random hosts in three days. You can glean the latest results here. This is the third scan that we have done together over the past four years. The statistics are interesting. For example, the percentage of OpenSSH servers decreased slightly since the last scan in 2005 but the actual number of OpenSSH servers have doubled on the Internet in that same period!

Just so that we are clear about this, the scan data is kept confidential and is never used for anything other than statistical analysis. Even I don't have a copy of the data. Essentially, I prepare and provide the resources (machines and pipes) and Damien comes in over ssh to set up and kick-start the random scans. The most recent scan was done using 9 Dell PE1550 machines (yes, I like Dell) on a 100 Mbps fibre line.

When I helped with the first scan about four years ago, I received one complaint from my ISP. They sent me an email telling me that one of my servers was most likely infected and politely asked me to look into it. Over three years later on my third scan, not much has changed. I received a few emails that were directed to me and the others were fielded by my second ISP. They just wanted an explanation for the scan. I figure that there is so much bad stuff hitting everyone that these innocuous scans have become all but noise in the background.

I have to admit that I sleep better at night knowing that OpenSSH is so mature and has stood the test of time. You couldn't ask for a better team of developers looking out for everyones' security interests. I also realize that it is a heavy burden to bear. However, I can't think of another program other than postfix that has a better security track record over time. When you consider Internet penetration with over 80% of the SSH servers on the Internet running OpenSSH, then I would think OpenSSH is in a league of its own. This is a program that people have come to trust. Damien's "Security Measures in OpenSSH" talk (slides and paper) may help you understand why so many have come to trust OpenSSH. Personally, I think that coupled with authpf-noip, it makes for an ideal remote access solution for Windows users :-).

But let's talk about the lack of funding for OpenSSH development. It's a pretty sad case. There was an incredible amount of work done prior to the release of OpenSSH 5.1. In fact, this was mainly possible because of the two hackathons, n2k8 and c2k8, held during the regular release cycle. Sadly, most of this is funded by the developers themselves. Damien and many other developers travelled to c2k8 on their own dime. And what of their time? It is no secret that more funding means more hackathons which result in substantial improvements. However, there are other basic infrastructure costs such as hardware and co-location expenses. Damien has informed me that he actually does need a new 1RU server for portable OpenSSH really badly. Can I make a plea to all the satisfied OpenSSH users out there to contact Damien privately to help out with this purchase? Everyone who helps will be credited irrespective of the amount. He can accept PayPal donations to djm@mindrot.org. Both Damien and I would be grateful for any help that you can give.

As I've mentioned in the past, Damien is one of my favourites and as you'll read below, OpenSSH is not the only thing that he works on. Here is what Damien had to say about his time at the c2k8 hackathon:

Having barely recovered from the n2k8 network hackathon in Japan, I find myself at another one! beck@ did a great job of organising this one, and I was quite productive:

AES-XTS

My first work at c2k8 was to implement the XTS (XEX with ciphertext stealing) mode of operation for AES in the kernel crypto(9) framework. This cipher mode has some really nice properties for block device encryption: there is no length expansion, it resists the "cut and paste" attack on CBC modes by diversifying the plaintext for each cipher block and it is quite computationally cheap: only one extra encryption per disk block and some basic finite field arithmetic per cipher block. It is also recommended by the IEEE P1619-2007 block encryption standard so it has received a good deal of cryptanalytic attention.

crypto regress tests

After adding AES-XTS to crypto(9) I added a proper regression test that uses /dev/crypto to submit test vectors to the kernel. Fortunately markus@ had written a DES regress test that I was able to adapt, which saved me a fair bit of work. Later, I added a few regress tests for AES using the same /dev/crypto mechanism. Neither these nor the AES-XTS tests uncovered any bugs, but it is good to have these around to detect any future breakage. I'm trying to get into the habit of making unit or regression tests for everything I write recently - it can be a bit tedious but they often uncover bugs, especially latent ones that would otherwise only manifest after the code is run in an unfamiliar environment or has otherwise been re-purposed.

Non-repeating 32-bit ID generator

While I was still in a crypto mood, I added a new non-repeating ID generator for 32-bit IDs based on a modified block cipher. Block ciphers are effectively keyed permutations, so a small blocksize cipher is a natural fit for network applications that need unpredictable IDs with a long period between repetitions of the same ID; just encrypt a counter using a random key. Well, almost: we actually want to rekey the block cipher fairly often, otherwise the sequence would be 100% predictable after its first cycle - an attacker could just record the sequence and replay it.

Unfortunately rekeying the cipher destroys its cycle-length guarantee. A new key means a new permutation and runs the risk one of the early numbers out of the generator under the new key could be be equal to one of the recent numbers generated with with the old key. A neat trick that was implemented in Niels Provos' LCG ID generator was to toggle the most significant bit at the time of rekeying - this allows the lower bits to repeat after a new key has been applied without shortening the overall cycle length.

However this leads to yet another problem: there are no common block ciphers with odd block sizes (32 bits - 1 bit for the MSB twiddle = 31 bits). The solution was to modify an existing 32-bit block length cipher (SKIP32 by Greg Rose, itself a modification of Skipjack) into a 31-bit pseudo-random permutation by tweaking its round structure to accommodate the odd block width. The top bit is reserved to toggle when the generate is rekeyed, which yields a period close to 2^32 (we rekey a little early to prevent a "card counting" attack, so it isn't exactly 2^32). The new ID generator, idgen32(9), was immediately put to use generating IPv6 IDs and NFS XIDs. This just leaves one LCG-based ID generator in the tree: the 20-bit IPv6 flow label generator. I'm not aware of any attacks that depend on predicting the IPv6 flow label, so I'm going to leave it alone.

OpenSSH

Next, I did a bunch of OpenSSH work. Darren Tucker, another OpenSSH developer and fellow Australian was enjoying his first hackathon and was at my table in the room. Our proximity made for rapid progress which he has already described. Between c2k8 and n2k8 there have been many feature improvements and bug fixes that made their way into OpenSSH 5.1 release which was our biggest in several years.

crypto softraid

I spent quite a bit of time at n2k8 working with hshoexer@ to design and implement the new softraid(4) crypto discipline. Fortunately marco@ was on hand to help with the general softraid parts too. The new crypto RAID support uses AES-XTS to encrypt the blocks on disk, and has quite a nice key management system that supports the common "unlock with a passphrase" mode that everyone expects but could be extended to support keys on separate removable media (unlock your HD with a USB key you keep in your pocket) or other schemes later. hshoexer@ did most of the hard work in implementing the softraid crypto code and ironing out the kinks. By the end of the hackathon, it was working and enabled in the kernel. Again, sitting across a table from Marco and Hans made progress much easier.

After that, marco@ and I went through and redesigned the on-disk metadata format for softraid in general. The new format will be less complex but will give better detection of metadata corruption and will handle unclean shutdowns much better. Once this work is complete, I plan to start looking at boot loader support for softraid volumes (especially crypto volumes), but it is unlikely that this will be done in time for 4.4.

Looking at omalloc

I also finally got some time to review otto@'s new malloc implementation, which had been on my TODO list for some time. If you have ever looked at malloc code, you may agree that they are generally fairly complex and difficult to follow. Not so omalloc - it is fantastically clean and readable code. Otto's code was very high quality and I could find no real bugs, but did have a few suggestions on how to protect against attacks on the malloc data structures themselves. I'm sufficiently confident in the new code that I'm now running it on all my boxes.

-djm

Once again, I would like to thank Damien and all the OpenSSH developers for their brilliant work. I hope that people will show their appreciation with a small donation to Damien at djm@mindrot.org for his urgent hardware needs. Thanks in advance!

(c2k8 hackathon summary to be continued)