Monday, 07 January 2019
I've posted this analysis to Debian mailing lists already, but I'm thinking it's also useful as a blog post too. I've also fixed a few typos and added a few more details that people have suggested.
This has taken a while in coming, for which I apologise. There's a lot of work involved in rebuilding the whole Debian archive, and many days spent analysing the results. You learn quite a lot, too! :-)
I promised way back before DebConf 18 last August that I'd publish the results of the rebuilds that I'd just started. Here they are, after a few false starts. I've been rebuilding the archive specifically to check if we would have any problems building our 32-bit Arm ports (armel and armhf) using 64-bit arm64 hardware. I might have found other issues too, but that was my goal.
The logs for all my builds are online at
for reference. See in particular
for automated analysis of the build logs that I've used as the basis for the stats below.
As far as I can see we're basically fine to use arm64 hosts for building armel and armhf, so long as those hosts include hardware support for the 32-bit A32 instruction set. As I've mentioned before, that's not a given on all arm64 machines, but there are sufficient machine types available that I think we should be fine. There are a couple of things we need to do in terms of setup - see Machine configuration below.
I (naively) just attempted to rebuild all the source packages in unstable main, at first using pbuilder to control the build process and then later using sbuild instead. I didn't think to check on the stated architectures listed for the source packages, which was a mistake - I would do it differently if redoing this test. That will have contributed quite a large number of failures in the stats below, but I believe I have accounted for them in my analysis.I built lots of packages, using a range of machines in a small build farm at home:
using my local mirror for improved performance when fetching build-deps etc. I started off with a fixed list of packages that were in unstable when I started each rebuild, for the sake of simplicity. That's one reason why I have two different numbers of source packages attempted for each arch below. If packages failed due to no longer being available, I simply re-queued using the latest version in unstable at that point.
I then developed a script to scan the logs of failed builds to pick up on patterns that matched with obvious causes. Once that was done, I worked through all the failures to (a) verify those patterns, and (b) identify any other failures. I've classified many of the failures to make sense of the results. I've also scanned the Debian BTS for existing bugs matching my failed builds (and linked to them), or filed new bugs where I could not find matches.
I did not investigate fully every build failure. For example, where a package has never been built before on armel or armhf and failed here I simply noted that fact. Many of those are probably real bugs, but beyond the scope of my testing.
For reference, all my scripts and config are in git at
Almost half of the failed builds were simply due to the lack of a single desired build dependency (nodejs:armel, 1289). There were a smattering of other notable causes:
Considering the number of package builds here, I think these numbers are basically "lost in the noise". I have found so few issues that we should just go ahead. The vast majority of the failures I found were either already known in the BTS (260), unrelated to what I was looking for, or both.
See below for more details about build host configuration for armel builds.
FTAOD: I attempted fewer package builds for armhf as we simply had a smaller number of packages when I started that rebuild. A few weeks later, it seems we had a few hundred more source packages for the armel rebuild.
The armhf rebuild showed broadly the same percentage of failures, if you take into account the nodejs difference - it exists in the armhf archive, so many hundreds more packages could build using it.
In a similar vein for notable failures:
Arch-specific failures found for armhf:
and the new bugs I filed:
Again, these small numbers tell me that we're fine. I liked to 139 existing bugs in the BTS here.
To be able to support 32-bit builds on arm64 hardware, there are a few specific hardware support issues to consider.
Our 32-bit Arm kernels are configured to fix up userspace alignment faults, which hides lazy programming at the cost of a (sometimes massive) slowdown in performance when this fixup is triggered. The arm64 kernel cannot be configured to do this - if a userspace program triggers an alignment exception, it will simply be handed a SIGBUS by the kernel. This was one of the main things I was looking for in my rebuild, common to both armel and armhf. In the end, I only found a very small number of problems.
Given that, I think we should immediately turn off the alignment fixups on our existing 32-bit Arm buildd machines. Let's flush out any more problems early, and I don't expect to see many.
To give credit here: Ubuntu have been using arm64 machines for building 32-bit Arm packages for a while now, and have already been filing bugs with patches which will have helped reduce this problem. Thanks!
Deprecated / retired instructions
In theory(!), alignment is all we should need to worry about for armhf builds, but our armel software baseline needs two additional pieces of configuration to make things work, enabling emulation for
Again, there is quite a performance cost to enabling emulation support for these instructions but it is at least possible!
In my initial testing for rebuilding armhf only, I did not enable either of these emulations. I was then finding lots of "Illegal Instruction" crashes due to CP15 barrier usage in armhf Haskell and Mono programs. This suggests that maybe(?) the baseline architecture in these toolchains is incorrectly set to target ARMv6 rather than ARMv7. That should be fixed and all those packages rebuilt at some point.
There are a few things I found that I'd like to highlight:
Other things to think about
Building in VMs
So far in Debian, we've tended to run our build machines using chroots on raw hardware. We have a few builders (x86, arm64) configured as VMs on larger hosts, but as far as I can see that's the exception so far. I know that OpenSUSE and Fedora are both building using VMs, and for our Arm ports now we have more powerful arm64 hosts available it's probably the way we should go here.
In testing using "linux32" chroots on native hardware, I was explicitly looking to find problems in native architecture support. In the case of alignment problems, they could be readily "fixed up / hidden" (delete as appropriate!) by building using 32-bit guest kernels with fixups enabled. If I'd found lots of those, that would be a safer way to proceed than instantly filing lots of release-critical FTBFS bugs. However, given the small number of problems found I'm not convinced it's worth worrying about.
Utilisation of hardware
Another related issue is in how we choose to slice up build machines. Many packages will build very well in parallel, and that's great if you have something like the Synquacer with many small/slow cores. However, not all our packages work so well and I found that many are still resolutely chugging through long build/test processes in single threads. I experimented a little with my config during the rebuilds and what seemed to work best for throughput was kicking off one build per 4 cores on the machines I was using. That seems to match up with what the Fedora folks are doing (thanks to hrw for the link!).
Migrating build hardware
As I mentioned earlier, to build armel and armhf sanely on arm64 hardware, we need to be using arm64 machines that include native support for the 32-bit A32 instruction set. While we have lots of those around at the moment, some newer/bigger arm64 server platforms that I've seen announced do not include it. (See an older mail from me for more details. We'll need to be careful about this going forwards and keep using (at least) some machines with A32. Maybe we'll migrate arm64-only builds onto newer/bigger A64-only machines and keep the older machines for armel/armhf if that becomes a problem?
At least for the foreseeable future, I'm not worried about losing A32 support. Arm keeps on designing and licensing ARMv8 cores that include it...
I've spent a lot of time looking at existing FTBFS bugs over the last weeks, to compare results against what I've been seeing in my build logs. Much kudos to people who have been finding and filing those bugs ahead of me, in particular Adrian Bunk and Matthias Klose who have filed many such bugs. Also thanks to Helmut Grohne for his script to pull down a summary of FTBFS bugs from UDD - that saved many hours of effort!
Please let me know if you think you've found a problem in what I've done, or how I've analysed the results here. I still have my machines set up for easy rebuilds, so reproducing things and testing fixes is quite easy - just ask!Friday, 31 August 2018
As is becoming something of a tradition by now, Jo and I hosted another OMGWTFBBQ at our place last weekend. People came from far and wide to enjoy themselves. Considering the summer heatwave we've had this year, we were a little unlucky with the weather. But with the power of gazebo technology we kept (mostly!) dry... :-)
I was too busy cooking and drinking etc. to take any photos myself, so here are some I sto^Wborrowed from my friends!
We continued to celebrate Debian getting old:
We had much beer from the nice folks at Milton Brewery:
Much meat was prepared and cooked:
And we had a lot of bread too!
Finally, many thanks to a number of awesome companies for again sponsoring the important refreshments for the weekend. It's hungry/thirsty work celebrating like this!Thursday, 16 August 2018
We had a small gathering in the Haymakers pub tonight to celebrate 25 years since Ian Murdock started the Debian project.
We had 3 DPLs, a few other DDs and a few more users and community members! Good to natter with people and share some history. :-) The Raspberry Pi people even chipped in for some drinks. Cheers! The celebrations will continue at the big BBQ at my place next weekend.Sunday, 12 August 2018
So I'm slowly recovering from my yearly dose of full-on Debian! :-) DebConf is always fun, and this year in Hsinchu was no different. After so many years in the project, and so many DebConfs (13, I think!) it has become unmissable for me. It's more like a family gathering than a work meeting. In amongst the great talks and the fun hacking sessions, I love catching up with people. Whether it's Bdale telling me about his fun on-track exploits or Stuart sharing stories of life in an Australian university, it's awesome to meet up with good friends every year, old and new.
For once, I even managed to find time to work on items from my own TODO list during DebCamp and DebConf. Of course, I also got totally distracted helping people hacking on other things too! In no particular order, stuff I did included:
I was involved in a lot of sessions this year, as normal. Lots of useful discussion about Ignoring Negativity in Debian, and of course lots of updates from various of the teams I'm working in: Arm porters, web team, Secure Boot. And even an impromptu debian-cd workshop.
I loved my time at the first DebConf in Asia (yay!), and I was yet again amazed at how well the DebConf volunteers made this big event work. I loved the genius idea of having a bar in the noisy hacklab, meaning that lubricated hacking continued into the evenings too. And (of course!) just about all of the conference was captured on video by our intrepid video team. That gives me a chance to catch up on the sessions I couldn't make it to, which is priceless.
So, despite all the stuff I got done in the 2 weeks my TODO list has still grown. But I'm continuing to work on stuff, energised again. See you in Curitiba next year!Friday, 25 August 2017
It's that time again! Another year, another OMGWTFBBQ! We're expecting 50 or so Debian folks at our place in Cambridge this weekend, ready to natter, geek, socialise and generally have a good time. Let's hope the weather stays nice, but if not we have gazebo technology... :-)
Many thanks to a number of awesome companies and people near and far who are sponsoring the important refreshments for the weekend:
I've even been working on the garden this week to improve it ready for the event. If you'd like to come and haven't already told us, please add yourself to the wiki page!Thursday, 22 June 2017
Here's a nice comment I received by email this morning. I guess somebody was upset by my last post?
From: Tec Services <firstname.lastname@example.org> Date: Wed, 21 Jun 2017 22:30:26 -0700 To: email@example.com Subject: its time for you to retire from debian...unbelievable..your the quality guy and fucked up the installer! i cant ever remember in the hostory of computing someone releasing an installer that does not work!! wtf!!! you need to be retired...due to being retarded.. and that this was dedicated to ian...what a disaster..you should be ashames..he is probably roling in his grave from shame right now....
It's nice to be appreciated.Tuesday, 20 June 2017
Things mostly went very well, and we've released Debian 9 this weekend past. Many many people worked together to make this possible, and I'd like to extend my own thanks to all of them.
As a project, we decided to dedicate Stretch to our late founder Ian Murdock. He did much of the early work to get Debian going, and inspired many more to help him. I had the good fortune to meet up with Ian years ago at a meetup attached to a Usenix conference, and I remember clearly he was a genuinely nice guy with good ideas. We'll miss him.
For my part in the release process, again I was responsible for producing our official installation and live images. Release day itself went OK, but as is typical the process ran late into Saturday night / early Sunday morning. We made and tested lots of different images, although numbers were down from previous releases as we've stopped making the full CD sets now.
Sunday was the day for the release party in Cambridge. As is traditional, a group of us met up at a local hostelry for some revelry! We hid inside the pub to escape from the ridiculouly hot weather we're having at the moment.
Due to a combination of the lack of sleep and the heat, I nearly forgot to even take any photos - apologies to the extra folks who'd been around earlier whom I missed with the camera... :-(Friday, 12 May 2017
When you're giving a presentation, the choice of font can matter a lot. Not just in terms of how pretty your slides look, but also in terms of whether the data you're presenting is actually properly legible. Unfortunately, far too many fonts are appallingly bad if you're trying to tell certain characters apart. Imagine if you're at the back of a room, trying to read information on a slide that's (typically) too small and (if you're unlucky) the presenter's speech is also unclear to you (noisy room, bad audio, different language). A good clear font is really important here.
To illustrate the problem, I've picked a few fonts available in Google Slides. I've written the characters "1lIoO0" (that's one, lower case L, upper case I, lower case o, upper case O, zero) in each of those fonts. Some of the sans-serif fonts in particular are comically bad for trying to distinguish between these characters.
It may not matter in all cases if your audience can read all the characters on your slides and tell them apart, put if you're trying to present scientific or numeric results it's critical. Please consider that before looking for a pretty font.Wednesday, 15 February 2017
This probably won't mean much to people outside the UK, I'm guessing. Sorry! :-)
The Crystal Maze was an awesome fun game show on TV in the UK in the 1990s. Teams would travel through differently-themed zones, taking on challenges to earn crystals for later rewards in the Crystal Dome. I really enjoyed it, as did just about everybody my age that I know of...
A group have started up a new Crystal Maze attraction in London and Manchester, giving some of us a chance of indulging our nostalgia directly in a replica of the show's setup! Neil NcGovern booked a load of tickets and arranged for a large group of people to go along this weekend.
It was amazing! (Sorry!) I ended up captaining one of the 4 teams, and our team ("Failure is always an option!") scored highest in the final game - catching bits of gold foil flying around in the Dome. It was really, really fun and I'd heartily recommend it to other folks who like action games and puzzle solving.
I just missed the biting scorn of the original show presenter, Richard O'Brien, but our "Maze Master" Boudica was great fun and got us all pumped up and working together.Tuesday, 01 November 2016
So, it's now been twenty years since I became a Debian Developer. I couldn't remember the exact date I signed up, but I decided to do some forensics to find out. First, I can check on the dates on my first Debian system, as I've kept it running as a Debian system ever since!
jack:~$ ls -alt /etc ... -rw-r--r-- 1 root root 6932 Feb 10 1997 pine.conf.old -rw-r--r-- 1 root root 6907 Dec 29 1996 pine.conf.old2 -rw-r--r-- 1 root root 76739 Dec 7 1996 mailcap.old -rw-r--r-- 1 root root 1225 Oct 20 1996 fstab.old jack:~$
I know that I did my first Debian installation in late October 1996, migrating over from my existing Slackware installation with the help of my friend Jon who was already a DD. That took an entire weekend and it was painful, so much so that several times that weekend I very nearly bailed and went back. But, I stuck with it and after a few more days I decided I was happier with Debian than with the broken old Slackware system I'd been using. That last file (fstab.old) is the old fstab file from the Slackware system, backed up just before I made the switch.
I was already a software developer at the time, so of course the first thing I wanted to do once I was happy with Debian was to become a DD and take over the Debian maintenance of mikmod, the module player I was working on at the time. So, I mailed Bruce to ask for an account (there was none of this NM concept back then!) and I think he replied the next day. Unfortunately, I don't have the email in my archives any more due to a disk crash back in the dim and distant past. But I can see that the first PGP key I generated for the sake of joining Debian dates from October 30th 1996 which gives me a date of 31st October 1996 for joining Debian.
Twenty years, wow... Since then, I've done lots in the project. I'm lucky enough to been to 11 DebConfs, hosted all around the world. I'm massively proud to have been voted DPL for two of those twenty years. I've worked on a huge number of different things in Debian, from the audio applications I started with to the installer (yay, how things come back to bite you!), from low-level CD and DVD tools (and making our CD images!) to a wiki engine written in python. I've worked hard to help make the best Operating System on the planet, both for my own sake and the sake of our users.
Debian has been both excellent fun and occasionally a huge cause of stress in my life for the last 20 years, but despite the latter I wouldn't go back and change anything. Why? Through Debian, I've made some great friends: in Cambridge, in the UK, in Europe, on every continent. Thanks to you all, and here's to (hopefully) many years to come!Friday, 09 September 2016
Another year, another OMGWTFBBQ. By my count, we had 49 people (and a dog) in my house and garden at the peak on Saturday evening. It was excellent to see people from all over coming together again, old friends and new. This year we had some weather issues, but due to the delights of gazebo technology most people stayed mostly dry. :-)
Also: thanks to a number of companies near and far who sponsored the important refreshments for the weekend:
As far as I could tell, everybody enjoyed themselves; I know I definitely did!Friday, 02 September 2016
A couple of weekends back, we had an awesome time at Jonathan and Charlene's wedding. I'd have blogged sooner, but I had to wait for this photo of the happy couple with the Debian gang... :-)
Unfortunately, somebody let Charlene hide at the back! Follow the link for more photos...Sunday, 29 November 2015
I'm happy to chip in and help the awesome folks at the Software Freedom Conservancy with funding for their work, standing up for copyleft. It's clear that there are a lot of people who will ignore the terms of Free Software licensing, whether by oversight or deliberately, so the SFC are doing an important job working to defend the Freedoms that lots of us are using every day as developers and users.
It seems that some of SFC's corporate supporters have stopped sponsorship since the beginning of their lawsuit in Germany against VMware for GPL violations. Maybe some folks are happy to support SFC, but not when they really push things like this. Let's hope that we can find many more individual supporters to cover SFC's funding needs instead. This week, there's an anonymous pledge to match donations from new supporters so right now is an even better time to sign up!
Thanks to Paul Wise for posting about this and indirectly prodding me to sign up too.Friday, 25 September 2015
VLANd is a python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together.
There's more information in the README file. I've just released v0.4, with a lot of changes included since the last release:
VLANd is Free Software, released under the GPL version 2 (or any later version). For now, grab it from git; tarballs will be coming shortly.Sunday, 02 August 2015
We've just started a new team in Debian for maintaining our UEFI packages together, with git repositories in a shared project on alioth etc. We're just working out the exact details of how we're going to manage things, but for now we've moved the following packages under the team's umbrella:
and in the future we'll clearly end up adding more. We've also started a new IRC channel (#debian-efi) on irc.debian.org aka irc.oftc.net. New members always welcome to help with the work here!
There can be issues with shipping installer images including UEFI. But they're mainly due to crappy UEFI implementations that vendors have shipped. It's fairly well-known that Apple have shipped some really shoddy firmware over the years, and to allow people to install Debian on older Apple x86 machines we've now added the workaround of a non-UEFI 32-bit installer image too. But Apple aren't the only folks shipping systems with horrendously buggy UEFI, and a lot of Linux folks have had to deal with this over the last few years.
I've been talking to a number of other UEFI developers lately, and we've agreed to start a cross-distro resource to help here - a list of known-broken UEFI implementations so that we can share our experiences. The place for this in in the OSDev wiki at http://wiki.osdev.org/Broken_UEFI_implementations. We're going to be adding new information here as we find it. If you've got a particular UEFI horror story on your own broken system, then please either add details there or let me know and I'll try to do it for you.
You might have seen some of the posts I've written in the last few months about adding support in Debian for so-called Mixed-EFI systems like the Intel Bay Trail: a 64-bit processor shipped with a 32-bit EFI implementation.
I've finally seen a public justification from Intel evangelist Brian Richardson as to why these systems are crippled^Wconfigured this way, and it's nice to see our guesses confirmed. The reason is simply cost - like most consumer PCs shipped today, they come with Windows. In terms of system design, it's cheaper to just include the limited memory and storage needed for 32-bit Windows. 64-bit Windows takes a lot more storage in particular. And on modern systems 32-bit Windows can only boot using 32-bit UEFI. Fair enough...
However, Brian goes on to state some more things that are simply out of date, saying that "Linux support for UEFI IA32 is still an unanswered question". Ummm, Brian: we've got working 32-bit x86 UEFI support in our standard Jessie (and newer) installation images already, and they work just fine on CD/DVD or USB stick. We've even gone one stage further than anybody else (thus far!) in adding easy support for running a full 64-bit Linux system on top of those 32-bit UEFI implementations.
I say "thus far" here because all the work here here is Free Software. Other folks added the support in Linux for making a 64-bit kernel work with a 32-bit UEFI; I added code in Linux to expose some of the details to userspace, and code in Grub to work with it. My changes have gone upstream already, so I'd expect to see other distros like Fedora or Ubuntu also using them soon.Friday, 31 July 2015
VLANd is a python program intended to make it easy to manage port-based VLAN setups across multiple switches in a network. It is designed to be vendor-agnostic, with a clean pluggable driver API to allow for a wide range of different switches to be controlled together.
There's more information in the README file. I've just released v0.3, with a lot of changes included since the last release:
VLANd is Free Software, released under the GPL version 2 (or any later version). For now, grab it from git; tarballs will be coming shortly.Wednesday, 01 July 2015
Jo and I spent a few days in Sweden and had an awesome time! The main reason for being there was Leif and Maria's wedding way up north in Skellefteå. They cunningly organised their ceremony for the Midsummer weekend, which was an excellent plan - we had a full weekend of partying while we were there. :-)
We had some time to ourselves while we were there, so we wandered about a little and got to see some of the beautiful coastal countryside.
Then on the way home we stopped off in Umeå to visit Mattias Wadenstein (maswan) and his wife Melanie, and he showed me around some of the machines that he's been admining on behalf of Debian. Maybe I'm a sad geek, but I feel quite a bond with one of the machines there, pettersson.debian.org. It's the official CD build machine for Debian, and I've been responsible for thrashing it really hard for the last 5 years or so... :-)
The only downsides from the trip were the massive tiredness (midnight sun is pretty, but notconducive to sleep!) the mosquito bites and the nasty plague^Wcold that we picked up while we were there... Ah well. :-)Thursday, 11 June 2015
I've had some 8GB USB keys made, with the Debian swirl and text. By buying a reasonable number, I've got what I think is a good price for nice high-quality keys (metal body with a solid loop for attaching to a keyring). I'm now selling these for 7 pounds each, and I'm planning on bringing some to DebConf 15 too, where they'll be 10 EUR.
They're selling faster than I expected - if you're interested in buying one (or several!), please let me know. If there's enough demand, I may order more.