Not Much Ado About Anything

OK, not the most original title. But quite apt as I really don't intend to blog very much or very seriously.

See the archive of all posts for what little is there.

If you want to comment on an entry, please do so by email. I will post updates if you send relevant stuff my way.

If you haven't seen it, try to. Amazing this movie did not get any major awards.

Maggie Gyllenhaal is well worth looking out for in other movies.

Posted Mon Aug 9 05:02:56 2010

I got this neat little Chinese netbook after a mail to the debian-arm list where one machine was offered in exchange for porting Debian to it. So I offered to get Debian Installer running on it.

In total 20 of these ChiTech PC89E netbooks were bought as a group by various people in different countries. The netbook is based on an ARM S3C6410 SoC, has 256 MB RAM and an excellent 8.9" 1024x600 LCD display. The machine itself is 11" and weighs under 800 grams. Most of the weight is the LCD display; they've even had to add a small extra weight in the base to avoid it toppling over backwards.

The goal of getting D-I running was achieved last week, though not without needing to overcome some steep hurdles.

Our main problem is that we don't have the source code to either the kernel they use (2.6.24.2 with patches from Samsung and others), or the u-boot bootloader.

SSH was available without password for the user account and various basic errors allowed us to root the system relatively quickly. Brute force cracked the root password a few days later.

The only information we did have was from the provided desktop system (which is really quite good and certainly looks very polished, but in the end still way too limited), what's on the flash memory and the contents ofa "firmware upgrade" image they provided.

Having that firmware upgrade image proved very important as it gave us a good idea how to boot the system from SD card and made it possible to locate the kernel, u-boot bootloader and other files in flash memory.

Googling for some boot messages we found two kernel source trees (for a different device: SmartQ) that looked promising as a basis. After some disassembly work by Luke Kenneth Casson Leighton on the kernel to get LCD timings and correct GPIO data we managed to get a basic working kernel: LCD, USB (and thus keyboard and touchpad), and wired networking.

Although there are still loads of things that need improving/fixing, that gave me enough to work with to try to get Debian Installer going.

The main limitations for running D-I are that we cannot change the bootloader configuration and that their boot procedure does not use an initrd. But we did work out how to boot a custom kernel from an external SD card by making creative use of their "firmware upgrade" procedure.

Essentially I needed to find a way to run D-I by only booting a kernel. Piggy-backing an initrd onto a kernel is possible (using the 'bootpImage' target from the upstream kernel build system), but then the size of kernel plus initrd is limited to 4 MB and that is really too limited for a decent D-I initrd (especially if you want full i18n support).

I ended up creating a micro-initrd (only 66 kB!) which only function is to mount the external SD card, load the real D-I root initramfs (which now no longer has any real size limitation) and then run init in that.

After adding some relatively minor customizations for this netbook (such as installing the kernel from a custom repository on alioth), the installer was up and running.

As the framebuffer worked without problem for the newt frontend, I next wondered if it would be possible to also get the graphical installer working. This should be easier for non-x86 systems after the recent switch from DirectFB to X.Org as backend for the graphical installer. And the image below shows it was. I had to create an /etc/xorg.conf to get the USB keyboard and touchpad working (apparently auto-detection does not quite work for less standard hardware), but that was basically it.

So now all we have to do is get all the remaining kernel issues sorted out...

Posted Tue Mar 30 14:36:19 2010

Until two years or so ago I used VMware Workstation intensively for development and testing of Debian Installer, using one of the licenses generously made available to the D-I team by VMware. It was much faster than Qemu and thus very welcome.

But due to increasing problems keeping it working with new kernels on a desktop system running unstable, I switched to VirtualBox (the OSE version), which has the huge advantage of being packaged for Debian and thus automatically follows the Debian kernel packages.

That advantage has disappeared somewhat as I'm currently compiling my own kernels — I'm currently running 2.6.33 on a Lenny desktop. So far I've been able without too much trouble to find patches for the VirtualBox modules to keep them compatible with new upstream kernels.

This week I had reason to try VMware again. The Server edition this time as that has free licenses. The Server concept, where you connect to the management system and virtual machines via a web browser, is quite nice. It means I could install it on a server and keep my laptop nice and responsive.

To keep VMware separate from the rest of the system and not mix it with software installed cleanly from Debian packages, I decided to install it in /opt. And that works beautifully. There are only a few bits installed in /etc, but that's OK. I have /etc managed by etckeeper anyway [1].

More challenging was building WMware modules to match my kernel. VMware wants you to run their vmware-config.pl script every time you switch kernels and build the kernel modules on the system where it runs (which requires the kernel headers). I wanted to build the modules as part of my kernel build process, on a totally different machine. The same as I already do for VirtualBox. I save the custom modules in a tarball that has the same uname and package version as the kernel package I build and have a simple script that installs them correctly.

I succeeded by hacking the Makefiles (deleting most of their content) and then integrating them in my kernel build wrapper script.

The module source provided by VMware built fine with a Debian 2.6.26 kernel, but not with the 2.6.32.9 kernel on my server (of course). But luckily others had already run into the same problems and solved them, so with some searching I found the required patches (first a full updated source for 2.6.29, then patches needed for .31 and for .32). Isn't open source just wonderful?

So with that the VMware server started beautifully (well, the init script needed to be told that module names have changed from *.o to *.ko), but I could not connect to it from my laptop using SSL. Connecting without SSL (using an instance of iceweasel running on the server but started remotely from my laptop using X forwarding SSH) worked fine. That turned out to be a known issue too. After applying all the suggested workarounds in my iceweasel configuration (disable TLS 1.0; set cache size to zero; explicitly disable use of proxy for the server) I can now connect reliably to the VMware Server management system.

Phew.

[1] I had to exclude a fair number of VMware files for etckeeper as they are updated by VMware itself and thus really belong in /var. Example are DCHP status files...

Posted Fri Feb 26 16:36:33 2010 Tags: debian

The Debian FTP-masters recently changed the way gzipped meta files are compressed in order to make them more efficient to update using the rsync option. This was done by adding the --rsyncable option when calling gzip.

Consequence was however that when debmirror compressed Packages, Sources and Contents files after updating them by applying diffs, the md5sum of the gzipped file created by debmirror no longer matched the md5sum listed in the Release file (because debmirror did not use --rsyncable).

Result was that debmirror would also download the full gzipped Packages, Sources and Contents files from the parent mirror, something the diffs are meant to avoid. Not nice.

Anyway, this has been fixed in debmirror 2.4 which now by default also uses --rsyncable when gzipping the updated meta files.

I've also uploaded a fixed version for Lenny (20070123lenny1), which should soon be available from proposed-updates and will be included in the next stable point release.

For archives that also provide diffs (most archives don't have them) but do not have rsyncable gzipped files, the default options used when calling gzip can be overruled using the new option --gzip-options (only in version 2.4).

Tip: if you are using the rsync method to download files, using --diff=none may well be more efficient now that the archive has rsyncable gzipped meta files.

Version 2.4 also has a few other improvements and fixes. If you're currently using version 2.3.x an update to the new version is probably a good idea.

Posted Sat Dec 19 22:39:04 2009 Tags: debian debmirror

debmirror 2.3 should be hitting the mirrors about now. Main change is that it will now use the available diffs to update Contents files, which should give a nice bandwidth reduction for users who mirror those.

With that the option --pdiff (for "package diff") no longer really covered its function, so I decided to change it to --diff.

There's also a fix for mirroring archives that don't have a Release file.

Question for users

The option --add-dir has been marked as deprecated (for quite some time now I suspect). I'm considering to remove it in the next release as I cannot see any use cases for it, but it's quite possible I'm missing something and there are still people using it. If you would like that option preserved, then please mail me at debmirror@packages.d.o with an explanation of why and how you use it.

Managing the size of a local mirror

The archive has grown a lot over the past Debian releases and keeping even a partial local mirror can require quite some disk space. Luckily debmirror offers quite a few options to tune what is mirrored.

My own mirror covers testing and unstable 'main' for 6 architectures (i386, amd64, armel, hppa, sparc and s390), no source, no D-I images. It uses only 61G. I say "only" as that's about 33GB less than it could have been without tuning. In other words, I'm saving a bit more than one third!

Here are the options I added to achieve this:

--exclude-deb-section='^debug$'
--exclude='/(xen-)?linux-[a-z]+-2\.6[.0-9]*-[-[:alnum:]]*(openvz|vserver|xen)[-[:alnum:]]*_'
--exclude='(k/kde|g/gnome|o/openoffice\.org).*/.*_(armel|hppa|s390)\.deb'
--exclude='(a/axiom/|d/debian-edu-doc/|e/ember(|-media)/|e/eclipse(/|-))'
--exclude='(e/erlang|g/(gcl(cvs)?|ghc6)/|l/llvm(/|-)|p/paraview/|o/openturns/)'
--exclude='(s/scalapack(-doc)?/|f/festvox-|g/gcc-snapshot/)'
--exclude='(/acl2-books_|/digikam-doc_|/fluid-soundfont-gm_|/deal.ii-doc_)'
--exclude='(/libxmpp4r-ruby-doc_|/lilypond-doc_|/qt4-doc_|/vtk-doc_)'
--exclude='/i18n/Translation-.*\.bz2' --include='/i18n/Translation-(nl|de)\.bz2'

And the explanation is:

I rarely use debug packages and they are relatively big; if I do need one I'll download it manually from a remote mirror.
I don't run vserver or xen kernels (and if I did I'd probably compile custom kernels anyway). I do want "regular" kernels because of D-I work.
I doubt I'll ever want to install KDE, GNOME or OpenOffice on my armel, hppa or s390 boxes, but I do want them for the other three arches.
Selected individual (mostly scientific) source packages that I doubt I'll ever use but use up significant disk space (and bandwidth when updated). These were found by a simple:
```
du -s pool/main/*/* | sort -rn | head -n 50
```
Selected individual huge binary packages (mostly documentation), found using:
```
du -s pool/main/*/*/*.deb | sort -rn | head -n 50
```
I'm only interested in Dutch and German translations of package descriptions. Well, actually I'm not even interested in those, but it's useful to have them for testing debmirror.

Obviously I have nothing against any of the packages that I exclude. It's just that I don't need them.

Posted Sat Oct 3 18:09:08 2009 Tags: debian debmirror

Yay! I've done it: 1160 lines of bash script are now 1215 lines of perl, and:

    'debtree aptitude': 1m2.832s -> 0m0.596s

The new release is available as version 0.9.9 from the debtree web site and has been uploaded for the archive as version 1.0.

This was the starting position, the run time for my complete test set:

    real    22m33.583s
    user    18m29.709s
    sys     4m21.320s

I began with a pure language conversion from bash to perl, i.e. I kept the call-outs to dctrl-tools. This allowed me to easily identify problems in the language conversion by running my test suite, without having to worry that a change might have been caused by getting different data. The language conversion itself was fairly straightforward; most time was spent on finding all the little errors made during the conversion.

This resulted in "only" a 10% speedup:

    real    20m56.368s
    user    18m3.996s
    sys     2m46.986s

So bash itself isn't even horribly slower than perl, even with all the recursion and starting of subshells for calls to grep, sed, etc.

Then I replaced the call-outs to dctrl-tools one by one, adding the dependency on libapt-pkg-perl. And that resulted in the amazing:

    real    0m21.350s
    user    0m19.797s
    sys     0m1.372s

So, from 22 minutes to 21 seconds for 22 graphs, including some pretty complex ones. Not bad.

I had to keep a call-out to dctrl-tools for build dependencies as it turned out libapt-pkg-perl does not expose architecture conditions.

The full conversion process can be seen in the source repository, which was recently moved from my $HOME on alioth to collab-maint.

Posted Wed Sep 16 12:21:02 2009 Tags: debian debtree

debtree 0.8.0, including the new option to display reverse dependencies, is now officially (or rather: unofficially) available.

The new feature is of course documented in the man page, but also on the website.

And now I think the time has come to port the script to perl. If I manage that I plan to upload the package into the archive as version 1.0.

P.S. debtree now also supports generating trivial graphs:

$ debtree --max-depth=0 dpkg

Funnily enough that same graph is less trivial for apt. Support for --max-depth=0 was added to allow to generate graphs showing only reverse dependencies.

Posted Sat Sep 12 14:22:48 2009 Tags: debian debtree

I've just uploaded version 2.2 of debmirror, which introduces yet another new feature: mirroring the i18n/Translation files that contain translations of package descriptions. Many thanks to Joerg Jaspert for his quick response to my request to include those files in the Release file.

Joerg also implemented the change needed to use the diffs for Contents files but that requires a fairly big code restructuring in debmirror.

The package has jumped from version 1.0 to 2.2 in just three weeks (closing 28 bug reports in the process), but I think the changes justify that. Here's an overview.

Automatic creation and update of suite->codename symlinks (1.0)

This also means it no longer makes any difference whether you tell debmirror to mirror sid or unstable.
Option to cache the mirror state between runs (2.0)

This significantly reduces the trashing of the hard disk during mirror updates and cleanup, and improves the efficiency of individual runs.

The disk trashing has always been the main reason I did not want to do more than one update per day for my local mirror. Now it hardly matters how many runs I do: almost everything is done based on the cache data.

To ensure the mirror stays consistent the cache has a (configurable) maximum life time after which a full check of the mirror will be done, if desired including an md5sum check of all files.
Significant speed increase for parsing Packages/Sources files (2.0)

For my mirror that stage now takes seconds rather than minutes. Additional speed increases should be possible in the stage that fetches the Packages and Sources files.
Mirroring of "current" Debian Installer images (2.0)

Which architectures and suites should be mirrored can be specified independently from the rest of the mirror.
Mirroring additional files from specific directories (2.1)

This allows mirroring of "trace files", of the contents of the ./doc and ./tools directories (which are needed if you want to create CD images using debian-cd, and of the ./indices directory.

The transfer method used for this is always rsync, independent of the transfer method used for the rest of the archive. This is a restriction, but rsync is also the only usable option for files for which no real index or checksums are available.
Mirroring translation files (2.2)

As debmirror is primarily intended to be used for local, often partial, mirrors, it is of course possible to mirror only selected languages. Interested only in German and French translations? Simple, just use:
```
--i18n --exclude='/Translation-.*\.bz2$' --include='/Translation-(de|fr).*\.bz2$'
```
I've used '(de|fr).*' so that also country-specific variants (e.g. fr_FR) will be included.

If you're currently using the Lenny version of debmirror and would like to use the new features: the package from unstable can be installed on Lenny without any problems. The changes have been well tested, but I would advice to do use --dry-run after the upgrade to check there are no unexpected problems.

One area where you may experience problems is when using debmirror for other archives than the official Debian mirrors. If you do encounter issues then please file a bug report.

Note that debmirror is not intended to be used for official mirrors. There are different scripts available for that from the Debian mirror team.

Posted Sat Sep 12 08:19:51 2009 Tags: debian debmirror

Funny how working on a program immediately inspires to do more.

Remember that the initial motivation for debtree was to find out why a package was installed? It can now show that in the dependency graphs!

I'm not quite ready to do a new release, but the new version is available from the git repository.

Let's start with a simple example (all graphs are based on Lenny).

$ debtree -I --rdeps-depth=3 apt

Only installed packages are displayed here; if the -I option is omitted, debmirror will display all, but that does tend to explode the graphs, especially for common libraries. As for forward dependencies, the color of the arrows indicates Pre-Depends, Depends and Recommends.

The reverse dependencies are shown three levels deep (one is default). The graph will always include all direct reverse dependencies (both on the package itself and all virtual packages provided by it). For indirect reverse dependencies there's a cut off that is set at five by default. Example is debconf, that apparently has 9 reverse Pre-Depends and 58 reverse Depends installed on my system.

The next one is simply beautiful.

$ debtree -I --rdeps-depth=20 --no-conflicts libcairo2

Because of the --rdeps-depth=20 this shows the full recursion! I was surprised that this graph remained a reasonable size. Apparently no packages depend on the virtual package libcairo, at least none that I have installed.

The final one is extreme, and I must confess that I have cheated a bit by suppressing the least interesting reverse depends (which explains why it does not match the numbers from the apt graph).

$ debtree -I -R --no-recommends --no-conflicts debconf

The most interesting thing here is how it shows the debconf-2.0 transition. Most packages depend on 'debconf|debconf2.0'; tex-common instead has 'debconf|cdebconf', while tasksel and exim4 have both combinations (probably one explicitly in debian/control and the other added by debhelper. ucf is missing the alternative; apparently does not use debhelper (no prizes for guessing who the maintainer is :-).

Notice anything about iamerican and ibritish? Yes, they really have a double dependency on 'debconf|debconf2.0'.

The one thing missing is the version info for versioned dependecies. Not sure yet if I want to add that for reverse dependencies.

P.S. SVG versions of the images are available in the same directory as the JPGs.

Posted Tue Sep 8 14:38:19 2009 Tags: debian debtree

It's been quite some time (almost two years) since my previous "release" of debtree, but now version 0.7.3 is available. And it still generates very nice graphs :-)

The changes are relatively minor: a few nice fixes for corner cases that were not handled correctly, and an update of the default lists of "skip" and "end" packages which help to limit the size of graphs for a fair number of packages I tried (including konqueror and openoffice.org).

Reason to revisit debtree was a recent nice mail from a debtree user, but also the current discussions about udev and the FHS. I'm on the side of "let's please keep /usr mountable separately". Mostly because I like a (small) encrypted root with a separate (large) unencrypted `/usr'.

I'm also increasingly unhappy with the default size of Debian's desktop installs, especially now that it looks as if Squeeze will see installation of Recommends by default by tasksel (and thus Debian Installer).

For comparison, the size of a default Gnome desktop install for Etch was 1360MB; for Lenny it is 1830MB; for Squeeze it looks like it will be well over 3000MB! Remember that for Sarge we installed both Gnome and KDE from CD1 with both together taking 1390MB?

Sure, some of that is real functionality, but a lot is also (IMO) redundant visual effects that only serve to slow the desktop down and junk needed to do stuff automagically. And a heck of a lot is duplicated functionality. One of the main reasons I switched to Linux was because it gave me back control over my systems, but with KDE4 and pervasive stuff like hal and all the various "kits" Linux is on a fast track that's giving priority to flashiness over real functionality and eroding that control.

Here's a fairly default dependency graph for hal (click for full image). Looks reasonable, right?

But that's only because most major dependencies, such as dbus, policykit and pm-utils have been pruned. Here's a complete graph, with only libc6 omitted (full image is 1.5MB). Truly a tangled web. Scary.

One can also look at it from the other side. Today I upgraded my sid chroot and found I suddenly needed to install libavahi-client3, libavahi-common3, libavahi-common-data and libdbus-1-3. Why? Reason turned out to be libcups2, so I checked if I really needed that. And here's why I do.

Most of these dependencies of libgtk2.0-0 I can understand, but isn't gtk supposed to be a graphical toolkit library? Couldn't printing support be implemented in some more specialized Gnome printing toolkit library? But I'm probably missing something.

Anyway, now that I have a bit more perl experience through my recent work on debmirror, maybe I should finally port debtree from shell script to perl...

See the debtree home page for a full overview of how to read the graphs, but here's a quick intro. Purple arrows are Pre-Depends, blue are Depends and black are Recommends; green connections show Provides. The green packages are currently installed in my sid chroot, while the white ones are not. The diamonds show where the graph has been pruned: dependencies for these packages are not shown.

Posted Sun Sep 6 15:57:00 2009 Tags: debian debtree