OK, not the most original title. But quite apt as I really don't intend to blog very much or very seriously.
See the archive of all posts for what little is there.
If you want to comment on an entry, please do so by email. I will post updates if you send relevant stuff my way.
If you haven't seen it, try to. Amazing this movie did not get any major awards.
Maggie Gyllenhaal is well worth looking out for in other movies.Posted Mon Aug 9 05:02:56 2010
I got this neat little Chinese netbook after a mail to the debian-arm list where one machine was offered in exchange for porting Debian to it. So I offered to get Debian Installer running on it.
In total 20 of these ChiTech PC89E netbooks were bought as a group by various people in different countries. The netbook is based on an ARM S3C6410 SoC, has 256 MB RAM and an excellent 8.9" 1024x600 LCD display. The machine itself is 11" and weighs under 800 grams. Most of the weight is the LCD display; they've even had to add a small extra weight in the base to avoid it toppling over backwards.
The goal of getting D-I running was achieved last week, though not without needing to overcome some steep hurdles.
Our main problem is that we don't have the source code to either the kernel they use (184.108.40.206 with patches from Samsung and others), or the u-boot bootloader.
SSH was available without password for the user account and various basic errors allowed us to root the system relatively quickly. Brute force cracked the root password a few days later.
The only information we did have was from the provided desktop system (which is really quite good and certainly looks very polished, but in the end still way too limited), what's on the flash memory and the contents ofa "firmware upgrade" image they provided.
Having that firmware upgrade image proved very important as it gave us a good idea how to boot the system from SD card and made it possible to locate the kernel, u-boot bootloader and other files in flash memory.
Googling for some boot messages we found two kernel source trees (for a different device: SmartQ) that looked promising as a basis. After some disassembly work by Luke Kenneth Casson Leighton on the kernel to get LCD timings and correct GPIO data we managed to get a basic working kernel: LCD, USB (and thus keyboard and touchpad), and wired networking.
Although there are still loads of things that need improving/fixing, that gave me enough to work with to try to get Debian Installer going.
The main limitations for running D-I are that we cannot change the bootloader configuration and that their boot procedure does not use an initrd. But we did work out how to boot a custom kernel from an external SD card by making creative use of their "firmware upgrade" procedure.
Essentially I needed to find a way to run D-I by only booting a kernel. Piggy-backing an initrd onto a kernel is possible (using the 'bootpImage' target from the upstream kernel build system), but then the size of kernel plus initrd is limited to 4 MB and that is really too limited for a decent D-I initrd (especially if you want full i18n support).
I ended up creating a micro-initrd (only 66 kB!) which only function is to mount the external SD card, load the real D-I root initramfs (which now no longer has any real size limitation) and then run init in that.
After adding some relatively minor customizations for this netbook (such as installing the kernel from a custom repository on alioth), the installer was up and running.
As the framebuffer worked without problem for the newt frontend, I next wondered
if it would be possible to also get the graphical installer working. This should
be easier for non-x86 systems after the recent switch from DirectFB to X.Org as
backend for the graphical installer. And the image below shows it was.
I had to create an
/etc/xorg.conf to get the USB keyboard and touchpad working
(apparently auto-detection does not quite work for less standard hardware), but
that was basically it.
So now all we have to do is get all the remaining kernel issues sorted out...Posted Tue Mar 30 14:36:19 2010
Until two years or so ago I used VMware Workstation intensively for development and testing of Debian Installer, using one of the licenses generously made available to the D-I team by VMware. It was much faster than Qemu and thus very welcome.
But due to increasing problems keeping it working with new kernels on a desktop system running unstable, I switched to VirtualBox (the OSE version), which has the huge advantage of being packaged for Debian and thus automatically follows the Debian kernel packages.
That advantage has disappeared somewhat as I'm currently compiling my own kernels — I'm currently running 2.6.33 on a Lenny desktop. So far I've been able without too much trouble to find patches for the VirtualBox modules to keep them compatible with new upstream kernels.
This week I had reason to try VMware again. The Server edition this time as that has free licenses. The Server concept, where you connect to the management system and virtual machines via a web browser, is quite nice. It means I could install it on a server and keep my laptop nice and responsive.
To keep VMware separate from the rest of the system and not mix it with
software installed cleanly from Debian packages, I decided to install it
/opt. And that works beautifully. There are only a few bits installed
/etc, but that's OK. I have
/etc managed by
etckeeper anyway .
More challenging was building WMware modules to match my kernel. VMware
wants you to run their
vmware-config.pl script every time you switch
kernels and build the kernel modules on the system where it runs (which
requires the kernel headers). I wanted to build the modules as part of
my kernel build process, on a totally different machine. The same as I
already do for VirtualBox. I save the custom modules in a tarball that
has the same
uname and package version as the kernel package I build and
have a simple script that installs them correctly.
I succeeded by hacking the Makefiles (deleting most of their content) and then integrating them in my kernel build wrapper script.
The module source provided by VMware built fine with a Debian 2.6.26 kernel, but not with the 220.127.116.11 kernel on my server (of course). But luckily others had already run into the same problems and solved them, so with some searching I found the required patches (first a full updated source for 2.6.29, then patches needed for .31 and for .32). Isn't open source just wonderful?
So with that the VMware server started beautifully (well, the init script
needed to be told that module names have changed from
I could not connect to it from my laptop using SSL. Connecting without SSL
(using an instance of
iceweasel running on the server but started remotely
from my laptop using X forwarding SSH) worked fine. That turned out to be
a known issue too. After applying all the suggested workarounds in my
iceweasel configuration (disable TLS 1.0; set cache size to zero; explicitly
disable use of proxy for the server) I can now connect reliably to the VMware
Server management system.
 I had to exclude a fair number of VMware files for
etckeeper as they are
updated by VMware itself and thus really belong in
/var. Example are DCHP
The Debian FTP-masters recently changed the way gzipped meta files are
compressed in order to make them more efficient to update using the rsync
option. This was done by adding the
--rsyncable option when calling
Consequence was however that when
debmirror compressed Packages, Sources
and Contents files after updating them by applying diffs, the md5sum of the
gzipped file created by
debmirror no longer matched the md5sum listed in
the Release file (because
debmirror did not use
Result was that
debmirror would also download the full gzipped Packages,
Sources and Contents files from the parent mirror, something the diffs are
meant to avoid. Not nice.
Anyway, this has been fixed in
debmirror 2.4 which now by default also
--rsyncable when gzipping the updated meta files.
I've also uploaded a fixed version for Lenny (20070123lenny1), which should
soon be available from
proposed-updates and will be included in the next
stable point release.
For archives that also provide diffs (most archives don't have them) but do
not have rsyncable gzipped files, the default options used when calling gzip
can be overruled using the new option
--gzip-options (only in version 2.4).
Tip: if you are using the
rsync method to download files, using
--diff=none may well be more efficient now that the archive has rsyncable
gzipped meta files.
Version 2.4 also has a few other improvements and fixes. If you're currently using version 2.3.x an update to the new version is probably a good idea.Posted Sat Dec 19 22:39:04 2009
debmirror 2.3 should be hitting the mirrors about now. Main change is that
it will now use the available diffs to update Contents files, which should
give a nice bandwidth reduction for users who mirror those.
With that the option
--pdiff (for "package diff") no longer really covered
its function, so I decided to change it to
There's also a fix for mirroring archives that don't have a Release file.
Question for users
--add-dir has been marked as deprecated (for quite some time now I
suspect). I'm considering to remove it in the next release as I cannot see any
use cases for it, but it's quite possible I'm missing something and there are
still people using it. If you would like that option preserved, then please
mail me at
firstname.lastname@example.org with an explanation of why and how you
Managing the size of a local mirror
The archive has grown a lot over the past Debian releases and keeping even a
partial local mirror can require quite some disk space. Luckily
offers quite a few options to tune what is mirrored.
My own mirror covers testing and unstable 'main' for 6 architectures (i386, amd64, armel, hppa, sparc and s390), no source, no D-I images. It uses only 61G. I say "only" as that's about 33GB less than it could have been without tuning. In other words, I'm saving a bit more than one third!
Here are the options I added to achieve this:
--exclude-deb-section='^debug$' --exclude='/(xen-)?linux-[a-z]+-2\.6[.0-9]*-[-[:alnum:]]*(openvz|vserver|xen)[-[:alnum:]]*_' --exclude='(k/kde|g/gnome|o/openoffice\.org).*/.*_(armel|hppa|s390)\.deb' --exclude='(a/axiom/|d/debian-edu-doc/|e/ember(|-media)/|e/eclipse(/|-))' --exclude='(e/erlang|g/(gcl(cvs)?|ghc6)/|l/llvm(/|-)|p/paraview/|o/openturns/)' --exclude='(s/scalapack(-doc)?/|f/festvox-|g/gcc-snapshot/)' --exclude='(/acl2-books_|/digikam-doc_|/fluid-soundfont-gm_|/deal.ii-doc_)' --exclude='(/libxmpp4r-ruby-doc_|/lilypond-doc_|/qt4-doc_|/vtk-doc_)' --exclude='/i18n/Translation-.*\.bz2' --include='/i18n/Translation-(nl|de)\.bz2'
And the explanation is:
- I rarely use debug packages and they are relatively big; if I do need one I'll download it manually from a remote mirror.
- I don't run vserver or xen kernels (and if I did I'd probably compile custom kernels anyway). I do want "regular" kernels because of D-I work.
- I doubt I'll ever want to install KDE, GNOME or OpenOffice on my armel, hppa or s390 boxes, but I do want them for the other three arches.
Selected individual (mostly scientific) source packages that I doubt I'll ever use but use up significant disk space (and bandwidth when updated). These were found by a simple:
du -s pool/main/*/* | sort -rn | head -n 50
Selected individual huge binary packages (mostly documentation), found using:
du -s pool/main/*/*/*.deb | sort -rn | head -n 50
I'm only interested in Dutch and German translations of package descriptions. Well, actually I'm not even interested in those, but it's useful to have them for testing
Obviously I have nothing against any of the packages that I exclude. It's just that I don't need them.Posted Sat Oct 3 18:09:08 2009
Yay! I've done it: 1160 lines of bash script are now 1215 lines of perl, and:
'debtree aptitude': 1m2.832s -> 0m0.596s
The new release is available as version 0.9.9 from the
debtree web site and has been
uploaded for the archive as version 1.0.
This was the starting position, the run time for my complete test set:
real 22m33.583s user 18m29.709s sys 4m21.320s
I began with a pure language conversion from
perl, i.e. I kept
the call-outs to
dctrl-tools. This allowed me to easily identify problems
in the language conversion by running my test suite, without having to worry
that a change might have been caused by getting different data.
The language conversion itself was fairly straightforward; most time was
spent on finding all the little errors made during the conversion.
This resulted in "only" a 10% speedup:
real 20m56.368s user 18m3.996s sys 2m46.986s
bash itself isn't even horribly slower than
perl, even with all the
recursion and starting of subshells for calls to
Then I replaced the call-outs to
dctrl-tools one by one, adding the
libapt-pkg-perl. And that resulted in the amazing:
real 0m21.350s user 0m19.797s sys 0m1.372s
So, from 22 minutes to 21 seconds for 22 graphs, including some pretty complex ones. Not bad.
I had to keep a call-out to
dctrl-tools for build dependencies as it
does not expose architecture conditions.
The full conversion process can be seen in the source repository, which was recently moved from my $HOME on alioth to collab-maint.Posted Wed Sep 16 12:21:02 2009
The new feature is of course documented in the man page, but also on the website.
And now I think the time has come to port the script to
perl. If I manage
that I plan to upload the package into the archive as version 1.0.
debtree now also supports generating trivial graphs:
$ debtree --max-depth=0 dpkg
Funnily enough that same graph is less trivial for
apt. Support for
--max-depth=0 was added to allow to generate graphs showing only
I've just uploaded version 2.2 of
debmirror, which introduces yet another
new feature: mirroring the
i18n/Translation files that contain translations
of package descriptions. Many thanks to Joerg Jaspert for his quick response
to my request to include those files in the
Joerg also implemented the change needed to use the diffs for
but that requires a fairly big code restructuring in
The package has jumped from version 1.0 to 2.2 in just three weeks (closing 28 bug reports in the process), but I think the changes justify that. Here's an overview.
Automatic creation and update of
This also means it no longer makes any difference whether you tell
Option to cache the mirror state between runs (2.0)
This significantly reduces the trashing of the hard disk during mirror updates and cleanup, and improves the efficiency of individual runs.
The disk trashing has always been the main reason I did not want to do more than one update per day for my local mirror. Now it hardly matters how many runs I do: almost everything is done based on the cache data.
To ensure the mirror stays consistent the cache has a (configurable) maximum life time after which a full check of the mirror will be done, if desired including an md5sum check of all files.
Significant speed increase for parsing
For my mirror that stage now takes seconds rather than minutes. Additional speed increases should be possible in the stage that fetches the
Mirroring of "current" Debian Installer images (2.0)
Which architectures and suites should be mirrored can be specified independently from the rest of the mirror.
Mirroring additional files from specific directories (2.1)
This allows mirroring of "trace files", of the contents of the
./toolsdirectories (which are needed if you want to create CD images using
debian-cd, and of the
The transfer method used for this is always
rsync, independent of the transfer method used for the rest of the archive. This is a restriction, but
rsyncis also the only usable option for files for which no real index or checksums are available.
Mirroring translation files (2.2)
debmirroris primarily intended to be used for local, often partial, mirrors, it is of course possible to mirror only selected languages. Interested only in German and French translations? Simple, just use:
--i18n --exclude='/Translation-.*\.bz2$' --include='/Translation-(de|fr).*\.bz2$'
I've used '
(de|fr).*' so that also country-specific variants (e.g.
fr_FR) will be included.
If you're currently using the Lenny version of
debmirror and would like to
use the new features: the package from unstable can be installed on Lenny
without any problems. The changes have been well tested, but I would advice to
--dry-run after the upgrade to check there are no unexpected problems.
One area where you may experience problems is when using
other archives than the official Debian mirrors. If you do encounter issues
then please file a bug report.
debmirror is not intended to be used for official mirrors.
There are different scripts
available for that from the Debian mirror team.
Funny how working on a program immediately inspires to do more.
Remember that the initial motivation for debtree was to find out why a package was installed? It can now show that in the dependency graphs!
I'm not quite ready to do a new release, but the new version is available from the git repository.
Let's start with a simple example (all graphs are based on Lenny).
$ debtree -I --rdeps-depth=3 apt
Only installed packages are displayed here; if the
-I option is omitted,
debmirror will display all, but that does tend to explode the graphs,
especially for common libraries. As for forward dependencies, the color of
the arrows indicates
The reverse dependencies are shown three levels deep (one is default).
The graph will always include all direct reverse dependencies (both on the
package itself and all virtual packages provided by it). For indirect reverse
dependencies there's a cut off that is set at five by default. Example is
debconf, that apparently has 9 reverse
Pre-Depends and 58 reverse
installed on my system.
The next one is simply beautiful.
$ debtree -I --rdeps-depth=20 --no-conflicts libcairo2
Because of the
--rdeps-depth=20 this shows the full recursion! I was
surprised that this graph remained a reasonable size. Apparently no packages
depend on the virtual package
libcairo, at least none that I have installed.
The final one is extreme, and I must confess that I have cheated a bit by
suppressing the least interesting reverse depends (which explains why it does not
match the numbers from the
$ debtree -I -R --no-recommends --no-conflicts debconf
The most interesting thing here is how it shows the
Most packages depend on '
tex-common instead has
exim4 have both combinations
(probably one explicitly in
debian/control and the other added by
ucf is missing the alternative; apparently does not use
prizes for guessing who the maintainer is :-).
Notice anything about
ibritish? Yes, they really have a
double dependency on '
The one thing missing is the version info for versioned dependecies. Not sure yet if I want to add that for reverse dependencies.
P.S. SVG versions of the images are available in the same directory as the JPGs.Posted Tue Sep 8 14:38:19 2009
It's been quite some time (almost two years) since my previous "release"
debtree, but now version 0.7.3 is
And it still generates very nice graphs
The changes are relatively minor: a few nice fixes for corner cases that
were not handled correctly, and an update of the default lists of "skip"
and "end" packages which help to limit the size of graphs for a fair number
of packages I tried (including
Reason to revisit debtree was a recent nice mail from a debtree user, but
also the current discussions about udev and the
FHS. I'm on
the side of "let's please keep
/usr mountable separately". Mostly because
I like a (small) encrypted root with a separate (large) unencrypted `/usr'.
I'm also increasingly unhappy with the default size of Debian's desktop
installs, especially now that it looks as if Squeeze will see installation
of Recommends by default by
tasksel (and thus Debian Installer).
For comparison, the size of a default Gnome desktop install for Etch was 1360MB; for Lenny it is 1830MB; for Squeeze it looks like it will be well over 3000MB! Remember that for Sarge we installed both Gnome and KDE from CD1 with both together taking 1390MB?
Sure, some of that is real functionality, but a lot is also (IMO) redundant
visual effects that only serve to slow the desktop down and junk needed to
do stuff automagically. And a heck of a lot is duplicated functionality.
One of the main reasons I switched to Linux was because it gave me back control
over my systems, but with KDE4 and pervasive stuff like
hal and all the
various "kits" Linux is on a fast track that's giving priority to flashiness
over real functionality and eroding that control.
Here's a fairly default dependency graph for
hal (click for full image).
Looks reasonable, right?
But that's only because most major dependencies, such as
pm-utils have been pruned. Here's a complete graph, with only
omitted (full image is 1.5MB). Truly a tangled web. Scary.
One can also look at it from the other side. Today I upgraded my sid chroot
and found I suddenly needed to install
libdbus-1-3. Why? Reason turned out to be
libcups2, so I checked if I really needed that. And here's why I do.
Most of these dependencies of
libgtk2.0-0 I can understand, but isn't gtk
supposed to be a graphical toolkit library? Couldn't printing support be
implemented in some more specialized Gnome printing toolkit library?
But I'm probably missing something.
debtree home page for a
full overview of how to read the graphs, but here's a quick intro.
Purple arrows are Pre-Depends, blue are Depends and black are Recommends;
green connections show Provides. The green packages are currently installed
in my sid chroot, while the white ones are not. The diamonds show where the
graph has been pruned: dependencies for these packages are not shown.