Tukaani logo

Feature details of the Tukaani pkgtools

I (Larhzu) felt that just listing the changes made in the pkgtools wouldn't make it clear why exactly all the changes had been done. In this document I try to describe in detail the most important and interesting changes. In addition, there are bits of the development process history. Understanding this document requires basic knownledge of the package management system of Slackware Linux.

Support for new compression formats

Traditionally Slackware packages have been tar files compressed with gzip. They are easy to handle with standard tools and gzip is very fast in both compressing and decompressing. However, the compression ratio of gzip is not very high compared to some advanced tools available nowadays. Compressing the packages into a smaller space is an essential part of shrinking Slackware back to a one compact disc; that's why we decided to add support for other compression programs to pkgtools.

At this point it's probably good to make it clear that even if the Tukaani pkgtools support different compression formats the tar archive being compressed and its format has been kept intact. In practice that means that you can convert between different compression formats without touching the tar file which holds the actual uncompressed package contents.

At first the natural choice was bzip2. It's widely used and bundled with all modern GNU/*/Linux distributions. Unfortunately bzip2 is quite slow in decompressing. That makes installing bzip2 compressed packages much slower than their gzip compressed counterparts. Despite the speed optimizations made into pkgtools, we felt it wasn't fast enough to be the primary choice to compress packages.

We looked for alternatives and found LZMA. It's the default compression format used in 7-zip. There is a port of 7-zip to *nix systems called p7zip. However, in the beginning of the year 2005 the 4.xx versions of 7-zip and p7zip were beta versions. They also contain not only a stream compressor but an archiver and even support for other compression formats. So, using a beta software with tons of unneeded features didn't sound a good idea. Luckily, the separate LZMA compression program was already stable, and the code from the LZMA SDK compiled cleanly on Linux (and also *BSDs).

The LZMA format appeared to be excellent for a package management system. In decompression it's only little slower than gzip and is definitely many times faster than bzip2. LZMA compressed packages are 30% smaller on average. In practice this means that what takes a gigabyte as gzipped can be now fitted in a 700 MB CD-ROM.

For more informatation about LZMA related software used in Tukaani see LZMA utils.

Network support

In the beginning of the Tukaani project there were no other changes planned into pkgtools than support for better compression than gzip. Things changed because I was getting more requests to add FTP and HTTP support to the installation program of Tukaani Linux. At first using ftpfs kernel module sounded like an easy choice, but soon it was clear that adding network support into pkgtools using wget would be a lot better choice for many reasons.

At first the network support was added to installpkg and upgradepkg. The response from users was very positive, so I decided to extend the idea. The result was a revised 'pkgtool' having support for multiple package repositories via both command line and dialog based menus. The feedback "why other distros don't have this simple and easy package manager" confirmed that Tukaani pkgtools were going in the right direction.

The first stable release, 1.0.0, doesn't support verifying signatures using the GNU Privacy Guard (GPG). The next versions will add support for GPG, and after that we think it is reasonable to expect pkgtool to be a superb alternative to other package tools with network support made for Slackware. The time will show the truth. ;-)

Improving makepkg

Creating packages using makepkg has in practice required root priviledges to get all ownerships correct. Also file and directory permissions seem to be a common problem – sometimes even The Man releases packages having e.g. some files in /usr/doc being not world readable. Everyone makes mistakes since human nature is known to be error prone.

Optimizing the common case is often the best way to good results. Most packages have only files owned by root:root and files and directories world readable. We made two changes:

  1. Tukaani makepkg calls tar automatically with parameters --owner=root --group=root, if makepkg is run as non-root a user, and
  2. the makepkg option --chown now changes not only permissions of directories but also all the files in the package tree, setting files with executable bit to "chmod 0755" and others to "chmod 0644". Even if this feature makes creating most packages easy, think twice before using it; first doublecheck that there really is no need for any special permissions and then, if that is the case, use it.

Packager veterans reading this have probably already noticed a problem with the new behavior of makepkg --chown y: all the Slackware packages have files in directories bin and sbin owned by group bin. This is a leftover from the early days of Slackware and has been already abandoned by many other distributions. The comments in the Pat's glibc.SlackBuild clarify things quite a bit. In Tukaani Linux distribution we no longer require that files in the directories bin and sbin are owned by group bin. Creating packages having a few files owned by root:bin and the rest by root:root adds unneeded complexity and would make the new behavior of makepkg --chown y useless, when it was designed to prevent common mistakes.

Note that all the .tgz packages of Tukaani intended for users of Slackware still follow the Slackware standard having the files in the directories bin and sbin (and the directories itself) owned by the group bin.

There are also some minor changes for better usability. As you know, if run without parameters, makepkg prompts if it should put symlinks to doinst.sh script and if it should touch file permissions. Usually, you want to have doinst.sh created and permissions kept intact. With Tukaani makepkg you can just press enter on those prompts instead of typing y and n. That is clearly shown within prompts in a similar way that Linux's make config shows: [Y/n] or [y/N].

Slackware makepkg scans the package tree for empty files and empty gzipped files and lists them if found. In my opinion this simple feature is a great piece of functionality. Inspired by this I added a similar check (and a warning) for missing package description (slack-desc file). As a last step makepkg prints the package basename, version number, arch and build version each on a new line. I have a few times seen packages named like foo-0.12-rc3-i486-1barney.tgz which is obviously incorrect (should be e.g. 0.12_rc3). Hopefully seeing the package name split in fields prevents this mistake.

Fixing some minor bugs

At first it's worth pointing out that the Slackware pkgtools are never meant to be perfect and usually do their job very well. The purporse of this section is not to bash Slackware or its pkgtools. I have seen only few questions relating to unexpected behavior of the pkgtools caused by the bugs described below. That's why I think these bugs are only minor or very minor, and the big success of Slackware Linux supports that. Anyway, I have personally encountered all these in real life and found them irritating and definitely worth fixing.

Some problems cannot be fixed trivially by editing just one or two lines of code. For example, most strings are used unquoted. This prevents path names given on the command line to contain spaces. Commands like

installpkg "my packages/foo-0.12-i486-1barney.tgz"

give only misleading error messages about missing files "my" and "packges/foo-0.12-i486-1barney.tgz".

If you have ever used upgradepkg when the replacing package has been corrupted (e.g. truncated on downloading), you will have noticed that it doesn't correctly detect that the package is broken. This does not apply to installpkg, which correctly refuses to install a corrupted package file. A similar problem appears with upgradepkg, when the new package file exists but cannot be read. This can happen when the replacing package is on a NFS mounted directory. Looking the problem more closely reveals that this is actually a bug in GNU bash. (Also "ash" has this bug, "zsh" works correctly. And yes, I know the if-clause in the bug report has "then" missing.) Both of these bugs lead to the same result: instead of getting upgraded the package gets completely removed in the same way as if you had removed it using removepkg command. Both of the bugs can be also fixed simply by checking the package integrity before trying to upgrade it.

The way how symlinks are stored in the doinst.sh script has a fundamental limitation preventing special characters like spaces, dollar signs, and quotation marks being used in symlink paths, names, or targets they point to. Trying to create a package using makepkg from a tree having these kind of symlinks results a borken doinst.sh script. Tukaani makepkg quotes the special names correctly, thus making it possible to have special characters in symlinks *). For the best compatiblity with Slackware, only symlinks that really need quoting are quoted; practically with almost all packages, Tukaani makepkg creates identical doinst.sh scripts that Slackware makepkg would create. The new style quoted symlinks are incompatible with Slackware removepkg which is unable to extract them from doinst.sh and thus won't delete the symlinks.

*) Actually newline character in file or symlink name still breaks pkgtools in a few places. However I expect spaces, dollar signs, quotes etc. being more common than newlines in filenames so this bug shouldn't be a problem at all. ;-)

Because of symlinks, two different file or path names can point to the same file or directory. removepkg doesn't check if two distinct entries in the package database actually point to the same target. For example, if package "foo" has file usr/lib/qt/lib/libfoo.a and package "bar" has file usr/lib/qt-3.3.4/lib/libfoo.a, removing either "foo" or "bar" will always remove libfoo.a because "qt" is a symlink pointing to "qt-3.3.4". Unfortunately, this bug isn't easy to fix without making removepkg very slow. This bug is not fixed (at least not yet) in Tukaani pkgtools. As far as I know, you cannot encounter this bug on a system that has only official Slackware packages installed. The bug should still be fixed in future unless it's impossible without unreasonable speed penalty.

Reliable and fast

It's important that package manager doesn't do anything behind your back. Far too many times we have seen on IRC how people have upgraded "something" and then sound/X/mouse/whatever stopped working. Users tend to blame the package manager, although those problems aren't always the automated package management tool's fault, but people lacking the knownledge of how Slackware package management really works (some users expect the tool to do all the work for them). From this point of view, if you dare to use wget or rsync to download the packages and then install them with installpkg or upgradepkg, you should dare to use the network features of the Tukaani pkgtools.

Tukaani pkgtools remind you to check your boot manager (e.g. run lilo) after upgrading a kernel package, show the list of *.new files and a few other informative messages meant to freshen the administrator's memory. But all those extra things are only informative; like the original pkgtools from Slackware, Tukaani pkgtools don't run lilo for you or play Russian roulette with package dependencies.

I still actively use i586 computers (100 to 200 MHz, 32 to 64 MB RAM). Before Tukaani pkgtools I used package utilities written for Slackware that had network support. Unfortunately, some of them were very slow on older computers, e.g. updating the package database could take over two minutes excluding the download time! I didn't accept that and decided to spend some time making Tukaani pkgtools fast. I think I succeeded pretty well; measured on the same computer updating package database with Tukaani pkgtool takes about five seconds. :-)

Not only network related code is fast. The code of installpkg and upgradepkg is optimized and cleaned. When upgrading a package using upgradepkg of Slackware 10.1, the file packagename.tgz was decompressed four or six times depending on if packagename.txt containing the package description was available or not. With Tukaani upgradepkg, decompressing is always done three times. You may wonder why decompress more than once. There are good reasons for that, but it's not ruled out to create a version of Tukaani pkgtools that decompress the package file only once. In any case, the minor slowdown from decompressing LZMA compressed packages is well compensated with the speed optimizations made into pkgtools.

Slackware powerusers remember Jim Hawkins' patch for pkgtool that was included in Slackware Current before Slackware 10.1 was released. The patch dramatically improved the speed of the View command. Unfortunately it had a few small bugs and the fast version wasn't included in Slackware 10.1. Those bugs got fixed, and the working version of the patch is included in Tukaani pkgtools. Opening View or Remove menu no longer takes dozens of seconds.

Other improvements

Some scripts had command line options beginning with one dash and some with a doubledash. For consistency, in Tukaani pkgtool all the long options begin with a doubledash. There are also short equivalents available, which begin with one dash and can be grouped together. Thanks to the flexibility of GNU getopt, command line options and package names can be specified in any order, i.e. options can follow package names.

The option --warn of installpkg previously showed a full file list and a list of symlinks extracted from doinst.sh script. The problem was that it didn't tell which files would be overwritten in reality. Tukaani installpkg checks which files actually exist on the filesystem and produce a list of conflicting files based on that.

Keep it simple

Slackware philosophy has been to keep things simple which is, in our opinion, one of the greatest strengths of Slackware Linux. Adding this big number of new features into package management tools naturally increases complexity. On the other hand the code of the pkgtools has been cleaned up notably and some bugs have been fixed. The result is more readable and easier to maintain.

Still, don't take this wrong: Slackware pkgtools have done their job excellently. I could have not written them from scratch. That's mostly not because of the actual code but the idea and the design of the package system itself: it's simple and elegant which just works. Go and ask any Slackware user and they confirm that.

pkgtools tukaani_1.2

Getting the next stable release out took much longer than originally planned, mostly because I liked to concentrate on other parts of the project. The new version provides more speed and a few features to ease admin's work.

pkgtool now checks MD5 sums if they are available in the repositry. Verification of GPG signatures can be turned on per repository basis. installpkg and upgradepkg check GPG signatures too, and the default GPG settings for them can be set in /etc/pkgtools/config.

pkgtool will now upgrade glibc-solibs as the very first package, if it has been selected. This should prevent accidents which could make newer packages (such as bash) unable to run on the old glibc. Naturally this wouldn't happen if users would read the upgrade documentation ;-) or if the package management included dependency checking. The former is hard to fix and the latter won't be implemented to pkgtools.

After installing/upgrading the requested packages, you will be shown the list of new *.new files. It reminds the admin to check them out, which is especially important in with a sysvinit package. pkgtools will now preserve ownership and permission information of existing directories when installing/upgrading packages. This helps a little if a package with invalid permissions is installed. This feature could be useful with files too, especially with setuid binaries, but implementing it to pkgtools is not planned.

Last but not least, pkgtools tukaani_1.2 adds support for LZMA utils ≥4.30.

Future

It was January 2005 when I took the first look at pkgtools' internals and thought to add support for other compression formats than gzip. Since then I have learned many things about sh/bash and sed scripting. 14 months later pkgtools has got network support and a menu based interface to browse package repositories, and many other features. It is a major part of Tukaani Installer too.

Unfortunately pkgtools have a few fundamental issues which cannot be fixed without breaking Slackware compatibility. Keeping pkgtools compatible with Slackware has been the focus of development from start, because that way the Tukaani project can be useful to wider range of users.

One problem is that pkgtools allow identically named files to appear in more than one installed package. This is not a bug in pkgtools; it's a feature which has been intentionally included (it is essential e.g. for aaa_elflibs package). The negative side is that installing or upgrading a package can overwrite newer file versions. Removing a package gets more complex and slower, because it requires scanning through the whole database of installed packages for duplicate filenames.

Some other issues include the way symlinks are stored in packages. For historical reasons, symlinks are not stored like the other files are in the tar archive, but created by the postinstall script (doinst.sh). This adds extra complexity here and there in pkgtools. Many packages would not have a postinstall script at all if symlinks had been handled like other files. It has also a notable effect on speed, since creating lots of symlinks this way can easily take many seconds even on a relatively fast box (1-2 GHz). The version 1.2 has a speed optimization for symlink creation, but it's still far from optimal.

I want to get rid of the limitations of pkgtools. Because it cannot be made without breaking Slackware compatiblity, I rather see the whole pkgtools replaced with something better:

As you can now guess, my part with developing pkgtools is getting to its end. I'm not planning to add any new features to pkgtools, but I will keep fixing possible bugs. There are many nice features which I've been asked to code, but now I have to say that most of them won't be implemented by me. I'm hoping to find someone who is willing to maintain the Tukaani pkgtools, because I think pkgtools will have their place in a few Slackware derived distributions.

When the time is ready, the Tukaani Linux distribution will move to UniPKG. If you happen to have a feeling "oh no, not that package manager", don't worry! We are not going to switch before everything really is ready for it. The transition should be easy, since UniPKG will get read-write support for Slackware package database. Wrapper commands will allow users to use UniPKG in the same way as they used pkgtools. The package manager part of UniPKG does no dependency checking; the download tool can optionally do that (roughly similar to pkgtools and slapt-get/swaret). As a bonus there will be no need to first rpm2tgz a foreign package before installing it; UniPKG supports many package formats out of the box. However, support for multiple package formats is not the primary reason to go with UniPKG even when it is a nice extra feature.