About
Steve's blog,
The Words of the Sledge
steve@einval.com

Subscribe
Subscribe to the RSS feed.

Links

Friends

JTE^H^Higit 1.14 released

I've added a script "jigit" into the JTE package, and renamed the package jigit too. The Ubuntu people will hopefully be testing jigit soon to help them and their users reduce the bandwidth used by CD downloads. It's another wrapper around mkimage and wget, similar to the way the jigdo-lite wrapper program uses wget and jigdo-file.

There are only some small changes otherwise. The main change is a bugfix in the rsyncsum code. People have been complaining lately that jigdo-lite wasn't happy with the templates produced by JTE 1.12 on gluck. Oops - my fault; a silly bug in the code meant I was simply checksumming the wrong data. :-(

Usual place for download: https://www.einval.com/~steve/software/JTE/. I've also uploaded jigit 1.14 into Incoming, so it will make it into the official Debian archive now I've finally packaged it up.

NOTE: the JTE patch to mkisofs needs to be applied AFTER debian patches have been applied during the package build process, as that's easiest for people who want to use the rest of the Debian-applied patches too.

02:15 :: # :: /debian/JTE :: 0 comments

JTE 1.12 released

Again, I think this is the last release I need to make. Of course! I've tested the patches against cdrtools and debian-cd and I've built several jigdo images that jigdo-mirror can rebuild into ISOs. It works.

Quite a few changes since 1.10:

• Added rsync64 sums that Richard asked for
• Added jigit "friendly" wrapper script around mkimage
• mkimage will now do some of the work needed to rebuild images where we are missing parts
• Improved docs
• Updated and fixed the patches for both mkisofs and debian-cd

Manty has been testing on gluck, and things are looking promising now. The next CD/DVD creation runs are probably going to use JTE rather than jigdo. My work here is done... :-)

Usual place for download: https://www.einval.com/~steve/software/JTE/

NOTE: the JTE patch to mkisofs needs to be applied AFTER debian patches have been applied during the package build process, as that's easiest for people who want to use the rest of the Debian-applied patches too.

01:14 :: # :: /debian/JTE :: 0 comments

JTE 1.10 released

I've streamlined the MD5 code further, and improved the mirror-check capability with a new -jigdo-force-md5 option.

I hope that's it now; at least I don't expect to make any feature changes before sarge is released. Afterwards, I'm going to extend the jigdo .template format and add bzip2 support; that will need changes in jigdo as well, so it will wait a while.

Usual place for download: https://www.einval.com/~steve/software/JTE/

NOTE: the JTE patch to mkisofs needs to be applied AFTER debian patches have been applied during the package build process, as that's easiest for people who want to use the rest of the Debian-applied patches too.

21:56 :: # :: /debian/JTE :: 0 comments

JTE 1.9 released

I've fixed a silly bug found by Manty when creating HFS hybrid images - the template code interacted badly with the HFS data and sizes were calculated wrongly. It would be nice if jigdo would actually print useful error messages rather than simply "Invalid template data - corrupted file?" in all the places where it checks the template file. Some context helps!

I've also cheated a little to cope with boot files being moved from their normal path (as on the mirror) to /install. I already have MD5 sums supplied up front for those files in the mirror, so when looking for files to jigdo encode I can match up checksums very cheaply and replace the /install path with the real one.

In my testing for JTE 1.9, I'm still creating template files orders of magnitude faster than jigdo-file. But now JTE is also producing smaller template files too. The following sizes are for i386 DVD#1:

sledge:/mirror$ ls -alU OLD.1/sarge-i386-1.* NEW.1/sarge-i386-1.*
-rw-r--r--  1 root root  159395 Aug 27 03:40 OLD.1/sarge-i386-1.jigdo
-rw-r--r--  1 root root 6357306 Aug 27 03:41 OLD.1/sarge-i386-1.template
-rw-r--r--  1 root root  103924 Aug 29 04:54 NEW.1/sarge-i386-1.jigdo
-rw-r--r--  1 root root 3375212 Aug 29 04:54 NEW.1/sarge-i386-1.template

Usual place for download: https://www.einval.com/~steve/software/JTE/

NOTE: the JTE patch to mkisofs needs to be applied AFTER debian patches have been applied during the package build process, as that's easiest for people who want to use the rest of the Debian-applied patches too.

02:12 :: # :: /debian/JTE :: 0 comments

JTE 1.7 released

I've fixed a couple of bugs found by Manty when not creating template files, updated the mkisofs man page and added docs about the new $ARCH-boot support. I've also added a patch against debian-cd HEAD - use this OR the jte_support branch in CVS if you want to use JTE to create debian jigdos.

I was hoping Joerg would take at least the boot patches into the upstream cdrtools package, but his licensing flamewar means it will probably not happen now. I'll have to talk to the various distro maintainers about taking them instead.

Usual place for download: https://www.einval.com/~steve/software/JTE/

NOTE: the JTE patch to mkisofs needs to be applied AFTER debian patches have been applied during the package build process, as that's easiest for people who want to use the rest of the Debian-applied patches too.

17:13 :: # :: /debian/JTE :: 0 comments

Joerg will take some of JTE, he says...

Yay!

I wasn't expecting it, but Joerg Schilling the upstream author for cdrtools (aka mkisofs, cdrecord etc.) and I have been discussing some of the patches I've written for mkisofs. He's not convinced by the JTE core code itself, but is quite interested in the bootable CD code I've added for alpha, hppa, mips and mipsel. I'm tweaking the source now to fit his (horrible, old-fashioned) coding style, and writing some docs to explain what it does.

Even if Joerg doesn't want to take the core JTE stuff, anything I can get upstream to take is a bonus - it's less stuff I have to maintain later...

13:23 :: # :: /debian/JTE :: 0 comments

JTE 1.6 released

JTE is nearing completion, I think. I've added support for checking MD5 sums during mkisofs runs, to remove the need for the very time-consuming mirror-check step.

I've branched debian-cd in CVS to add support for JTE. I considered making the changes optional by config, but the existing Makefile in debian-cd was already getting way too complex and brittle. Hopefully I'll be able to merge some of these changes in soon once I can convince more people to test them. Check out the jte_support branch.

Phil Hands helped me test the iso-image.pl script from JTE 1.5, and started working on packaging JTE. Hopefully if I prod him I'll even be abe to get him to write the docs for me too... *grin* In the long term, I'm hoping to get the JTE patch accepted by the Debian mkisofs maintainer. Failing that, I'll end up maintaining a forked version of mkisofs. I'd obviously like to avoid that if I can.

See https://www.einval.com/~steve/software/JTE/ for docs and downloads.

02:23 :: # :: /debian/JTE :: 0 comments

MD5 considered harmful

The CD/DVD creation process in debian-cd is very very slow for two reasons:

1. the need to read and write CD- and DVD-sized lumps of data
2. checksumming that data over and over and over...

The first part is kind of unavoidable - to be able to make an ISO image, you have to actually read in all the data that will go into that image, and then write it out. To make this go faster, you simply need to supply good disk hardware - there's not really much that can be done algorithmically.

The second part is the bit we can do something about. At the moment, the CD creation process includes:

1. mirror check - check the MD5 sums of the files we're going to use against the Packages and Sources files
2. calculate what will fit on each disk, and lay the files out
3. apt-ftparchive - create the Packages/Sources files to go on each disk
4. Bootable - set up the necessary magic to make a CD/DVD bootable, if applicable/possible/necessary
5. md5sum.txt - list the checksum of every file on each disk in a file in the root directory of that disk
6. make the image file
7. generate jigdo files - compress the images by working out which files make up the image and replace those portions with file references instead

In reverse order:

Steps 6 & 7: I've already written JTE to make step 7 much faster: generate the jigdo files directly from mkisofs while we still have all the data we need (paths to each file), instead of having to work back from the image by brute force. This makes step 6 slightly slower, but the cost of md5summing data we're already reading and writing is not too bad.

Step 5: Phil Hands has modified debian-cd to use fast_sums to generate the md5sum.txt file. It uses the pre-caclculated md5 sums from the mirror, rather than reading all the packages again.

Step 4: Making disks bootable is normally trivial and take almost no time, so it can be ignored

Step 3: apt-ftparchive currently generates all the md5sums from all the files it will place into a Packages file.

Step 2: working out what files will fit where and creating the CD trees is also reasonably quick these days. Even "copying" the data into place is fast, as we can simply create trees of hard links rather than actually copy the data.

Step 1: the mirror check is the next thing I'm looking at for a performance gain. It's necessary for release builds, to make sure that the packages and sources that go on the CDs and DVDs exactly match what's on ftp-master.debian.org and haven't been corrupted in transit. However, this step takes a long time, so long that many people disable it when running debian-cd.

What I've done is to move the md5 check to later in the process. My JTE patch already pushes steps 6 and 7 together into one stage and also calculates md5 sums as it goes. The obvious change to make is to check the files at that point. Instead of checking the mirror up-front, simply build a list of files and md5sums and feed that to mkisofs so it can do the work, almost for free. If any files fail to match when we're building the image, fail at that point. I've written support for this, and it will be in JTE 1.6, coming Real Soon Now (TM).

I'm not sure of how to progress JTE further - it clearly needs packaging, but that will probably involve forking mkisofs. Joerg is infamously difficult to please in terms of accepting patches for cdrtools, and the current mkisofs maintainers haven't responded to my mails about JTE AT ALL

In other news, I'm about to commit a debian-cd change to fix the problem I've been seeing of HFS hybrid discs (powerpc and m68k) being too big.

15:32 :: # :: /debian/JTE :: 1 comment

JTE 1.5 released

I've added another tool to the JTE package: a CGI wrapper called iso-image.pl. Now mkimage can be set up on a debian mirror to allow on-the-fly creation of ISO images from jigdo files.

See https://www.einval.com/~steve/software/JTE/ for docs and downloads.

I'm still waiting on a response on the outsized HFS hybrid images problem. Surely I can't be the only one to have seen this?

19:44 :: # :: /debian/JTE :: 0 comments

JTE 1.4 released

I've added a mkimage tool. See https://www.einval.com/~steve/software/JTE/ for docs and downloads. I'm basically finished on the debian-cd patches to use JTE, and I'm testing them extensively now.

I also need to hack debian-cd to not produce oversized images due to HFS hybrid information on powerpc and m68k.

00:51 :: # :: /debian/JTE :: 0 comments