[Starkit] Improving Tcl compression and performance

[Jean-Claude Wippler]

Guaca Mole wrote:

> So, if I take /usr/lib in my Linux system it is 622Mb
> If I do tar cvfz lib.tgz /usr/lib
> it takes 9 minutes and its size is 220Mb
>
> If I pack it in the starkit it takes 50 minutes and
> its size is 405Mb

Not bad!

You're comparing C code versus scripted code and single compression 
versus per-file compression.

It's unlikely that speed can be improved substantially without a major 
rewrite of the Mk4vfs driver and the compression logic used (there's 
more copying going on than strictly needed).  I'd be interested to hear 
about acces/unpack performance comparisons, if you have any.

W.r.t size the comparison is not quite fair: starkits (and zip 
archives) support instant access to individual files, tar/gzip does 
not.  If you want a more meaningful comparison, you should compare 
starkits to zip archives.  I wouldn't be surprised if starkits can 
open+fetch a single file faster than zip archives, btw - that's where 
the database should kick in - but then again I've never done any 
measurements.

Having said that, it may well be that your starkit contains free space 
from multiple commits made while it is being created.  I've just added 
a new subcommand to "sdx" to pack any Metakit datafile optimally (hence 
also starkits/starpacks):
	sdx mkpack <infile> <outfile>
Only light testing so far, but it's relatively simple code so it should 
be ok.

On a similar note, I recently also added an "sdx mksplit" command which 
splits starkits and starpacks in two, i.e. a head (script or exe) and a 
tail (Metakit data).  Can be of use to splice and muck with starkits 
and starpacks at times.

You can get the new SDX as usual, the home page is 
http://www.equi4.com/sdx.html

-jcw