PNG Rant

I've been running wallpapers I find online through pngout for quite some time now and am thoroughly impressed with the results. I unleash it upon /usr/share/pixmaps/*png yearly as well. How can one not get satisfaction out of making a file smaller and losing nothing in the process? This habit has left me a little shocked with the results I've been gathering. The subjective observation I'm left with is that over 90% of the images I find lying around can be recompressed losslessly with a fairly sizable saving. It should be standard procedure for a publish-once, subscribed-many medium to ensure optimal size to save on bandwidth, but this just isn't happening....

Quickly, pngout is a closed-source png written by Ken Silverman (yep, of Ken's Labyrinths fame) recompressor which pretty much always beats optipng -o7, pngcrush and advpng in terms of filesize.

I don't use browser disk caches so I searched my homedir for a cache of PNGs. It turns out the only place I hadn't unleashed pngout on was my .themes directory. There weren't too many files in there but I only have a few minutes to whip this together so someday I'll post more convincing results, but for the moment here's what I have...

299 files. Before pngout, took 468473 bytes of disk space, after pngout, took 164470 bytes. 304003 bytes saved. (64% savings) If you want to duplicate these tests, here's my terrible sample data. (Compressed paq8o2 tarball)

Multiply every unoptimized png downloaded from the web by every uncached browser hit it receives and realize how big of a deal this is...

months and months of PAQ

Exciting stuff... PAQ8M has narrowed the gap with StuffIt in terms of lossless JPEG compression. For those unfamiliar with PAQ compression, I highly recommend checking it out. For the uninitiated you're probably best off checkout out LPAQ1 on your non-JPEG data first as it can run in a sane amount of time.

WRT PAQ XML Compression

While brainstorming ideas for testing the metascheduler I'm building I thought to look into data compressors again... Specifically the PAQ family of compressors. The latest update is PAQ8jc (fixed tarball). I whipped up an ebuild and took it town using Intel's C++ library. I tested it out on a 1.8M XML file:

   reference   1.8M
   gzip -9     168K
   bzip -9     108K
   PAQ8jc -5   61K
   PAQ8jc -7   61K (2 bytes smaller, but longer runtime/memusage)
   

Okay... So this shows that if I feel like getting my hands dirty with C++, there's actually some value in parallelizing this algorithm.

Something that caught my eye while looking into this is XML-WRT. It's a fantastic project which scratches an itch I developped in the middle of a lecture on WebServices some time ago. XML-WRT can be thought to work in two distinct steps; substitute common tagnames, attributes etc with shortened tokens; run result through zlib or FastPAQ depending on user preference. I tested its WRTified zlib/fastpaq targets on a the 15M Locations.xml file from gnome-applets (wow that's big):

   reference   15M 
   gzip -9     2.0M
   bzip2 -9    1.2M
   xml-wrt -2  1.8M (zlib default after wrt)
   xml-wrt -3  1.7M (zlib best after wrt)
   xml-wrt -10 693K (FastPAQ normal)
   xml-wrt -11 693K (FastPAQ best)
   

I also tested it on a 684M XML database (the default buffer size is too small for dictionary generation on this particular file):

reference           684M
gzip -9             102M
bzip2 -9             74M
xml-wrt -l10        ----
xml-wrt -l10 -b100   51M

What I want you to take away from this is that xml-wrt/PAQ is pretty slick and actually quite usable. xml-wrt -10 will actually complete in a sane timeframe. PAQ8jc on the same file however will take literally ages and probably won't serve any practical purpose for you...