Dr. Dobb's Data Compression Newsletter Issue #51 - March 2004

Welcome to this issue of Dr. Dobb's Data Compression Newsletter! This newsletter keeps you up to date with news items, product information, and the occasional editorial aside about the world of Data Compression. Your input is always welcome in the form of comments, submissions, or original material.

Roberto Amorim published the results of his AAC 128 Kbps listening tests last week, and Apple should be very pleased with the results. Statistically, Apple and Nero tied for the best encoder, but when you look at the charts, you're going to clearly see Apple on top.

Roberto tests a good variety of music, which is quite important, given the fact that every codec has its own sweet spots and problem areas. The test is of course a blind listening test, and the competing codecs were:

• Nero
• FAAC
• Compaact!
• Real
• iTunes

Hydrogenaudio forum: AAC at 128kbps v2 listening test - FINISHED

This library on the CodeProject web site is really great, totally free, and seems to be in constant development by Davide Pizzolato.

I haven't tried out CxImage yet, but I think Davide has done a pretty good job of designing this thing so it's programmer-friendly. It seems to get nice high ratings from CodeProject reviewers, and that must be worth something, right?

CxImage home page

The LZO libraries attract a lot of interest from programmers working in the embedded world. LZO is careful not to place too many demands on the CPU or RAM of the target system, which lets it get into a lot of places that a library like deflate just doesn't fit.

It makes sense then to try to wrap this up in a .NET framework to see if we can't get LZO working in the Windows CE framework. After all, Microsoft is still doing its best to get WinCE adopted as a standard embedded platform, so there's certainly a market out there waiting for this product.

This is the 1.0 release, and it looks like it went out the door in late February.

LZO.NET

I've been warning you about this for almost two years now, and my predictions appear to have come true. It looks as though Microsoft has managed to convince the DVD Forum to include the Windows Media 9 video codec as part of the High Definition DVD standard.

This makes it quite likely that Microsoft's codec is going to be shipped in every DVD player made for the HD market. And who knows, in a few years that might be the entire market.

Of course, the players are still going to support MPEG-2 and H.264 as well, but like the cuckoo bird in the sparrow's nest, Gates et. al. will quickly muscle those two pretenders out of the nest, leaving Microsoft to rule the roost.

Microsoft is now going to get an excellent revenue stream for the mandatory royalties box-makers are going to have to pay, and in addition stands to profit immensely from selling production tools to a captive audience. Well done, Redmond!

CNET.com: Microsoft on every DVD?

Joemyim Kim posts a pretty reasonable question to comp.compression. He wants to know, if wavelet compression is so hot for still images, why isn't it the way to go for moving pictures as well? We have a pretty good JPEG-2000 standard that does a great job with wavelets. Yet the state of the art for video is H.264, which is still using the familiar block-based DCT scheme.

In short, it appears that the consensus answer on this is that the block-oriented motion estimation used in video compression doesn't mesh well with wavelet transform stuff. And without motion estimation, video compression rates really fall.

Some good discussion here, although the thread executes a classic taper-off move by veering away into a discussion of estimation models in PNG.

Google archive of comp.compression thread

A nice article in IEEE Spectrum describing the turbo codes that were a bit of Information Theory excitement back in the 90s. Error-correcting codes go hand-in-hand with data compression when you are trying to get the most out of a communications channel, so you need to bone up on this stuff.

IEEE Spectrum online: Closing in on the perfect code

This is of course the big news of the week in the compression world. WinZip 9.0 has been in beta a long time, but apparently the company has finally resolved to pull the trigger and ship this product.

The biggest changes in 9.0 include the following:

• AES encryption
• 64-bit extended Zip format
• Improved execution of installers in Zip files.
• One-click Zip and email functionality
• WinZip Self-Extractor Personal Edition

WinZip home page
WinZip info
WinZip 9.0 press release

Igor Pavlov is the developer behind the amazing 7-Zip compressor, which has always been available under the GPL. Igor has now created a separate LZMA SDK, which implements his compression algorithm in a way that makes it suitable for embedded applications.

On the SDK web page, Igor says that the LZMA code can decompress up to 1 MB/s on a 100 MHz ARM, MIPS, or other RISC CPU. The memory requirements for decompression are as low as 8-23 KB, and the code may take up as little as 2-8KB.

This sounds like a great piece of work for embedded developers. Up until now, the best library out there for this community has been LZO, which has a few problems that hold it back. Perhaps Igor's product will now be the go-to library for this community.

7-Zip SDK page

Well, here is an article on ExtremeTech.org that is bound to make the faithful a bit angry. It does make one point that I certainly sympathize with, which is that iTunes puts an awful lot of restrictions on your ability to enjoy purchased music over your entire lifetime.

ExtremeTech.org

Jason Schmidt was kind enough to send me an email recently pointing out the existence of a project called MP3Sharp. This is a port of the JavaZoom MP3 project to the world of C#.

The author of this project used Microsoft's Java porting tool to do the job, which presumably did all the heavy lifting, leaving him with some touching up and tweaking. It's probably a testament to the folks in Redmond that this could be done in a semi-automated fashion, and it's certainly a big asset for the C# community.

MP3Sharp home page

From the author's Web page:

rzip is a compression program, similar in functionality

to gzip or bzip2, but able to take advantage of long distance redundancies in files, which can sometimes allow rzip to produce much better compression ratios than other programs. The original idea behind rzip is described in my PhD thesis, but the implementation in this version is considerably improved from the original implementation. The new version is much faster and also produces a better compression ratio.

rzip home page

The well-known JasPer JPEG-2000 codec has been around for a while, but found its promulgation a bit stifled by a somewhat restrictive license. Well, there's good news in graphics land: JasPer is now available under a more conventional Open Source license. This should clear the way for this product to show up right and left in commercial and free products.

Jasper is currently found in a few familiar places, including:

• KDE
• ImageMagick
• Netpbm

JasPer home page

I really like it when a program's web page has a great one paragraph description of what it is and what it does. That means I can sit back and blockquote it without doing any work of my own. As demonstrated here:

KDar is the KDE Disk Archiver. It is a convenient graphical interface to Denis Corbin's libdar archiving and backup C++ library. You can safely backup (with compression if you like), view and restore files using KDar, along with a few easy UNIX commands for mounting and burning CD-R's, Zip and Jazz drives, DVD-R's, floppies, or any other disk-based storage media.

KDar home page

It-Guy Cory Doctorow has a blog of sorts, which is mostly a liberal arts affair, but one of his recent entries ventured well into our little world. Cory posted a neat story on the creation of innocent-seeming compressed archives, that when opened, mushroomed into gargantuan files that immediately consumed all your resources.

Think about it, some pretty cool programs such as bzip2 can compress with ratios of hundreds or even thousands to one. Imagine a 10 MB file named MyWifesBootie.bz2 that turns into a 25G file when your email program tries to extract it.

In the article linked to here, the author mentions a gzip encoded file going from 7K to 100G, which I supposed could only happen with a mangled header of some kind - I'm not even sure it's possible.

Boing Boing: Decompression bombs: email attachments expanded and expanded and expanded
AERAsec: Decompression Bomb Vulnerabilities

Someday we're going to have H.264 video codecs that can do realtime two-way processing on a PC. We're all looking forward to that day, and it may take nothing more than upgrading to something at or above a 3GHz P4.

While waiting for that, we can at least check out MainConcept's H.264 decoder for Windows, now available as a free download. The fact that the technology is deployed as a DirectShow filter means you can use it in an application as friendly as Windows Media Player. Why not give it a look?

While you're at it, don't fail to note that MainConcept has a nice SDK that you can use as well.

EE Times: MainConcept Releases H.264 Encoder Technology Preview Version

Cringley Observes in Burst vs. Microsoft
February 17, 2004

Robert X. Cringley traveled out to the east coast to sit in on some hearings in the big to-do between Burst and Microsoft. This is a pretty important case for Microsoft, as losing it might hamper their ongoing plan to own a piece of all digital media bought, sold, or borrowed in this great world of ours.

Cringely floats an interesting and provocative theory. He seems to think that the Microsoft team, with effectively unlimited money and lawyers, is going to be able to wear down any lone judge facing them from the bench, leading to insanity, which leads to opinions that are thrown out by higher courts.

Not a bad defense strategy, if you can afford it.

I, Cringely, Feburary 12, 2004

A few virus writers have managed to figure out that encasing their payload in a zip file can go a long way towards hiding it from the prying eyes of anti-viral products.

This article in ComputerWorld discusses the use of Zip files to distribute viral payloads with the top-of-the-charts MyDoom virus. Seems that this idea really took off when Windows XP started guaranteeing that PC users would have the goods needed to open the zip file.

Computerworld: .zip files putting the zap on antivirus products

That's all until next month. Write me with your feedback at [email protected]. Thanks! - mrn