Welcome to this issue of Dr. Dobb's Data Compression Newsletter! This newsletter keeps you up to date with news items, product information, and the occasional editorial aside about the world of Data Compression. Your input is always welcome in the form of comments, submissions, or original material.
The Juggernaut Rolls On
As part of the Windows Media 9 product launch, Microsoft has posted a nice set of developer downloads on their web site. At this time, the list of available packages includes:
- Windows Media Format 9 SDK
- Windows Media Player 9 SDK
- Windows Media Encoder 9 SDK
- Windows Media Player for Pocket PC SDK
- Windows Media Format 7 SDK for Macintosh
- Windows Media Services SDK
The link below brings you to that page, and also makes it easy to navigate to a nice batch of technical articles, plus SDK documentation. There's enough stuff here to make a career of it:
As Windows Media 9 rolls over the civilized world, reviews continue in the popular press. Larry Blasko of the Associate Press offers high praise indeed. He says that Windows Media Player works as advertised, doesn't break anything else, and is kind of nifty. Does this count as a rave review? I'm not sure, but at a minimum it counts as another point for Bill's team.
Microsoft Media Player Draws Praise
Streaming Media Patent Scare
Bad news for people broadcasting compressed audio or video across the web. A company called Acacia Media Technologies has been dropping little bundles of joy all over the industry for the past month, in the form of requests for licensing agreements.
Acacia claims that it has some basic patents covering just about any form broadcasting on the web. Big companies like Virgin are caving when faced with the strength of the claims, leaving not much hope for the little guy.
About the only good news here is that Acacia is only asking for a little nibble of the profits. One web site operator quoted in the article below threw out a number of .75 percent of revenue. Of course, that's a small figure, but since Acacia will be attempting to get this from virtually an entire industry, they stand to pull in a pretty big revenue stream if their strategy holds up.
News.com - Patent scare hits streaming industry
After it gets all the world's Webcasters in its pockets, Acacia says it is going after cable operators, online movie sites, and others who offer video on demand. And no, they're not forgetting the folks who are streaming more video than all of Hollywood combined: the streaming porn industry. Turns out those guys were one of Acacia's first targets!
Acacia Says Webcasters Infringing on Patents
Incredible Claims
My position as proprietor of DataCompression.info and editor of this humble newsletter guarantees that I get my share of missives from those with incredible compression claims. I enjoy reading them, although I have to confess that the lifetime accumulation of knowledge from these sources is still pegged at 0.
Last month I got an email packed with the usual breathless excitement from someone named Carla Koerner Hein, and it made such good reading that I thought I would reproduce it here in all its glory, with no touchups on my part:
Dear Mark Nelson,
What's this I read about data compression? Take a look at a hyperfractal -- it starts, processes and ends neatly data compressed. A billion serial, interfaced with a billion parallel, is still 1,000,000,000:1 data compression. So why the big fuss?
Does anyone use hyperfractals? I can't believe all the unnecessary processes that engineers go through, just to make the serial/parallel convergence! If you would just start with a hyperfractal, like binary 1,2,4,8,16,32,64 -- interface -- 64, 32,16,8,4,2,1 aligned pair-to-pair, the serial/parallel convergence is self-evident.
And the same thing applies to supersymmetry -- a hyperfractal starts, expands, compresses and ends -- right on supersymmetrical. Scale up, scale down. No sweat. I can't understand how prestigious high-performance supercomputer labs can spend millions of dollars and decades of time chasing computational rainbows when a hyperfractal would do the task in minutes at no cost at all.
Maybe it's because of my unorthodox background in computing -- I'm an art major surrounded by straight-line-thinking computer geeks who dote on complexity and dense, convoluted logic. They adore computer algorithms but toss off massively parallel genetic algorithms as too difficult and messy to bother with. Show them how and why a hyperfractal is the epitome of order, balance, harmonic convergence and realtime serial/parallel exchange and they scoff -- while looking right at a hyperfractal wiring diagram!!! There's something spooky about the hyperfractal architecture that makes engineers go instantly deaf, dumb and blind and I intend to find out why, even if I have to do a frontal lobotomy on 'em to discover the glitch in their thinking. Cheesh!
I take it that instant/total data compression is a Big Thing with MP3 file-sharers?
If so, good buddy, I'd enjoy a brainstorming exchange about hyperfractal wiring diagrams and all the innovative applications that absolutely, positively require total data compression from the getgo. Just start/process/end -- nothing fancy.
Stay focused on fractal-to-fractal, pair-to-pair convergence. Basics only -- no frills. Tell 'em that I have been so supersensitized by knee-kick criticism and being told that total data compression is impossible and supersymmetry is too complex for my poor simplistic little brain, that I'm apt to be very rude to the next party who can't understand S.I.M.P.L.E. I mean it! No joke! Moore's Law is double/double/double/double and Ashby's Law is half/half/half/half and running creative problems up and down the scale in the binary recursive by multiple progressions is great fun and wonderful mental exercise. Hyperfractals are a flying trapeze for daring and adventuresome supercomputing highflyers.
Try it. You'll like it!!
Carla Koerner Hein
I have a stock response to any message such as this, which is "Show me some code." In other words, let's see you flesh out these concepts with some implementation details. Naturally, Carla had none, but did offer the usual get-with-the-program advice:
It's OK by me if you want to trip over your own feet and compress data the hard way -- but wouldn't it be easier to make the direct-connect serial-to-parallel, fractal-to-fractal, set-to-set and field-to-field? By eliminating the pile-up of serial algorithms in realtime, there's nothing to compress and you're home free.
Nature loves fractals -- hates squares, cubes, lattices and grids. Engineers love squares, cubes, lattices and grids that they can quantify the long, hard, slow way, searching in vain for the perfect quantum-to-quantum match between serial algorithms and parallel genetic algorithms (galgorithms). Someday, when you experience that AHA!! moment when you can distinguish a hypercube from a hyperfractal, let's talk again.
Of course, Carla is right about one thing. You can get incredibly complex output from fractal expressions. (Although I have to confess that I have no idea what a hyperfractal is.)
You can also get incredibly complex output from a random number generator, and this leads many neophytes to suppose that a universal compressor might be made by simply selecting just the right generator function for a particular sequence.
I've dubbed this particular fallacy Magic Function Theory, and enjoy it so much that I dedicated a section of my FAQ to an explanation of just why it doesn't work.
For Carla's sake I guess I would point out that the Mandelbrot function is capable of creating incredibly complex (even
beautiful) pictures, but just try to get it to generate the assembly drawings for a Buick Riviera.
My final comment to Carla follows:
BTW, I do have a standard challenge to give to people are moderately out of touch with reality when it comes to data compression. It's simple. Compress this file:
http://datacompression.info/Miscellaneous/AMillionRandomDigits.bin
All you have to do is create a decompressor and a compressed file that together take up fewer bytes than the original file. (Which will run on a naked PC off the net, of course.)
No succesful response to the challenge from Carla yet! (Nor anyone else.)
DivX Dissed
The DivX video compression format, basically a wrapper for an MPEG-4 stream, has long been the format of choice for storing movies on hard drives or CD-Rs. DivX lets you squeeze the content of an average Hollywood movie down to as little as 1 GByte, making the concept of a drive-based video jukebox a reality.
Well, DivX is a nice codec, and available for free, but it turns out that there's more than one way to skin this particular cat. C.J. Stallings wrote up an article for ExtremeTech on the viability of using Microsoft's Media 9 video codec to do the same thing:
ExtremeTech - DivX? We Don't Need No Stinking DivX!
This step-by-step guide has links to the appropriate software along with specific instructions on how to perform the process. (No help on how to rip content of your DVDs, but that information is available with the help of a search engine.)
Unfortunately, the one thing missing from this article is a head-to-head comparision of DivX vs. Windows Media 9 at similar bit rates. We've heard a lot about the Corona encoder in WMP 9, but there is a dearth of competitive analysis at this point.
DivX is still humming away, despite the competition from Corona. Developer DivX Networks announced at the end of January the release of DivX 5.03, a codec that includes a verification suite designed for consumer products, including handheld computers, Home Theatre systems, and HDTV. The goal is to make sure that the codec can be installed in these types of devices with confidence.
Internet.com - New DivX Video Codec Supports CE Profiles
News.com - DivX software delivers movable media
MPEG-4 DRM On the Way
Hollywood and our various media conglomerates have always been horrified by uncontrolled file formats such as MP3. The notion that files can be freely copied from one computer to another is anathema to this crowd.
What the moguls want are media formats that contain Digital Rights Management (DRM) features. For example, Windows Media 9 supports DRM for audio files, which means a service like PressPlay is happy to let you download all the audio files you like to your PC. The secret bits in those audio files mean you can't give them to your buddy, and you can't listen to the files once your subscription runs out.
The MPEG-4 video format is now going to have security and rights management features added to its specification by a global group known as the Internet Streaming Media Alliance.
This is of course a classic good news/bad news story. Adding DRM to MPEG-4 opens the door for a lot of new distribution channels for the movies we all love to watch -- but makes it a lot harder to bootleg them. In other words, you will probably be able to download movies to handheld players quite soon, but it won't be from Kazaa, and you're going to have to pay for them.
News.com - MPEG-4 on road to rights management
More good news for MPEG content providers comes from the MPEG-LA licensing consortium, which has now issued a final set of terms for standard licensing of the format.
I am not a lawyer, but checking the MPEG-LA site makes it look like streaming content providers are not going to get hit too hard by the new terms. You can get off the hook completely for $1,000,000 per year, but if that seems high, you can pay twenty five cents per subscriber per year, or the modest rate of .03333 cents per minute per subscriber.
As always, read the terms carefully before signing the agreement!
MPEG-4 Systems License Released
Vprove from Vqual
I received an email from Thomas Dove of Vqual Ltd. in Bristol, UK, regarding his company's product Vprove. Vprove is a very interesting niche product that looks like it would be quite valuable to content creators or people who build content creation tools for video.
In a nutshell, Vprove is a tool to perform detailed analysis of video streams in MPEG-4, H.263, and H.261 formats. It does nifty things like checking for standard conformance, analyzing motion estimation, performing macro block analysis, and more.
If you take a look at the product information on the web page, you're going to want a copy of this. I'm not sure that any other product provides this ability to look at your video stream in such detail.
Java/Zip Tips
A Q/A column in JavaWorld magazine from December, 2002 answered a couple of questions regarding how a Java programmer can zip up data resident in memory, as opposed to files. The second question in this column was particularly interesting as it addresses a quest to zip up data being passed through RMI in the reader's app.
Sample source code is provided along with links to numerous online resources addressing similar questions.
JavaWorld - Compress your data
Apparently the questions were inspired by a 1998 article by Todd Sundsted, in which he discussed the the use of the java.util.zip classes, a topic that I explored as well in a DDJ article. Todd's article is still available on the magazine's web site:
Zip your data and improve the performance of your network-based applications
Calibre Technologies CxZipCOM
We're always happy to see new compression libraries here at DataCompression.info. This month we got an email from Calibre Technolgies, a company that appears to be located in India. They've offered a new COM-based DLL library called CxZipCOM.
The interesting thing about CxZipCOM is that it not only performs all the normal Zip library functions, but it also adds functions to open Zip files via HTTP or FTP connections, and to email Zip files.
You can download an evaluation copy of CxZipCOM here:
Sample code is included for VB and FoxPro, unfortunately no sign of any C++ or C# code. But since this is a well-behaved COM object, you would think that it should be a simple matter to use with those languages.
Zlib compression / decompression wrapper as ISequentialStream
A fellow named Jens Nilsson has written a nice set of C++ classes that wrap zlib so that it can work properly with Microsoft's IStream COM interfaces.
If you're not familiar with zlib, it is a free implementation of the deflate compression engine used in PKZip and other Zip-compatible archivers. I liked it enough to write an article for DDJ describing its use.
Jens's project appears to have been posted to the CodeProject web site in January of 2003, so it doesn't have too many user miles under it yet. If you give it a try, by all means let me know what you think.
Zlib compression / decompression wrapper as ISequentialStream
Yet Another CodeProject Gem
A couple of days after I saw Jens's code on the CodeProject web site, I ran into another potential gem. Tony Lin, a programmer from China, created classes to read and write a few different kinds of image files.
This wouldn't be all that sweet if not for the fact that Tony included support for JPEG 2000 image files. Tony used the Jasper implementation of JPEG 2K for support of that format. Jasper is pretty well known and respected, and perhaps this project wraps it up in a convenient format.
Classes to read and write BMP, JPEG and JPEG 2000
wmplugins.com
This site describes itself as The ultimate resource for enhancing your Media Player 9 Series for Windows. By this they mean plugins, themes, skins, visualizations, etc.
What's interesting about this site is that it is owned and operated by Microsoft, but they are trying to play down the corporate aspect of the site. To wit: This site is owned by Microsoft and operated by Microsoft employees, many volunteering their own time to make this a great resource..
I guess we should all give a round of thanks to those Microsoft employees who have built this web site on their own time.
TurboPower's Abbrevia Goes Open Source
TurboPower has given up on the retail library business, and will be placing most of their products into an open source state. Abbrevia is TurboPower's compression library. The SourceForge project for Abbrevia has been set up, but it doesn't appear that work has begun yet, for it is sadly devoid of anything interesting.
Abbrevia was a Delphi product, and I believe that TurboPower was able to leverage that to work with C++ Builder as well. It will be really interesting to see if they can get this library into a usable form for other platforms, such as g++ under Linux. If they do, this might be a valuable addition to the Open Source world.
CxImage - Yet Another Win32 Imaging Library
Does the world really need another Win32 library that promises to read, write, and display BMP, GIF, PNG, JPEG, and J2K compressed images? That's what CxImage is offering.
Maybe not, but it would be a little presumptuous of me to act like this is a johnny-come-lately library. The fact of the matter is that Davide Pizzolato has been releasing new versions of this library for a couple of years.
CxImage is basically an effort to wrap up various imaging libraries and APIs in a consistent C++ set of wrapper classes. This makes it easy for your graphics application to support a lot of different image formats without stretching your coding abilities too badly.
CxImage comes with a nice overview on the beloved CodeProject web site. Check it out here:
FLAC Joins Ogg
In the world of free software, the announcment of a new alliance is not necessarily a big thing. Talk is cheap, and letting the world know that the Open Tuna Project has joined forces with the Axis of Java requires nothing more than a new page on a web server.
It has always seemed to me that the folks working on the Ogg Vorbis project have been responsible for more than their fair share of organizational announcments. At a minimum, they've sent out enough information to keep me totally confused about what they are doing.
To the best of my knowledge, the important point to be aware of is that the Ogg Vorbis project has created a high quality lossy audio codec that provides a free alternative to proprietary MP3 technology.
Ogg Vorbis was at some point absorbed into Xiph.org, an umbrella organization that works with many other free media software projects, including Icecast, Speex, paranoia, and the dubious MGM (Moaning Goat Meter.)
If you are interested enough to follow all that, then you will probably want to get to the punchline of this story. As described in the following press release, the Freeware Lossless Audio Codec (FLAC) has now joined up with the other projects at Xiph.org, presenting an even more united front for the forces of free audio:
In the world of Audio Compression, lossy formats tend to dominate the discussion, but there is definitely a need for lossless codecs as well, and FLAC provides an excellent way to do that without being locked in to proprietary for-profit software.
Data Compression Corpus Pointers
Juergen Abel has a great Data Compression site, and keeps a set of pointers to standard sets of files used for compression. He recently added a reference to the Protein Corpus, a set of difficult to compress files that were first published at the 1999 Data Compression Conference.
The site also points to the two classic sets of files used for testing compression, The Calgary Corpus and The Canterbury Corpus.
jzip.org
Greg Kowal has launched an open source project called jzip.org. He describes it thus:
jzip.org is a WinZip like program written in java. The project aims to provide the ultimate zip tool free of proprietary code and portable to virtually any modern operating system. Preconfigured binary packages are provided for Windows 9x, Windows 2000 and of course for Linux (that one should fit any Unix-like system).
Well, it sounds like a worthy venture. Greg does add that part of the reason for kicking off the project was to give himself some programming experience, so I guess we'd be unkind to not jump on board.
As is often the case, however, we are not at a 1.0 release quite yet. Check up on the progress so far here:
The Gray Lady Speaks - MPEG Formats
Ever wanted to know what the differences are between MPEG-1, MPEG-2, and MPEG-4? Lucky for you, the New York Times is happy to take a stab at it. You must register, but you need not pay:
NY Times - Technology Q and A, January 30, 2003
A New Page from Tilo Strutz
Dr. Tilo Strutz teaches at the Institut für Nachrichtentechnik und Informationselektronik. (Google happily translates that to The Institute for Communications Technology and Information Electronics.)
As part of his course work on Data Compression, Dr. Strutz has posted a few Data Compression items on his web site. He recently sent me an email to let me know he added a Data Compression overview page:
All you want to know about data compression
This page tackles an overview of the field, creating a general taxonomy of the field plus a framework for most compression operations. It's worth a look, and if you happen to be teaching a class on compression, you might be interested in the PowerPoint presentation that is available on the page.
Releases - New and/or Noteworthy
Compression flagship PKWare announced new versions of PKZip for Unix and Windows. Version 6.0 offers new encryption features, improved compression, and more.
TechWeb - PKZIP Upgrades For Unix, New Version For Windows
Yaakov Gringeler has released a 1.0 beta of his Compressia program, and he tells me that it is setting records with his new algorithms.
comp.compression discussion on results seen with Compressia
A company named THB Componentware has released THBJpeg2000, an addon to their THBImage Library. The Library is available as either a DLL or OCX, and supports a wide variety of image formats and operations. An 30-day evaluation version is available for download.
JZlib is an open source library that is designed to re-implement zlib in pure Java. It looks like the development team must feel that they've accomplished this goal -- they've just shipped version 1.0.
mplib is an open source library that reads and writes ID3 tags. (The text embedded in an MP3 file that gives the player info about the track.)
Aftex Software makes a Java version of Bzip. This has been updated to use BZip2, and now has a command line and GUI interface.
JazzLib will be a pure java implementation of the java.util.zip classes. It is still under development, with a new beta release shipping in January.
Aftex Software makes a Java version of Bzip2. This includes input and output stream classes, which can be used in other Java applications. The program has both a GUI and command line interface.
FLAC, the Free Lossless Audio Coder shipped a 1.10 release in January. A good roster of new features, including updates to the plugins for popular apps.
Steven T. Lavavej released a new version of bwtzip in early February, 2003. A wide variety of improvements, most of them in implementation -- not visible to the end user. A description of recent changes is found here:
comp.compression post from Steven T. Lavavej
BitZipper 3.3 shipped in January, 2003, and adds support for split-volume archives, archives greater than 4GB, and more.
A beta release of Guntar (the lazy man's untar/uncompress utility) shipped at the end of January.
mod_zipvfs is an Apache2 module that gives the httpd server the ability to read and browse zipped archives and allows traversal of their contents like a virtual filesystem.
PicoZip 2.6 was released in February 2003 with numerous small features and bug fixes.
That's all until next month. Write me with your feedback at [email protected].
Thanks! - mrn
