Subject: FITS compression

From: rlw at stsci.edu (Rick L. White)
Date: 28 Oct 91 18:18:32 GMT

A few remarks on the FITS compression proposal:

Greg Scott Hennessy (gsh7w at fermi.clas.Virginia.EDU) writes: 
% It is very common already to have FITS data sets that will not fit
% into RAM, and I think that it would be a mistake to rely on a
% compression scheme that required reading the whole data set into ram. 

Doug Tody NOAO/IRAF CCS (tody at noao.edu) writes: 
% Unless there are good reasons to do otherwise it would be wise to use a
% compression techinique which is local in nature ... This is particularly
% important if the data is to be randomly accessed at runtime, for example,
% when reading a FITS image from a CD-ROM, or from disk.  It would also aid in
% recovery from data losses.  A simple technique would be to compress each
% image line independently, using an index to record the offset of each
% compressed line.

This is an important point.  The GS digitized plates are 14000x14000.  It
would be terrible to have to uncompress the entire plate to get at any of
the data.  In fact, it would be terrible to have to uncompress an entire
14000 pixel line of data just to extract a small section of the plate!
The approach we have taken is to compress the image in blocks (500x500);
then one can extract an image section anywhere on the plate by
uncompressing only the required blocks.  The compression technique we're
using is fast enough that I don't think we would bother breaking small
images up -- e.g. it takes only 4 secs to decompress a 500x500 image on a
Sparc-1.

Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: 
% The second "trick" [of the FITS compression scheme] is that, when you
% compress the data, you compress everything, including a FITS header.  The
% reason for this is that it allows you to compress _any_ FITS data stream,
% and the result of the decompression will always be a fully-qualified FITS
% data stream, which you can feed directly back into your FITS reader.

Any scheme which is appropriate for binary data will not be appropriate
for text, so it does not really make sense to think of putting both
the header and data through some generic compression tool.  (E.g. if
you apply your successive differences to text and then feed it to
unix compress, I am sure it will not compress as well as the original
text.)  The beauty of the FITS compression proposal, though, is that
one is free to define a compression method that applies one technique
to the header (or does nothing to the header) and an entirely different
technique to the data.  After all, you only have to look for the
END record to know when to change horses.

Greg Scott Hennessy (gsh7w at fermi.clas.Virginia.EDU) writes: 
% Table 1 in the Warnock et. al. draft clearly shows what we all know now.
% Some data sets compress well (using a generic compress, not referring to any
% particular algorithm) and some data sets do not. My conclusion from
% Table 1 is that the space savings from different algorithms is a small
% fraction of the range of savings possible, hence picking a good but
% admittedly not best for all cases algorithm to standarize on is a win
% in keeping down software complexity over supporting multiple
% compression algorithms to gain 5-10 percent savings. 

Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: 
% Bang for the buck - a good concept.  Computing time goes up 
% substantially with increasing compression ratio.  I hope no one claims 
% to be able to decide for everyone else what investment level is 
% sufficient.

Actually, deciding how much noise to keep is far more important than
computing time.  It may be that the best technique for lossy compression
is not the best for lossless compression.  The JPEG group took this
approach -- their lossless method is tacked on after the fact and
has little in common with their main (lossy) method.  (Incidentally,
my method we're using for the GS plates performs well for either
lossless or lossy compression.)  In any case, I think the decisions
to be made can effect the compression achieved at the factor of 2--3
level, not just the 5-10 percent level.  I think the idea of having
a fair number of methods to choose from is a good one.

Rick White,   rlw at stsci.edu
Space Telescope Science Institute