Subject: Re: compressed fits format

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry M. Schlesinger)
Date: 30 Oct 1996 16:46 EDT

If other groups are developing compression algorithms, this would be a
good time to speak up, so that the community can work out common 
standards. 

In article <54jaio$2q8 at rossini.obs-nice.fr>, Alain MAURY <maury at ocar01.obs-azur.fr> writes...
> ...
>Is there any standard for compressed fits files ?.

There is no standard for compression within FITS and no standard 
compression algorithm.  The extension type name COMPRESS has been 
reserved for a future compressed FITS data type but the details of 
this type have not been worked out.  Warnock et al. have distributed a 
preliminary proposal, which is available at 
http://fits.cv.nrao.edu/documents/drafts/compression.*, where * is 
either tex (LaTeX) or ps (PostScript).  I suggest reading this paper 
to see what some previous thoughts on this subject have been.  
Individual installations are experimenting with local schemes. 

>I am about to include two modes of compression on our images, one
>with a lossless algorithm ( storage of the difference between 2 16
>bits pixels on a single byte ... and a lossy one after the one used for
>the digitised sky survey ( using H transform ). I ... 
> would like to know is there are standard FITS keywords
>describing these compression ( COMPRESS = T ? )
>We decided it was much better to perform real time processing of the
>data, then compress it by a factor of 10 and store it on CD ROMs ( which
>would then hold about a week of observations ) rather than to store
>everything on tape which cannot be read after 3 years... and would
>prefer to conform to some kind of standard if there is such a thing.

Such data could be included in a FITS file only as a nonstandard 
conforming extension.  The BITPIX, PCOUNT, GCOUNT, NAXIS and NAXISn 
keywords would have to have values that provided the size of the data 
area, as required by the generalized extensions agreement.  A separate 
extension type name would have to be defined for each kind of data, 
e.g., DIFFCOMP for the first algorithm and SURVCOMP for the second, and 
registered with the IAU FITS Working Group.  To conform to the rules, 
the names should be registered before the files are created. (Note: 
These are the first names that came off the top of my head; I am not
necessarily recommending them.)  There could then be software that
would recognize these names as signifying FITS images compressed in
the appropriate manner.  Other software would just use the information
in the header to skip the HDUs containing data compressed in these
ways.  

				Barry Schlesinger
				FITS Support Office
				GSFC/ADF

---

From: Alain MAURY <maury at ocar01.obs-azur.fr>
Date: 6 Nov 1996 11:06:56 GMT

Since my original message was posted, I was able to read the document
recommended by Dr Schlesinger, and found it to be a very good basis to
start from. I have also tried the Hcompress algorithm, and with
a factor of compression of 10, it does not seem to loose much
as far as faint stars are concerned if you blink the original image
and the compressed-decompressed one. This is rather subjective, a
more detailed analysis will be done here. Still, sounds good. The
compression time may be a concern, but we should run this algorithm
on a Pentium and see what we get. It seems that 2 minutes of compression
time for a 2K image ( i.e. 8.5 megabytes ) is doable.
What is the procedure for some modifications to the FITS standard to be 
registered by the IAU FITS WG. Next general assembly ( next year ? ),
or a separate meeting ?
Finally, after possible "registration", would compressed fits files still
have the .fit ot .fts extension, or would a new extension like .ftc ( fits
compressed ) be used ?
Alain