Subject: FITS compression

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Wed, 22 Sep 1999 15:07:43 -0400

A Proposal for Representing Compressed Images in FITS Format
------------------------------------------------------------

A small group of us have been experimenting with ways to represent compressed
images in FITS format and have drafted a proposal that we would like to submit
for comments or suggestions from other FITS users.  The current draft proposal
is available in html format at:

http://legacy.gsfc.nasa.gov/docs/heasarc/fits/compress/compress_image.html

or in postscript format at

http://legacy.gsfc.nasa.gov/docs/heasarc/fits/compress/compress_image.ps

In brief, our proposed technique is to first divide the n-dimensional image into
a rectangular grid of one or more subimages or 'tiles', then compress each tile
individually and store the resulting byte stream in a row of a variable-length
column in a binary table.  Any number of different compression algorithms may be
supported by this convention and we specifically describe a few possible
algorithms in the appendixes to the proposal.  The appendixes are currently
incomplete, but we eventually intend to provide a complete description of each
supported compression algorithm.

We are currently developing prototype implementations of this proposal.  In
particular, I will be giving a presentation at next month's ADASS meeting that
gives a progress report on supporting this convention from within the CFITSIO
library.  

Bill Pence, NASA/GSFC, on behalf of
Richard White, STScI
Perry Greenfield, STScI
Doug Tody, NOAO

---

From: thompson at orpheus.nascom.nasa.gov (William Thompson)
Date: 22 Sep 1999 23:21:15 GMT

>http://legacy.gsfc.nasa.gov/docs/heasarc/fits/compress/compress_image.ps

	(rest deleted)

This is a very interesting proposal.  However, if I understand it right, only
one column in a binary table could support compressed data, which is pretty
restrictive.  It would be better if multiple columns could be compressed to
better match the present capabilities of binary tables.  Otherwise, for some of
us at least, it would be a major change in the way that binary tables are
organized.  For example, in our own project, we use different columns for
different wavelengths.

William Thompson

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Thu, 23 Sep 1999 10:04:41 -0400

> organized.  For example, in our own project, we use different columns for

It is a little confusing, but this proposal only applies to compressing images
that would otherwise be stored in a FITS primary array or IMAGE extension.  The
compressed image bytes are then stored in a binary table, which serves as a way
of indexing the location of different pieces of the image.  Being able to
compress ASCII or binary tables is also an important issue, but the current
proposal does not address that topic.

-Bill Pence

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Thu, 23 Sep 1999 16:37:17 +0200 (MET DST)

On Thu, 23 Sep 1999, William Pence wrote:

> It is a little confusing, but this proposal only applies to compressing
> images that would otherwise be stored in a FITS primary array or IMAGE

that was clear enough (at least to me) ...

> extension.  The compressed image bytes are then stored in a binary table,

...being the binary table format already overloaded with different (semantic)
applications, and being the "variable format" the most awkward feature in it,
would it not be neater to define "compressed images" as an altogether *NEW*
FITS extension ? 

Its data area could then be arranged as a stream of compressed data according
to any of the proposed formats, and just header keywords used to describe it.

----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR - via Bassini 15 - I-20133 Milano (Italy)      
----------------------------------------------------------------------------
L'Italia ripudia la guerra [...] come    Italy repudiates war {...] as a
mezzo di risoluzione delle controversie  way of resolution of  international
internazionali                           controversies
                [Art. 11 Constitution of the Italian Republic]
----------------------------------------------------------------------------
For more info : http://www.ifctr.mi.cnr.it/~lucio/personal.html             
----------------------------------------------------------------------------

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Thu, 23 Sep 1999 13:26:12 -0400

> to any of the proposed formats, and just header keywords used to describe it.

We need the full flexibility offered by binary tables for this compression
convention, so we don't want to invent a new type of extension with all the same
features.  In the general case, the binary table containing the compressed image
will contain multiple rows (each compressed image tile goes into a separate
row), and multiple columns.  There may be 2 variable length columns, one for the
tiles that could be compressed and one for any tiles that could not be
compressed.  There can also be any number of other columns in the table
containing other parameters that are needed to compress or uncompress the image
tiles.

Only in the simplest case where an entire image is compressed as a single tile
would you end up with a binary table that has just a single row and potentially
only a single column.

____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)