Subject: Re: a FITS puzzle, suggestions welcome

From: dwells at fits.cx.nrao.edu (Don Wells)
Date: Sun, 22 Sep 1991 04:05:41 GMT

In article <21064 at darkstar.ucsc.edu> sla at helios.ucsc.edu (Steve Allen)
writes:
 SA> ... at Lick ... construction of ... 2x2 mosaic of ... 2048**2
 SA> chips... adjacent edges ... butted together ... When ... all 4
 SA> chips [are] readout, we expect that the best method of storing
 SA> these images is as a 3-d FITS array... akin to .. HST's WF/PC
 SA> images ... The question ... is...: What happens when subregions
 SA> of the CCDs are readout as part of a single exposure?  More
 SA> precisely, if the subregions of the 4 CCDs do not have the same
 SA> dimensions, how can these images be stored as a FITS image?  

This question does not currently have a widely-accepted answer.
However, the general version of your question, and related questions,
have been debated during the past two years. I can think of five
possible answers, and there may be several more that I have forgotten
at the moment.

(1) Pad to full size

 SA> ... pad the edges of the images with Null values .. up to
 SA> 4096x4096 ... take up A LOT of space ...  

This is not only inefficient, it is definitely not elegant.

(2) Use separate Basic FITS files

 SA> ... store... images as 4 separate FITS files ... unappealing ..
 SA> would like to keep data from the same exposure in one file.  

I also prefer to keep a set of related data in one file, but this
multi-file solution is a practical workaround until a better solution
becomes generally accepted.

(3) Argue to standardize XTENSION='MATRIX' (or 'IMAGE')

The Generalized Extensions Agreement of 1984 specifies an obvious
syntax which could concatenate multiple "Basic FITS" images of
different sizes in one FITS file.  As a matter of policy, we decided
to deprecate such usage in order to assure that Basic FITS continues
to be the preferred way to encode matricies, but in principle the FITS
community could agree to relax the prohibition for cases like yours.

(4) Use XTENSION='BINTABLE' with the "Green Bank Convention" and padding

The BINTABLE design encodes tables (arrays of C struct) in which
fields may be vectors. One can store a matrix in a vector, and encode
the dimensionality of the matrix in other fields by convention (the
BINTABLE draft also specifies an optional matrix dimensionality
keyword). Imagine a set of Basic FITS files, as in case (2) above. We
could have a vector field big enough to hold the largest matrix of the
set. We could define a column of the table for each keyword that
occurs in the set, including such keywords as NAXIS, NAXIS1 and
NAXIS2, and label the columns with the keyword names. Now let each
Basic FITS file become one row in our table, and pad the matricies
smaller than the maximum with blanks. It is typical that many of the
keywords of the set will be the same for all the headers, so that the
corresponding columns of the table will be constant. We can agree, by
convention, that we will put the keyword=value into the table header
and eliminate the column (this is the virtual column convention). The
resulting encoding of a set of Basic FITS files is efficient for a
wide variety of cases. In particular, at the "Single-Dish FITS"
workshop at Green Bank in November 1989 this whole concept was
proposed as the mechanism for encoding sets of spectra in FITS.  If a
set of fixed-length 1-D spectra has more than about 20 members the
overhead of the self-describing header structure for the table becomes
essentially negligible, thus removing the major classical objection to
the use of FITS for spectroscopy. (Obviously the efficiency is lower
for variable-dimension arrays.)  This concept also has the advantage
of keeping a *set* of observations together in one file.  Note that
generally the convention is invertible: the Basic FITS files can be
recreated from the table. Two radio single-dish prototype
implementations of the Green Bank Convention are currently being
tested for interoperability.

(5) Use XTENSION='BINTABLE' with the GBC and variable vectors

The May-91 version of Cotton and Tody's BINTABLE draft standard
specifies a "pointer" data type.  A field, rather than containing a
vector (matrix), can instead contain a byte-offset pointing to the
vector, plus a byte-length for the vector.  This mechanism supports
variable-length vectors of various datatypes in the table fields. The
vectors of various types would be concatenated into a vector of bytes
which would be appended to the body of the table of fixed-length rows,
and the length of the concatenated vector would be encoded in the
PCOUNT keyword of the generalized extension header. Therefore, we
could use the pointer type for our matrix field and the Green Bank
conventions for keywords of our set of variable-matrix observations.

As far as I know there are still *no* prototype implementations of
this *WONDERFUL* variable-structure idea, but in my opinion it is
likely to become the most popular FITS data structure by the end of
the 90's, because of its versatility, efficiency, and elegance. It
would be risky to use pointers for any production application at this
time, before the necessary R&D and testing have been done, and before
any of the major datasystems (MIDAS, IRAF, AIPS,...) have released
code supporting the pointer type in binary tables.
--

Donald C. Wells             Associate Scientist        dwells at nrao.edu
National Radio Astronomy Observatory                   +1-804-296-0277
Edgemont Road                                     Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N