Subject: DynaFITS (LONG. was: fitsio lib -> STS)

From: dwells at fits.cx.nrao.edu (Don Wells)
Date: Fri, 2 Aug 1991 19:51:38 GMT

In article <jwright.681120716 at cfht.hawaii.edu> jwright at cfht.hawaii.edu
(Jim Wright) writes:
 JW> ...How do you support adding an arbritary number of cards to a
 JW> FITS header?  The obvious solution (two separate files,
 JW> concatenated when finished) is less attractive as the size of the
 JW> data portion grows.  Right now we're at 8MB files (2Kx2K) with
 JW> the prospect of 32MB shortly (4Kx4K).  Any clever way to prepend
 JW> the header to the data without having to move megabytes of data? 

I assume that you are wanting to add more cards to an already-written
header in order to record ancillary information about the observation.
Many practical cases like yours are well-represented in tabular form.
If your ancillary data looks like a table, you should consider using
the FITS ASCII-table extension for this purpose. Voluminous tables can
be represented more efficiently in binary format; there is good reason
to expect that most astronomical data systems will be enhanced to
support the proposed BINTABLE extension during the next few years (see
file /FITS/doc/bintable.ps on fits.cx.nrao.edu [192.33.115.8]).

However! The table technique does not really address the basic problem
which you point out. The problem arises in two contexts: (1) RT data
acquisition systems in which header and data need to be created
_asynchronously_ (the case which concerns you) and (2) use of FITS as
a machine-independent _random-access_, _re-writeable_, _shareable_
disk format in data analysis systems.

In August 1990, at an NRAO workshop discussing what is now called the
aips++ project, item (2) was suggested as a desirable concept for the
disk file format of aips++, and Bob Hanisch (STScI) and I were asked
to bring back a proposal. The concept which we suggested was to define
an index file, a table, which would tabulate pointers to a sequence of
byte-strings which, when concatenated, would constitute a standard
FITS file. This would permit not only the content but also the
_length_ of any segment in the FITS file to be updated dynamically, an
essential feature in a data analysis system.  Accesses to the index
table for updates would be a "critical region" and would have to be
guarded by lock mechanism(s), of course. This idea for dynamically
updating FITS bytestreams was promptly dubbed "DynaFITS" by someone
else at the workshop (Geoff Croes?).

The original concept was that the index table would have three
columns, local filename (including path), byte offset and number of
bytes. Over the past year I have thought a lot about this idea, and I
am just about convinced that it is a "good thing". I now think that
the filename column should use some sort of syntactic convention to
indicate remote hosts and also a convention to indicate (remote)
_sockets_. My current thinking is that the index file should be
formatted as a naked FITS binary table (no Basic-FITS header). I have
not decided on column names, but 'PATH', 'OFFSET' and 'LENGTH' sound
like good choices. Presumably the name would be EXTNAME='DYNAFITS'.
Perhaps one or two more columns should be added to specify whether a
segment is the beginning of the Basic-FITS data, or an extension
header of a certain type, or the extension data, or...

Incidentally, such a table could have other, system-dependent,
proprietary columns which could be used by a data system to keep track
of such things as whether a binary copy of the header segment is
cached in the I/O system, whether a machine-dependent translation of
the data segment is in a scratch file, whether an index exists for a
column of a table, etc...

The idea of being able to specify a _socket_ instead of a path to a
file is potentially a very powerful feature of DynaFITS, because it
would enable an arbitrary server process on an arbitrary host to
_dynamically_ create a segment of a FITS file by an arbitrary
computational algorithm.

It would be possible to compute the various segments of the file with
different processes at different times. This feature of the DynaFITS
idea is particularly important for RT data acquisition systems. In the
case being cited in the original posting, an auxiliary table could be
computed and written to a file while the CCD image is being copied to
another file, and while yet another process is generating the
Basic-FITS header in a third file. The result would be a virtual FITS
file which could be copied to external media later on by a trivial
utility operation which reads the index file and does the obvious
thing.

This posting to alt.sci.astro.fits about the DynaFITS idea is not
really a proposal, it is merely a trial balloon. I may or may not
build a prototype implementation of DynaFITS myself for the VLBA
Correlator. I would be happy to discuss the idea with anyone...

--

Donald C. Wells             Associate Scientist        dwells at nrao.edu
National Radio Astronomy Observatory                   +1-804-296-0277
Edgemont Road                                     Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N