Subject: FITS as a working format

From: dbriggs at Mr-Hyde.aoc.nrao.edu (Daniel Briggs)
Date: Wed, 11 Aug 93 21:58:33 GMT

In article <9308111857.AA08076 at tetra.gsfc.nasa.gov>
  pence at tetra.gsfc.nasa.gov (William Pence) writes:
>>FITS and PDS are pretty good interchange and archival formats, but weak
>>working formats.
>
>The last sentence seems questionable since we have been using
>FITS as a working format within the HEASARC for the past 2 years with
>great success.  I need to give a talk on this subject in a couple
>weeks, so I would be very interested to hear other opinions or
>explanations on why FITS is not the ideal working format.

I'll just comment that we have an image processing system here at NRAO
called SDE that primarily uses FITS as the working format.  It's a
collection of cooperating programs that are starting from the command
line.  All images, visibility files and what have you are stored as native
files under the host OS, and are read into memory as needed by each task.

Now this is a system designed for coding up experimental tasks quickly.
(In fact, the acronym stands for Software Development Environment.)  We
have a very flexible in memory database format for objects, which can
potentially have all sorts of additional things hanging off of them that
are not necessarily easily representable by FITS.  Our FITS reader/writer
makes no effort to encode or decode unknown keywords into FITS format.  It
just writes out all the standard keywords, and a small list of local ones,
if they are present.  If we have a database object that uses the more
flexible features of the format, we store the object as a (nonportable)
memory image.  For most other objects, we use FITS format.  Since we use
SAOimage for image display, there is a strong incentive to use FITS rather
than the memory image format.  (SAOimage can be used on the memory image
files, but only with great effort.)

My point is this: I use this system every day, and do a lot of software
development with fairly complicated algorithms.  I can't remember the last
time I was forced to use the memory image format.  It's amazing how much
everyday work can done with a very simple FITS format, and small handful
of 'optional' header keywords.  Some of our software is a bit sloppy in
assuming that one or more of the local keywords are present in the header,
but that's only bitten me once in the last three years I've been working in
this system.  (That was when importing an image from a different system --
we used different keywords for pointing center.)  Basically, FITS has
worked out just fine as a working format for us, with the caveat that we
have the option of going to a less portable, more flexible format when
needed.  But we rarely need it.

-- 
| Daniel Briggs  (dbriggs at nrao.edu)                      |  USPA B-14993
| New Mexico Tech / National Radio Astronomy Observatory |      DoD #387
| P.O. Box O / Socorro, NM 87801   (505) 835-7391        | 
Support the League for Programming Freedom   (info from lpf at uunet.uu.net)

---

From: cflatter at cv3.CV.NRAO.EDU (Chris Flatters)
Date: Wed, 11 Aug 93 23:06:23 GMT

The limitations of FITS as a working format only become apparent if you have
to modify a FITS file in place.  In this case you start to hit problems due to
the fact that a FITS file contains several pieces that follow one after another.
If you have to extend any but the last you need to shuffle everything later
in the file to make room.

As a simple but contrived example suppose that you have a CCD image and are
removing background stars and CCD defects using some interactive editor.  Your
editor adds history records to the FITS header describing every edit.  After
a number of edits the empty space in the header will be exhausted and the
editor will have to add another 2880-byte block to the header.  In order to
do this it must add 2880 bytes to the end of the file and then copy all of the
image data down by 2880 bytes.  If the image is large this could be an expensive
operation.

This type of problem is made worse if the are multiple HDUs (Header and
Data Units) in the file and similar problems will arise in any format where
the data are encoded as a serial stream of data units.  

The problem is avoided if in-place modification is forbidden and every change
requires a new file to be written.  This is not always practical.

It can also be avoided by splitting the FITS file into its component parts
(but then it is no longer a FITS file).  The idea of using a disected version
of FITS as a common working format for AIPS, IRAF etc was floated about 3
years ago but didn't seem to get anywhere.

	Chris Flatters
	cflatter at nrao.edu