Subject: Use of FITS for real-time data logging

From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::[email-masked]>
Date: Fri, 20 Oct 89 07:02:56 EDT

Many thanks for your comments on real-time use of FITS. Before I
reply, I'd like to make a general observation. FITS has been
tremendously successful in easing the problems of data interchange,
and for that we must be eternally grateful to its originators. My
comments are not intended to point out deficiencies in FITS (in the
context of its original function) but to suggest a simple extension
which enables it to be used efficiently for an additional function:
that of data logging. 

Like most observatories, we have guest observers who bring their own
favourite data reduction packages, or want to take the data back with
them. At present it is necessary to transcribe from one of a number
of 'internal' formats to FITS or some other 'external' format. We are
proposing to rationalise our internal formats, and in fact FITS with
the extension I propose makes for a very convenient internal format.
I agree that there is in principle no reason why you shouldn't go on
generating multitudes of various different ad hoc internal formats at
various observatories, except that it's very wasteful of precious
manpower in inventing the formats, writing transcription programs,
and transcribing data. 

Having said all that, my specific comments on your replies are as
follows: 

stephane> FITS is a Flexible Image TRANSPORT System, NOT an all purpose data
stephane> format. Use it for what it is.

Sorry, I disagree. It could, with a small modification, be even more
useful. 
 
RN> i) You record the data on disk in an intermediate format, and only write
RN> it to tape at the end of the observation. But this defeats the point
RN> of using FITS as an on-line format. You're effectively using the
RN> intermediate format as your on-line format, and then later using
RN> valuable cpu cycles to transcribe the data into FITS.
stephane> Comment :
stephane> Cpu cycles concern only the Header, not the data because here a mere
stephane> copy is required. But you are losing many more cpu cycles in your
stephane> data reduction system formatting and unformatting the FITS header
stephane> later on. Are they less valuable ?

 I think we have to distinguish between two cases: (i) recording the
data  in one formnat, and then using an offline program to
transcribe, in  which case your comments apply, but see my comments
at the start of this mail.  (ii) using the observing program to
transcribe, in which case - yes cpu  cycles here are more valuable
than offline cycles! In the real-world the  real-time logging
computers tend to be under-powered, and tend not  to have the
number-crunching capacity of off-line processors, which can  always
spend a little longer than real time if a particular data set is 
particularly voluminous. 

RN> ii) You record each integration as a seperate FITS file. This works, but
RN> (a) you end up with a tape full of small files - messy!, and (b) a large
RN> fraction of the tape will contain redundant header information.
stephane>This is not efficient, but it works. You could add a packing program
stephane>later in your data processing stream.

 I agree - but's it's still messy and inefficient!

stephane>Your problem is not specifically related to FITS. It happens in
stephane>any acquisition system in which the data size must be written
stephane>but is not a priori predictable. I believe you missed the "classical",
stephane>easy to implement, yet standard and general solution, because you
stephane>reason in terms of TAPE (which is a purely sequential device) instead of FILE.
stephane>If you write your data on DISK first instead of directly on tape,
stephane>then it is easy to update the first 2880 bytes FITS block putting
stephane>the right GCOUNT when the scan finishes. The tape copy would be
stephane>done only when the file is closed. If you want to be safe against
stephane>computer crashes, you could update this header each time a new
stephane>information is written. I assume that your disk drives and system
stephane>are reliable enough, so that real-time tape copy is not absolutely
stephane>necessary.
 
Writing to disk is fine for most present systems, but becomes
impractical as data volumes get really large ( to take an extreme,
non-single-dish application, the AT produces data at a max rate of
about 1 Gbyte/hr. Future high-resolution multi-beam single dish
systems with rapid scanning may also approach this data rate). Here
you need either tape or optical disks, which are generally WORM, so
you can't easily overwrite the header. And you don't can't get round it
for WORM drives by leaving the writing of the header till later, 
because then the system becomes fragile to machine crashes.

stephane>There is no need to change anything in the FITS standard to answer
stephane>your request.

I agree that there is no NEED to have a universal format - we can
certainly continue with a proliferation of ad hoc internal formats if
thats what the community wants. But the same arguments that led to
the birth of FITS can be extended: the astronomical community is
generally under-manned - why make life difficult when with a minor
change it could be so easy! 
 
stephane>The success of FITS is largely due to its virtue of STANDARD, and
stephane>its original simplicity. Some people would argue that it is not
stephane>Standard or simple enough, and propose RITS (RIGID Image Transport
stephane>System) instead. I do not follow this opinion: up to now,
stephane>no unnecessary feature has been added. Instead, I believe we
stephane>should be carefull not to let FITS become RITS (RANDOM Image
stephane>Transport System) !

Yes but let's not allow it to become OITS (Obsolete Image Transport
System)! 
 
In response to Don's comments about RAM etc, I guess we should look
very carefully at data rates. The AT rate cited above is an extreme
case, and may appear not to be relevant to the single-dish problem.
However, I believe we should be looking ahead, rather than coping
with problems that confront us NOW. Multi-feed,
high-spectral-resolution surveys may bring the hardware to its knees
(as Greenbank has already demonstrated - oops sorry - shouldn't have
said that!) and any standard designed now should be designed to last
for a few years. 

To summarise, my arguments are that (i) to use FITS for on-line
logging makes sense in terms of efficient use of manpower and other
resources, (ii) it can be achieved with present FITS by some form of
buffering, but that is somewhat inefficient and may get impractical
as data rates increase, and (iii) allowing the option of GCOUNT=-1
has no effect whatsoever on the speed, efficiency, and implementation
of present FITS. You guys who don't like it can ignore it, and write
out GCOUNT as at present, EXCEPT that the FITS reading programs must
be extended to accomodate the possibility that they may encounter
data with GCOUNT=-1. 

Cheers,
Ray Norris