Subject: Use of FITS for real-time systems

From: [email-masked]
Date: Thu, 9 Nov 89 15:55:25 EDT

A SUGGESTION FOR USE OF FITS FOR REAL-TIME SYSTEMS

                    Eric W. Greisen, NRAO
                       November 9, 1989

     The fundamental FITS formats have often been considered for use
in real-time systems, but have generally been rejected.  In some cases,
FITS-like formats have been used, which incorporated some of the
strengths of the FITS conventions, but which undermined the general
interoperability of this IAU-supported data interchange format.
During the meetings in Green Bank, convened to discuss radio single
dish use of FITS, a scheme to support real-time systems with a
standard FITS format became evident to me and I described it briefly.
I am expanding that idea here to provide a document for discussion
and wider distribution.

     FITS has many advantages for real-time systems.  It is reasonably
self documenting, which gives users of the output data who are divorced
from the telescope itself by reasons of politics, distance, or time a
reasonable chance to determine what is on the tape and what may be done
to read and translate it.  FITS allows for the association of diverse
information through its character headers and through the use of special
records, which may be of standard types or, if absolutely necessary, of
types of the telescope programmers' own devising.  The association of
these collections of data is done by the position of tape end-of-file
marks; all data between EOFs are associated.  A new Fits-based real-time
tape format may be read immediately or with only a few modifications by
one or more of the generalized FITS readers which are now widely
distributed around the globe.  Blocked FITS tapes make very efficient
use of the recording medium.

     Unfortunately, the original FITS formats have a "fatal" flaw.
Both the basic format and the random groups format have the requirement
that the main data matrix or collection of matrices must follow the main
header immediately.  Any special records, such as tables describing the
source and instrumental parameters, must follow the main data.  Further,
the size of the main data must be fully specified in the character
header.  Thus, one is vulnerable to tape parity errors, telescope program
and power failures, and the like.  If these altogether too common problems
arise during the writing of the main data, then the essential control
tables are not written and the data that were recorded are essentially
useless.  Furthermore, the size of the main data is not necessarily
known a priori.  The exact number of samples may be determined by
accidents of timing, on-line data editing, or the users' decision to
terminate or extend the observation.  Again, the problem of placing
the critical control tables after the data causes serious difficulties.

     Therefore, I suggest that real-time systems use the binary tables
extension to FITS suggested by Cotton, not only for the control tables,
but also for the main data themselves.  The description and structure
of these real-time data tables should be determined by the ideas put
forward by Wells and adopted for use in the radio single-dish case.
The main header of a tape file would probably be fairly terse and be
used mainly to specify that the main data matrix is null (NAXIS = 0).
It could also provide a variety of documentation of the format
conventions and general history of the particular observations.  Then,
all control tables would be written following the standard conventions
for FITS extension tables.  Presumably the binary tables version would
be selected by the efficiency-conscious programmers of real-time systems.
Other extension files would then be written, also following the rules of
the generalized extensions of FITS.  Finally, one would write the main
data as a binary table.  Each row would contain one sample of the data
array together with any conformal arrays needed to describe those data
(such as a matching weighting array) and any other time-varying
parameters needed to describe those data (such as time, source ID,
observation coordinates, system temperatures, flagging, or whatever).
The use of the binary tables format allows all of these parameters to
be in natural forms (i.e., character, integer, floating, double precision,
bit array).  On a computer that uses standard byte order and IEEE
floating-point, the FITS writer would only need to copy the data from
internal data arrays into the n times 2880-byte tape IO buffers.  Less
standard computers would have to do some translation of the bits, but
that should not be too expensive.

     As given so far, this suggestion solves the problem of losing
control information when the data-writing is interrupted.  It also
improves the flexibility of the binary format over that allowed in the
basic and random-groups FITS formats, since in those all data must be
of a single numeric binary type.  However, the binary tables format
still requires the specification of the file size in advance.  Two
conventions --- not rules --- need to be agreed upon to resolve the
remaining difficulties.  The first is the understanding that no
important files are to be found after the main data table; that normally
only the tape end-of-file will follow that table.  Then, the corrective
response to a premature end-of-file is obvious.  One simply keeps all
the data previously read, discarding (or marking as uncertain) only
those tables rows which came from the last tape record prior to the
unexpected EOF.  The second understanding is that real-time systems
will normally write data tables of modest size, so that the observation
would occupy more than one table under many circumstances.  Then, most
data tables would be completely filled to the predicted size and the 
cost of filling the last such table with blank rows, rather than a
premature EOF, should be modest.  In this way, no "error" would be made
in reading them.

     I suppose that the only argument against writing many of these
data tables in a single tape file is the problem of parity errors.
Since there is no accord between the internal logical structure of a
data table and the physical blocking in FITS, all data following a
parity error within the current table may be lost because of that
parity error.  There are recovery mechanisms that will work under
many circumstances.  Following a parity error, the FITS reader can
check the first eight bytes of each logical record for the keyword
XTENSION signalling the start of a new table.  Systems which require
still greater certainty of data integrity could write their data
tables with synchronization columns at the beginning of each row,
namely a column with characters such as the value assigned EXTNAME
and a column with the row number.  Following a parity error, the
reader would search each logical record for this pattern and recover
its reading of the data table knowing how many rows had been lost.
In general, some compromise between writing the headers and control
files only once and the danger of parity or other device failures
should dictate the size of the individual tables and the number
written in a single tape file.  Advances in recording technology,
such as the new magnetic and optical cartridge devices, will probably
reduce this problem to a minimum.

     I note that random-group FITS has seldom been used outside of the
radio-interferometric community, and that the ideas above provide a
better solution for that community than the random-groups format.  I
suggest that we switch to that format in radio interferometry both for
real-time and post-observation systems.  Needless to say, AIPS would
have to support both forms forever including being able to write
random groups as well as tables.  Furthermore, I suspect that AIPS will
still treat the uv data as random groups internally, since the
differentiation in the internal forms would be small.