Subject: Use of FITS for real-time data logging

From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::[email-masked]>
Date: Thu, 19 Oct 89 01:32:07 EDT

Gday all
Since nobody from the Australia Telescope will be present at the 
Greenbank meeting, I'll be sending a written presentation instead. 
Before I do so however, I'd like to invite comments on an issue 
which hasn't so far been raised explicitly in this particular forum.

The issue is this: we would like to use FITS for logging data in real time,
thus avoiding the overhead of transcribing, and minimising the
proliferation of formats. But FITS generally requires the number of data 
to be entered in some form (e.g. GCOUNT, NAXISn) in a header prior to 
writing the data. However, in real life when observing you don't know in
advance how much data you'll be writing. You may, for example, plan to
do 100 integrations on a source, but then have to abort the observation half 
way through because of high wind, a hardware failure, or simply because
the on-line monitoring tells you it's not worth continuing. So, you want
to record the existing 50 integrations. If you are using FITS on-line,
the ways of dealing with this situation are:

i) You record the data on disk in an intermediate format, and only write 
it to tape at the end of the observation. But this defeats the point
of using FITS as an on-line format. You're effectively using the 
intermediate format as your on-line format, and then later using
valuable cpu cycles to transcribe the data into FITS.

ii) You record each integration as a seperate FITS file. This works, but
(a) you end up with a tape full of small files - messy!, and (b) a large
fraction of the tape will contain redundant header information.

iii) You argue that it's a rare event not to finish the scan, and in those 
rare instances you simply pad out the FITS file with blanked data so that
the data match the header. But this is impractical if you've started a
6-hour scan and decide to abort after one hour. Do you throw away the
1 hour of data or do you watch your tape going round for the next few hours?

This problem has been the principal reason that the AT is now using
(yet another) adhoc format (RPFITS) rather than true FITS for data logging.
This is a shame, as the use of standard FITS would be a much more 
elegant solution, and would remove the need for subsequent transcription.

I would like to invite comments on the proposal that this problem
could be circumvented by allowing GCOUNT etc to be set to -1, which 
means that the length of data is unknown. In this case, any FITS-reading
program skipping through the data will simply
have to read laboriously through until it sees a block of data
starting with a recognisable keyword (this approach is used by RPFITS 
and it works). Of course for normal data interchange between sites
a positve value of GCOUNT would be entered as normal. This extension
of FITS can therefore sit quite happily alongside the existing FITS files,
but opens up FITS to be useful for on-line logging as well as its
original function of data interchange.

Regards,
Ray Norris

---

From: [email-masked] (Don Wells)
Date: Thu, 19 Oct 89 10:22:17 EDT

I would like to add one more comment to Stephane's response to Ray's
issue. That is that in addition to using disk as an intermediate 
storage medium one can also use RAM. In the past this was not
practical because RAM was too expensive and data acquisition machines
never had enough RAM. This situation is now changed completely.
RAM is *very* cheap now, and current state-of-the-art real-time CPUs
can easily have large amounts of it. For example, the VLBA Correlator
has Motorola MVME147 CPUs. These are two VME slots wide (the latest
model of the 147 is only one slot wide) and have a 25 MHz 68030 (5 MIPS),
68882 FP coprocessor, Ether and SCSI controller, and 8 MB of RAM. NRAO
is able to buy these for US$4200. It would be easy to make a competitive
procurement of one or more VME cards of additional RAM if we needed it.
The point is that it is now easy to have I/O buffers that are quite large,
to organize data in memory and update the header just before writing to
the device. The data can be in files of a few megabytes, each with a 
proper header. I am not necessarily saying that this approach is preferable
to constructing larger files on a scratch disk, as Stephane suggested,
I am merely pointing out that it is yet another option for designers.

A small technical point about using disk as a buffer is that it can
boost performance of a streaming tape device by avoiding stop-start
overhead at intermediate data rates. One must be careful that aggregate
bandwidth on the backplane can support concurrent I/O from device to RAM,
RAM to disk, disk to RAM, and RAM to tape. At sustained rates of 200 kB/s
this is probably not a problem (assuming independent DMA controllers), 
somewhere above 500 kB/s it will start to be a problem.

-Don