Subject: Ray Norris issue

From: "GUILLOTE@FRGAG51" <[email-masked]>
Date: Thu, 19 Oct 89 08:43:03 EDT

Date sent:  19-OCT-1989 11:10:03.18
To:   DISHFITS@NRAO

Here are my own comments on Ray Norris issue of using FITS as an on-line
data format.
 
First general remark :
FITS is a Flexible Image TRANSPORT System, NOT an all purpose data
format. Use it for what it is.
 
RN> If you are using FITS on-line, the ways of dealing with this situation are:
 
RN> i) You record the data on disk in an intermediate format, and only write
RN> it to tape at the end of the observation. But this defeats the point
RN> of using FITS as an on-line format. You're effectively using the
RN> intermediate format as your on-line format, and then later using
RN> valuable cpu cycles to transcribe the data into FITS.
 
Comment :
Cpu cycles concern only the Header, not the data because here a mere
copy is required. But you are losing many more cpu cycles in your
data reduction system formatting and unformatting the FITS header
later on. Are they less valuable ?
 
RN> ii) You record each integration as a seperate FITS file. This works, but
RN> (a) you end up with a tape full of small files - messy!, and (b) a large
RN> fraction of the tape will contain redundant header information.
 
Comment :
This is not efficient, but it works. You could add a packing program
later in your data processing stream.
 
RN> iii) You argue that it's a rare event not to finish the scan, and in those
RN> rare instances you simply pad out the FITS file with blanked data so that
RN> the data match the header. But this is impractical if you've started a
RN> 6-hour scan and decide to abort after one hour. Do you throw away the
RN> 1 hour of data or do you watch your tape going round for the next few hours?
 
No Comment
 
RN> This problem has been the principal reason that the AT is now using
RN> (yet another) adhoc format (RPFITS) rather than true FITS for data logging.
RN> This is a shame, as the use of standard FITS would be a much more
RN> elegant solution, and would remove the need for subsequent transcription.
 
RN> I would like to invite comments on the proposal that this problem
RN> could be circumvented by allowing GCOUNT etc to be set to -1, which
RN> means that the length of data is unknown. In this case, any FITS-reading
RN> program skipping through the data will simply
RN> have to read laboriously through until it sees a block of data
RN> starting with a recognisable keyword (this approach is used by RPFITS
RN> and it works). Of course for normal data interchange between sites
RN> a positive value of GCOUNT would be entered as normal. This extension
RN> of FITS can therefore sit quite happily alongside the existing FITS files,
RN> but opens up FITS to be useful for on-line logging as well as its
RN> original function of data interchange.
 
Comment :
Your problem is not specifically related to FITS. It happens in
any acquisition system in which the data size must be written
but is not a priori predictable. I believe you missed the "classical",
easy to implement, yet standard and general solution, because you
reason in terms of TAPE (which is a purely sequential device) instead of FILE.
 
If you write your data on DISK first instead of directly on tape,
then it is easy to update the first 2880 bytes FITS block putting
the right GCOUNT when the scan finishes. The tape copy would be
done only when the file is closed. If you want to be safe against
computer crashes, you could update this header each time a new
information is written. I assume that your disk drives and system
are reliable enough, so that real-time tape copy is not absolutely
necessary.
 
There is no need to change anything in the FITS standard to answer
your request.
 
----------------------------------------------------------------------
A general, and personal, comment on FITS :
 
The success of FITS is largely due to its virtue of STANDARD, and
its original simplicity. Some people would argue that it is not
Standard or simple enough, and propose RITS (RIGID Image Transport
System) instead. I do not follow this opinion: up to now,
no unnecessary feature has been added. Instead, I believe we
should be carefull not to let FITS become RITS (RANDOM Image
Transport System) !
 
Computing power is no longer a problem, but programming power is still
one !
 
        Stephane