Subject: those trailing zeros in the last 2880 byte block

From: sla at helios.ucsc.edu (Steve Allen)
Date: 26 Sep 91 15:12:56 GMT

In article <1991Sep25.151241.1531 at murdoch.acc.Virginia.EDU> gsh7w at fermi.clas.Virginia.EDU (Greg Scott Hennessy) writes:
>                  The original FITS papers considered this point, and
>conceded that not all reader programs can read every FITS file, but
>the programs should at least exit gracefully. I have encounted data
>sets that were called FITS files, that did not have the last record
>padded out to 2880 octects, and some FITS readers die horribly at
>this. Blocking out the last record to 2880 octects is not very
>wastefull of space, at least for modern data sets
>-Greg Hennessy, University of Virginia

First, to make something perfectly clear:  When you observe at Lick Observatory
and bring home a magnetic tape (9-track or exabyte) written by the data
acquisition system, that tape always has physical records which are
multiples of 2880 bytes long.  The last record after the image data is
padded with zeros.  So Lick generates conforming FITS tapes.

However, most images spend at least some time on disk during the observing run,
and during this time they are usually manipulated by an image processing
package (namely Vista*).  The images on the disk are not written with the
zero padding to fill out the records to 2880 bytes.  They simply terminate
immediately after the last byte of the image.  (This is a Berkeley Unix
system).  Vista is not bothered by this.

It has been becoming more and more common for observers to transfer their
images via methods other than magnetic tape (namely ftp--Lick does not
officially endorse this yet.)  When such an image is then read by IRAF's
rfits, IRAF chokes and fails.  I do not consider this to be exiting
"gracefully."  I have suggested to these observers that they ask NOAO
to fix IRAF to be "graceful" in this case.

Lick is not opposed to changing the data acquisition system such that it
writes out the extra zeros.  This will cost some disk space on a disk
which is already too small.  Furthermore, when dealing with the images
>from some of the new (and small) IR array detectors, the proportional
increase in size is not insignificant.

We would welcome opinions on the importance of the zero-padding to 2880,
along with reasons that explain why not having those zeros is bad for
FITS readers other than IRAF.

(*) P.S.  Vista is available for most modern computers via anonymous ftp.
Contact Jon Holtzman (holtz at lowell.edu) for more details.

_______________________________________________________________________________
Steve Allen          |                                |   sla at helios.ucsc.edu
UCO/Lick Observatory |     This space for rent.       | If the UC were opining,
Santa Cruz, CA 95064 |                                | it wouldn't tell me.

---

From: bschlesinger at nssdcb.gsfc.nasa.gov (Barry Schlesinger)
Date: Fri, 27 Sep 1991 20:16:00 GMT

In article <21295 at darkstar.ucsc.edu>, sla at helios.ucsc.edu (Steve Allen) writes...
>				...
>				...

	The mandated logical record (hereafter record) size for FITS
is 2880 bytes.  FITS readers take advantage of this requirement by
picking up the data sets 2880 bytes at a time, particularly for media
other than tape and electronic transmissions where there is no
indicator of an end of record.  So, to begin, the software has to be 
able to handle a record of a different length and distinguish the end 
record from an error.  That's one complication.
	Now suppose someone gets this data set and adds extensions 
after it.  For standard FITS, the software need only check the 
beginning of each 2880-byte gulp for the XTENSION keyword and type 
name.  If the data are not padded, then the reader has to be prepared 
to look for the XTENSION keyword elsewhere.  That is a more complicated 
process than simply picking up 2880 bytes at a time.  A minimalist
FITS reader designed to take advantage of the 2880-byte unit specified
in the FITS papers will would not expect to have to make such a check.
	Standards are devised to provide a common ground for different 
users and systems.  As soon as a group starts letting a requirement 
slide because it is inconvenient, this common ground is lost.  As a
lack of fill is a violation of the FITS standard, as expressed in the
FITS papers, any FITS-checking software should at the very least flag
it.  Ours will.
				Barry Schlesinger
				NASA/OSSA Office of 
					Standards and Technology (NOST) 
				FITS Support Office