Subject: Comments on 'Binary Table Extension' draft

From: pence at heawk1.gsfc.nasa.gov ( William Pence)
Date: Thu, 3 Oct 1991 21:06:21 GMT

Comments on the "Binary Table Extension to FITS" Draft by Cotton and Tody
dated 20 September 1991:

For the most part, I have found that the binary table format proposed
in this document to be unambiguously defined and straight forward to
implement.  It appears general enough to be able to handle virtually
any type or format of data.  I was able to  support all the binary
table features in the FITSIO software package  except for the 'Variable
length array facility' which will be added in the near future.  I have
just a couple suggestion for changes to this draft:

1)  Sections 5.7 and 5.8 states:  'The IEEE NaN (not a number) values are
used to indicate an invalid number; ... All IEEE special values are recognized.'

I have 2 comments on this.  First, why we do we need more than one value
to represent an undefined pixel?  An IEEE NaN value is defined to have
all the exponent bits set to 1 and a non-zero set of fractional bits
which is a very large set of different possible values.  This makes it
more difficult for FITS readers, especially those written in Fortran,
to test if a pixel is undefined.  It would be very much simpler if FITS
were to adopt a single NaN value to represent undefined values; an
obvious choice would be to adopt a value with ALL bits = 1 (which is
equivalent to a value of -1 in 2's complement integer notation).  This
would have the advantage that the value has the same representation on
every type of computer (you don't have to worry about where the
exponent bits are on machines with swapped bytes)  and it would make it
much more efficient for FITS readers to test for undefined values.  (I
realize that the FITS definition for primary array states that any NaN
value may be used to represent an undefined value and it may be too
late to change this now.  But this does not imply that we have to adopt
the same convention for the binary table extension).

Secondly, what does it mean that 'All IEEE special values are recognized'?
If a pixel has the IEEE value representing + or - infinity, for instance, 
what is a FITS reader supposed to do with it?   These special values are 
not supported on all machines, so the FITS reader can't simply pass them
on without possible loss of information.

2)  The first paragraph of section 5 states: 'The last 2880-byte record
should be zero byte filled past the end of valid data.'  I believe that
this is an overly strict requirement, and is in fact unenforceable. 
Are FITS reading programs supposed to read these bytes to be sure they
are zero, and if not, refuse to read the rest of the FITS file??  I
would suggest modifying this sentence to state: 'The value of any bytes
in the last 2880-byte record past the end of valid data are undefined
and FITS reading software should make no assumptions as to their values.'
This is a rather trivial objection for static FITS files, but is more
important in applications where a FITS file is used in a dynamic real
time environment where the contents are frequently modified (e.g.,
in a database type application).  It might impose needless overhead
on the software if it has to explicitly reset these bytes to zero
any time the length of the table is shortened.

3)  The biggest problem I have with the current Binary Table draft has
to do with the way character fields are handled.   Basically this
standard treats these as arrays of single characters rather than as
strings.  I believe that this is a too primitive level for most FITS
applications which would prefer to treat a string of characters as a
single unit.  This becomes very important when one has to deal with
arrays of strings (and not single characters) in a table.  It is
perhaps clearer to illustrate the problem with an example:

Suppose a table contains a column with TFORM = '20A', i.e, each row of
the column contains 20 characters.  Now suppose a user calls a fortran
subroutine (in the FITSIO package for example) to read values from this
column.  The user passes a character string array to the subroutine
(for the sake of argument, suppose the user has declared the character
array as CHARACTER*10 CARRAY(50)) and the user expects the subroutine
to fill up the array with the first 50 strings contained in the FITS
table column.  But the subroutine cannot do this because it only knows
that there are a total of 20 characters in each row, but has no idea if
these should be interpreted as a single 20-character string, 2
10-character strings, 4 5-character strings, etc.  Thus, the subroutine
cannot pack characters from the FITS column into the user's Fortran
string array because it does not know the length of each FITS string.

The draft document proposes a way to solve this problem by defining
another keyword in the header, e.g.,  TDIM = '(5,4)',  to signify that
the 20A column is to be interpreted as 4 substrings of 5 characters
each.  But I object to this on 2 grounds:  first, this proposed
convention for the use of TDIM is only defined in Appendix A and is not
part of the formal binary table definition and thus is not binding on
the creators of FITS files.  FITS readers then cannot be sure that the
users will always adopt this convention.  The second, and to me more
compelling objection is that a subroutine interface like FITSIO does
not need to look for a TDIM keyword for any other column data type
except for ASCII characters.  For all other datatypes the TFORM keyword
itself is sufficient for the FITS reading subroutine to read the data.
For example, if the user wants to read a column of double precision
numbers, the reading program only needs to look at the TFORM keyword
(e.g, TFORM = '10D') to know how many values are contained in each row
of the column.  It is only in the case of ASCII strings that the
subroutine has to go hunting around for more information in order to be
able to interpret the data.
(This is analogous to the situation if you were asked to read  a column
of integer values, which contained 4 bytes per row, but you did not
know whether the the column contained 4 1-byte integers, 2 2-byte
integers, or 1 4-byte integer in each row).  Note in passing that in a subroutine interface like FITSIO, one does not need to worry about the
higher dimensionality of the data in the FITS files;  as far as the
FITSIO interface is concerned, every FITS array or table column can be
interpreted as a one-dimensional array of data;  it is the
responsibility of the calling program to group the data into higher
dimensions, if required.

As a better solution to this problem, I propose that the TFORM value
for ASCII columns be changed to the form:  TFORM = 'rAw'  where 'w' is
the length of a unit string, and 'r' is a repeat count.  For instance,
'3A5' would indicate that the field contains 3 strings each 5
characters in length.  An exception to this general rule would be the
case where no width is specified, as in TFORM = '20A'.  This would be
interpreted as if it were written 'A20', i.e., that this is a single
string 20 characters in width and not 20 1-character strings.

This modified form for TFORM is also the same is is used in FITS ASCII
table extensions, except for the fact that ASCII tables columns cannot
have a repeat count.

This new convention for the TFORM keyword has already been implemented
in the version 3.xx FITSIO software for anyone who wants to try it
out.  This is of course completely unofficial, so users should use this
feature sparingly for any real data that is to be distributed to other
users.

4) I do not care much for the ability to terminate a character string
before its explicit length by an ASCII NULL character.  This feature is
useless in Fortran implementations, and just requires more overhead
to test for NULL characters, and then pad them out with blanks.  Using
a NULL in the first character to represent an undefined string is fine
though.

-Bill Pence, HEASARC    pence at tetra.gsfc.nasa.gov  or  LHEAVX::PENCE

---

From: dwells at fits.cx.nrao.edu (Don Wells)
Date: Fri, 11 Oct 1991 05:35:26 GMT

In article <pence.686523981 at heawk1> pence at heawk1.gsfc.nasa.gov
(William Pence) writes:
 WP> Date: Thu, 3 Oct 1991 21:06:21 GMT  
 WP> Comments on the "Binary Table Extension to FITS" Draft by Cotton
 WP> and Tody dated 20 September 1991:  

 WP> ... Sections 5.7 and 5.8 states:  'The IEEE NaN (not a
 WP> number) values are used to indicate an invalid number; ... All
 WP> IEEE special values are recognized.' ... An IEEE NaN value is
 WP> defined to have all the exponent bits set to 1 and a non-zero set
 WP> of fractional bits ... difficult for FITS readers ... written in
 WP> Fortran, to test if a pixel is undefined... simpler ... adopt a
 WP> single NaN value ... ALL bits = 1 ... equivalent to -1 in 2's
 WP> complement ...  

This BINTABLE NaN rule is consistent with the NaN rule of the Floating
Point Agreement which became effective 01-January-1990. The motivation
of the NaN rule was to allow IEEE machines to use their hardware
support to propagate NaNs automatically.  The majority of computers
used in astronomical computing today support IEEE FP.

The usual solution to the problem that standard Fortran is unsuitable
for systems programming is to resort to a set of machine-dependent
utility subroutines ("Z" routines in the parlance of IRAF and AIPS).
This problem does not arise for C programmers because the standard
language includes shift and mask operators.

 WP> ... what does it mean that 'All IEEE special values are
 WP> recognized'? If a pixel has the IEEE value representing + or -
 WP> infinity, for instance, what is a FITS reader supposed to do with
 WP> it?   These special values are not supported on all machines, so
 WP> the FITS reader can't simply pass them on without possible loss
 WP> of information.  

IEEE machines support +/-Inf in their hardware and libraries(?). Most
data analysis systems do not exploit this fact, but I contend that
they should. My opinion is not just theoretical, I put it into
practice in the IPPS [Interactive Picture Processing System] which I
and others built at KPNO during ~1974 to ~1980, and which ran in
production until May 1985.  I operated the CDC-6000-series CPU in the
mode in which its hardware and its library routines would calculate
and propagate infinities by the formal mathematical rules.  In the
IPPS when you took the logarithm of an image the zero pixels produced
-Infs, and when you exponentiated that image you got your zeroes back!
The arctangent of +Inf was indeed pi/2, etc.  When the IPPS had to
convert floating to integer for output it would interpret Infs as
NaNs. This is just what a FITS writer should do today when writing
floating numbers in an integer format.  When reading floating FITS
data on a non-IEEE machine I recommend that the Infs and NaNs be
construed as "blanks".  If your datasystem does not support the
concept of "blanks" you will have to make some arbitrary decision.

Incidentally, Crays support NaNs and Infs, even though they are are
not IEEE machines. 

 WP> 2)  The first paragraph of section 5 states: 'The last 2880-byte
 WP> record should be zero byte filled past the end of valid data.'  I
 WP> believe that this is an overly strict requirement, and is in fact
 WP> unenforceable. ...  

This is consistent with the zero-padding rule of the Basic FITS
Agreement of 1979. I view the rule as a matter of neatness, of
craftsmanship. I can imagine some programmer new to FITS examining a
hex dump of a file and getting very confused when the data values in
the last logical record do not end at the expected place, and indeed
appear random -- or worse, display some pattern left over in the
memory of the writing program. The zeroes will reassure hir...

The draft NOST standard document specifies the zero-padding rule in
Sect.4.3.2 (Primary Data Array) and specifies ASCII-blank-padding for
the TABLE extension in 8.1.3 (Data Sequence). 

I myself am reluctant to relax these rules because of my neatness
motivation, but I do concede that you have a point when you say that
they are "overly strict requirement[s], and [are] in fact
unenforceable".
--

Donald C. Wells             Associate Scientist        dwells at nrao.edu
National Radio Astronomy Observatory                   +1-804-296-0277
520 Edgemont Road                                 Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N