Subject: Units in FITS and the NOST Document

From: egreisen at primate.CV.NRAO.EDU (Eric Greisen)
Date: Thu, 21 Jul 94 15:50:31 EDT

Units and the Proposed NOST Standard for FITS

Background
----------

	FITS was designed with the philosophy that data transport
reliability is enhanced if the number of options describing a data set
are kept to a reasonable minimum.  This "Occam's razor" philosophy was
applied, for example, in the choice of only 3 binary formats
originally allowable under FITS (8-bit unsigned and 16-bit and 32-bit
signed integer).  It was also applied in another crucial area:  data
units.  FITS deliberately adopted a minimalist approach to units to
simplify the standard and allow greatly simplified data transport.
The correctness of this approach is suggested by the wide acceptance
of FITS in the scientific community.

	The purpose of this document is to consider how the proposed
NOST standard deals with units, and to outline how the proposed NOST
standard should be changed in this area to enhance the usability of
the standard.  This enhancement will at the same time minimize the
costs to the astronomical community of using the NOST standard.  In
brief, the proposed NOST standard needs to be strengthened in the way
it deals with units.  If it is not, then it is in danger of
promulgating a standard which imposes an excessive burden on data
consuming sites, and could result in a standard which is significantly
less useful than it otherwise might be.

Philosophy of a Data Transport System
-------------------------------------

	The primary purpose of the FITS standard was to allow
straightforward sharing of scientific data, attempting to communicate
not only the data bits, but also the meaning of those bits.  In
several areas, restrictions were used in the standard to provide a
balance between the ease of writing FITS data and the ease of reading
FITS data.  In the selection of supported numerical formats, for
example, FITS chose not to do the "general" thing, which would be to
allow any number of bits with some keyword to define signed/unsigned
or one's versus two's complement data storage.  Such generality is not
NEEDED to transfer the image data.  Instead, some burden is placed,
_where it belongs_, on the data-producing site which understands fully
its internal formats and can easily provide a unique translation to a
well-defined external format.  A fully general system removes any
burden from the data producing site and places all the burden on the
data consuming site.  The latter would not know what to expect and
hence would have to attempt to provide generalized FITS readers that
anticipate all possible combinations of the allowed parameters.  In
effect, a fully general standard is no standard at all.

	The important point is that data communication will fail if
the format allows so many combinations that they cannot be reasonably
anticipated IN ADVANCE.  If a proposed standard is too general, data
consuming sites we will then be forced to write innumerable
specialized FITS readers:  one for IRAF, another for high-energy, a
third for Midas, and so on.  If this situation occurs, the advantages
of the standard will largely have been lost.

	The problem of the communication of data from the data
producing sites has increased substantially since FITS was first
proposed.  Any change in the standard forces all data consuming sites
to change their FITS readers or to be "left behind."  These sites are
not just those fully serviced by a few standard packages such as AIPS,
IRAF, and FITSIO, but also numerous other packages involving tens or
hundreds of programmers distributed all over the world (and over the
years as well).  We should not add complexity to the format for the
sake of generality and we should NEVER compromise our ability to
transmit data simply to make FITS more appealing for use as a format
internal to current software packages.  We should only add
complexities when they are TRULY needed to convey the data and their
meaning.

Coordinates and Units in FITS
-----------------------------

	The original FITS paper skirted the issue of coordinates to
avoid controversy which could have caused the community to reject the
entire format out of hand.  There was also concern over making errors
in specifying the coordinates.  While these concerns may have been
valid, the original FITS document should have provided more guidance
on how to convey MEANING along with image bits.  As a result of the
lack of clear specification in the standard, many FITS data sets have
been written with a diverse set of coordinates, even when the keywords
from the original papers were used.  This is unfortunate, because
doing science with accurate image bits and no coordinate information
is problematic at best:  To be told that the coordinate is 10 with an
increment of 2.345679 is of no value whatsoever without the
information that these are in meters!  (For simplicity, I will use
distance in all axis comments.)  To try and avoid this situation, the
original FITS paper stated, unambiguously,

    "     IV. Units
          A. Basic set
               1. Consistent with the International System of Units
               2. Add degrees for angles
               3. Add Janskys for flux
          B. Examples
               1. Distances in meters
               2. Masses in kilograms
               3. Times in seconds
               4. Temperatures in Kelvins "

Note that the SI system of units for distance is meters alone, not
millimeters, Megameters, etc.  The choice not to include prefixes was
a simple application of Occam's razor - since we allow exponential
formats (and usually scaling parameters), there is no NEED for
centimeters when meters are available.  The presence of a dozen or
more prefixes makes any attempt to determine units fraught with
potential for failure.

Coordinates and Units in the Proposed NOST Standard
---------------------------------------------------

	The proposed NOST standard has committed a serious error by
not addressing the subject of units and imposing sufficient standards
on units which may be used.  It does not grandfather the original
units into the standard, nor does it provide a replacement.  Instead,
it states "units defined in the IAU Style Manual are recommended".
Thus, instead of defining some sort of units, the NOST document
RECOMMENDS (which all readers interpret as providing the license to
ignore) the use of IAU units which include 17 versions of meters!
More seriously, the NOST document does not provide any mechanism by
which responsible FITS writers may document which units have been
selected.  (Note that the IAU Style Manual is a set of recommendations
for publishing manuscripts on paper for humans to read.  It is not a
standards document, nor does it address issues of computer to computer
communication.)

	The importance of the NOST documents, and the importance of
providing a clear, unambiguous standard, must not be underestimated.
Programmers are reading the NOST documents, correctly interpreting
words like "recommend" to mean "do what you like", and writing tapes
which are spreading throughout the community.  Any errors or omissions
in the NOST standard may quickly become "extensions" to the standard
simply by occurring on a large number of tapes.  For example, because
of an error in an early but well circulated DRAFT of the NOST
document, a major software package began to (and may still) write
tapes with the value field of a floating keyword surrounded with
quotation marks.  AIPS users now expect AIPS FITS readers to interpret
this aberration correctly for evermore.

     How do we improve the NOST standards?  After some considerable 
reflection, I propose the following:

(1) We create another keyword called CUNITn to describe the units
along each axis.  To assume that the CTYPE can imply the units
uniquely, even if only a pure SI is used, is naive.  If CUNITn is not
given, then a pure SI unit consistent with CTYPEn is the default.

(2) We change every place that units are mentioned in the NOST
document to refer to a new section of the document which will define
the limited set of units which will be allowed.  NOST compliant data
sets will use those units and only those units.

(3) In the new units section in the NOST document, we define:

     (a) units (e.g., values for CRVALn, CDELTn, TSCALn, et al.) are
always double-precision so questions of underflow and overflow do not
arise.

     (b) units are pure SI without character or numeric prefixes (how
can we require people to change from miles and mils to meters if we
refuse to change cm to meters?

     (c) some extensions to the units are allowed, such as degrees,
parsecs, atomic units, barnes and the like.  I prefer not to include
simple extensions like "Hz" since (Occam's razor) "/s" is so simple.

     (d) the grammar of the units string be defined in a manner like
that espoused in the document by Ian George and Lorella Angellini
entitled "Specification of Physical Units within OGIP FITS Files".
(My copy was dated 21-May-93; is there a more recent one with fewer
ambiguities?)  I would recommend strongly that a more restricted
grammar be defined to forbid the constructs those authors deprecate
but allow and to forbid "un-normalized" constructs (e.g. "m**4
m**(-3)" for "m" or "barn Gpc" which is about 3 liters).

     (e) state that some conventions on CTYPEn may be made which limit
the range of units still further (e.g., primary image spherical
coordinates may be restricted to degrees if the CTYPEn is stated using
the conventions of the Greisen/Calabretta proposal).  One reason for
this possible limitation is that we have to define a lot of math in
such conventions and the constants in the formulae depend on the
units.

     (f) that we explicitly allow even more extensions to the units
for ASCII tables which are representations of what appears in
journals, while binary tables and main headers are
computer-to-computer communications.

     (g) that we limit the length of units strings to some reasonable
number of characters (e.g. 40).

	Even with the simplifications to the full OGIP suggestions
given above, I do not expect most computer systems to understand units
all of the time.  Nevertheless, to define a reasonable string to be
echoed in all displays of units, would be a considerable service to
FITS users.  To allow almost any unit to be used, without
documentation, is a potentially dangerous position for the NOST
document to take.

Conclusion and Proposed Plan of Action
--------------------------------------

	The NOST standard needs to be strengthened in how it deals
with units, and needs to provide a clear framework within which
applications developers may work.  The proposal outlined above is an
attempt to provide a reasonable, workable compromise between the
desire to be able to use arbitrary units and the necessity of
providing a workable, well-defined standard.

	How should we proceed forward?  The following actions should be 
taken:

	(1)  Barry Schlesinger should be designated to suggest a
change, with all the necessary detail and exact language, in the way
the NOST document deals with units.  The NOST committee should then
consider and modify the suggestions as needed, using input from other
interested parties, in the manner already used to create the current
document.

	(2)  The official adoption of the NOST standards should be 
briefly delayed while these issues are resolved.  If this is not done,
it will create potential problems and delays in the eventual adoption
of the NOST standards by the IAU.

-----------------------------------------------------------------------
        Eric W. Greisen, Scientist
           National Radio Astronomy Observatory
           520 Edgemont Road, Charlottesville, VA 22903.
           egreisen at nrao.edu
-----------------------------------------------------------------------