Subject: Some FITS tables suggestions

From: "Clive Page, Leicester Univ, UK" <CGP at STARLINK.LEICESTER.AC.UK>
Date: Thu, 24 Sep 92 14:43 GMT

Astronomers are using FITS tables, especially binary tables, for a wide
range of applications from star catalogues to photon-event lists.  As
William Pence has already noted, FITS is starting to be adopted as a
storage format in its own right and not just as a data interchange
format. Packages that provide database functionality and can handle FITS
tables already exist or are under development at several places, for
example: 

o  At GSFC the EXOSAT database system is being converted to run on Unix 
   and use FITS tables.
o  At ESO the MIDAS table system is being extended with more relational
   database functionality, and it can read/write FITS tables.
o  At GSFC William Pence is developing a set of tools to work on
   FITS tables (sorting, selecting, etc.).
o  In Italy the latest version of the Astronet DIRA2 database system can
   read and write FITS tables.
o  In the UK, the Starlink Catalogue Access and Reporting (SCAR) package
   is highly VMS-dependent, so a little work has been done here on a
   portable replacement.  This uses the FITS binary table as the main
   file format.  Tests of a prototype seem quite promising. 

Thus FITS tables are starting to acquire relational database software
packages built around them.  The FITS table structure seems quite well
designed for this purpose, especially as it allows one physical file to
hold not only the tabular data but also column descriptors and table
descriptors (epoch, equinox, etc). This makes for a nice modular system 
and avoids all the complications which bedevil most commercial DBMS
packages, which use something like a "common data dictionary" or "system
catalog" to hold all their descriptors, and therefore need a database
administrator with special privileges to take charge of the whole thing.
This way of working does not suit most groups of astronomers, who find 
it much better if their tables are entirely self-contained.

The software packages noted above fulfil different sets of requirements,
and data interchange between them is extremely likely.  But there is a
danger of minor but annoying incompatibilities creeping in unless 
we can agree fairly soon on a few more details of the FITS table format,
or at least on some conventions for the use of FITS keywords.

(1) Form of column names.
The FITS Standard (section 8.1.2) suggests that names should only
contain letters, digits, and underscores.  I would like to propose that
the first character should not be a digit, since leading digits make it
hard to parse command-lines or expressions which may contain column
names as well as numerical constants. 

(2) Column name length.
There seems no length limit on a column name (except that FITS keyword
strings cannot be more than 68 chars long). But section 5.3.2.1 says
that "proper interpretation of the FITS file should not require decoding
any more than the first eight characters of a character string".  Does
this limit column names to 8-chars or perhaps just require them to be
unique in the first 8 characters? I note that DIRA2 does indeed have an
8-character limit, but other systems listed above have more relaxed
limits, typically allowing names at least 16 characters long. 

(3) Sorting.
Efficient access to large tables implies keyed access of some type.
In practice most large FITS tables are sorted on some field, allowing a
binary search for a value or range of values. For example most of the
tables on the NSSDC/ADC CD-ROMs seem to be sorted on ascending RA, but
there is no way of determining this for certain without reading the
entire file. As some systems (such as SCAR) prefer sorting on DEC (to
avoid the wrap-around problem at 24 hours/0 hours), we cannot assume
sorting on any particular field even for an astronomical catalog (and
some contain celestial coordinates using two or more different
equinoxes, any one of which may be chosen as the principal sort field).
I think we need a convention for specifying the sort field and sense in a
FITS header. The simplest way would just be to name the field (or
fields) and insert a minus sign for a descending sort, e.g. 
TSORTnnn  = "-DEC"
or
TSORTnnn  = "SP_TYPE,-VMAG"
if there were minor sort fields in addition to the principal one. 

(4) Displaying Sexagesimal Notation.
Most astronomers still use sexagesimal notation (e.g. 12:34:56,
-2:34:56.7) for celestial coordinates.  In fact sexagesimals already
appear in many FITS tables but have to be treated either as a set of
three separate columns for hours/degrees, minutes, and seconds (and
sometimes a fourth one for the sign), or as a character string which may
(or may not) have spaces or colons between the components.  Neither of
these methods is very easy to use when one has to decode the numerical 
value for a calculation (or for keyed access). I wonder if anyone has
considered an extension to allow the TFORM keyword to specify
sexagesimal formats?  The obvious way of specifying this is with
pseudo-fortran forms like "HMSw.d" or "DMSw.d" where the ".d" is needed
only if the seconds value has a (decimal) fractional part. Obviously
these formats would have to be handled by special code, but in programs
written in Fortran-77 (the great majority, I would guess) this is
already necessary when handling format descriptors such as "Bw.d" or
"ENw.d" which are not built-in to the language.   The value in the file
could be assumed to be a decimal value in hours/degrees (so there would
be no change to the integer part only the fractional part would be
converted to minutes and seconds (or arcmins and arcsecs).  Systems that
could not handle the conversion could (as is permitted at present for
ESw.d and ENw.d formats) just display the degrees/hours as a normal
decimal number with the specified width and some sensible precision. As
an enhancement, however, one could allow the TUNITnnn keyword to
indicate different units (such as radians) and have an implicit
conversion to hours (HMS) or degrees (DMS). But then we get into
slightly tricky areas of automatic units conversion. 

(5) Indexing.
The prototype that I am building can create an index on any column; it
uses B+ trees to make indexing dynamic.  MIDAS and perhaps other systems
will soon include indexing.  Although it would be possible to pack
indexes into other data units in the same FITS file, it seems better for
extensibility for them to be kept in separate files. Software that can
handle indexes really needs to know which columns have been indexed when
the table is opened, so that any update operations are also reflected in
the indexes.  Thus some keyword is needed to indicate the presence an
index and perhaps its location.  These can, of course, be ignored by any
package that is not able to make use of indexes.  We could use some
standard naming scheme in which case just existence needs to be shown,
for example: 
TINDXnnn  =                    T
Otherwise it would be sensible to indicate the name of the external 
file, for example:
TINDXnnn  = "filename"

Clive Page
Dept of Physics & Astronomy, University of Leicester, UK.
cgp at star.le.ac.uk   (or SPAN  19838::cgp).

---

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger)
Date: Fri, 25 Sep 1992 15:10:00 GMT

In article <9209241343.AA19253 at cv3.cv.nrao.edu>, "Clive Page, Leicester Univ, UK" <CGP at STARLINK.LEICESTER.AC.UK> writes...

>				...

>unique in the first 8 characters? 

All the information in character strings to read the accompanying 
data; i. e. transfer it from the FITS file to the user's system in 
proper form, must be in the first eight characters.  Column names may 
be of more than eight characters.  Information required only for the
scientific interpretation of the data, not for reading it, may be
after column 8.  I recommend that the first eight characters be 
unique and selected to contain the most important information about
the column name, but the practice is not required.  In many cases, a
longer name may be needed to communicate the meaning of the column
heading. 

>if there were minor sort fields in addition to the principal one. 

Except, as earlier discussions in this forum have noted, there are 
occasionally problems with the transport of hyphens (minus signs).  
That is why letters, numbers, and underscore are the only recommended 
characters for the value of TFORMn, and it might be advisable to 
extend the practice to characters of other keywords.

>			...

				Barry Schlesinger
				NOST FITS Support Office

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Sun, 27 Sep 92 16:10:59 ITA

On Thu, 24 Sep 92 14:43 GMT  <CGP at STARLINK.LEICESTER.AC.UK>said:
>format.

 Although I am personally questioning the use of FITS for storage, I
 believe the points raised by Clive Page below concerning use of FITS
 files as databases AND MORE IN GENERAL about the need of conventions
 about keyword usage are very relevant EVEN IF (or actually A FORTIORI if)
 FITS files are used for interchange.
 (I am personally developing a different, more compact, but system dependent
 storage format, but I am keeping as a basic requirement that it MUST be
 one-to-one mappable with FITS : for this reason for instance I am forcing
 all limitations in FITS kewyords, like 8-char max names, etc.
 into my own header kewyords - with binary values )

>or at least on some conventions for the use of FITS keywords.

 YES, conventions are needed to avoid those incompatibilities, which can
 be very annoying.
 And I would say those conventions are needed beyond the area of databases,
 but, for instance, in the area of keywords describing the content of a
 photon list.
 At the moment I am using provisional, local conventions for the designation
 of columns for a photon list : for instance I call THETA and PHI the angular
 cooordinates of a photon (from the optical axis), X and Y the coordinates in
 micron on the focal plane, and XPOSITN YPOSITN the coordinates in detector
 pixels ...or I call ENERGY true energy in keV, but PHA energy in channels,
 and I ignore whether they are consistent e.g. with EXSAS or PROS - which
 I am told are not compatible among them)
>the first character should not be a digit, since leading digits make it

  I tend to agree

>any more than the first eight characters of a character string".  Does

 Usage of readable column names is definitely a bonus, and therefore
 names longer than 8 char could be useful. But in the spirit of "once FITS
 forever FITS" I am ready to live with 8 char.
 However I note that :
 a) FITS binary tables are NOT YET an official standard
 b) the names of the columns (i.e. the VALUE of a TTYPEn keyword) have a
    lower rank than the names of the keywords themselves |
    The keyword names are used by the FITS reader, but the values are
    used by analysis s/w
    Therefore the limitation of 8 chars is appropriate on names like
    TFORMnnn, TTYPEnnn, TUNITnnn but less on their content (and in some
    cases meaningless ... how do you limit to 8 char a VALUE like
    TUNITnnn = 'PHOTON/CM**2/S/KEV' ?

>(3) Sorting.

 I have no comments on this since it is far from the problems I am working
 on at the moment.
 Concerning however keywords of the form Txxxxnnn, which indicate some
 characteristics of the columns, it would be useful to introduce some more
 conventional (optional) keywords.
 At the moment I use TMINnnn and TMAXnnn to indicate the minimum and maximum
 value in column nnn.

>-2:34:56.7) for celestial coordinates.
> I wonder if anyone has

 I believe here there is a mix-up of two different things.
 BINARY tables are concerned about the representation of the values, therefore
 an angular value shall be best represented as a DOUBLE PRECISION value
 (column of type 1D or nD)
 Sexagesimal notation concerns not internal representation, but displaying
 of the values (TDISPnnn) ... unless one is talking of ASCII tables (which
 I believe will be soon replaced by binary tables)

 Therefore if a binary table COLUMN contains angular quantities, the
 easiest thing is to have them in degrees as a double precision floating
 point value. It is up to the program interpreting them to know how to
 display the numbers.

 The story is different if we talk about header keywords. Here it might
 be worthwhile to define new "types" of keywords for quantities like :
 angles (in degrees), angles (in hours), dates and times

 For instance, in the software I was mentioning at the beginning, I keep
 header information as follows :

 <type> <length> <keyname> <keyvalue>

 where keyname is the FITS-like name (8 char, blank padded), type and
 length are two 1-byte codes, and keyvalue is one or more binary values
 (in general)

 I support basic types like characters, INTEGER*4 (also INTEGER*2 but I am
 going to deprecate it), REAL*4 and REAL*8, but also other derived types
 like ANGLE (which has the same value representation as a REAL*8), and
 perhaps DATE and TIME (for DATE I am inclined to use an INTEGER*4 value
 in seconds since 1/1/1970 which is customary in Unix, while for TIME I
 have not yet made clear my mind).
 The software which uses these files, may use the type field to decide to
 format an ANGLE value in sexagesimal notation, or a DATE value in a
 notation like YYYY-Mon-DD-hh:mm:ss.ff
 To convert my files to FITS I use Bill Pence's FITSIO, and in general
 I use the type field to know which of the FTPKYx routines to call (for
 the basic types)

 Definition of additional kewyord types (for ANGLEs, DATEs and TIMEs) within
 FITS would be useful (and possible, since it will be an addition).

(*) Arrays of keywords

I wonder also whether one should allow somehow arrays of keywords. In my
binary headers the <length> field corresponds to the number of characters
in a string, and is generally 1 for numeric values, but it is possible for
it to be > 1, in which case <value> is a numeric array.
(I am not and I WILL NOT be supporting arrays of strings)
I was reluctant to introduce keyword arrays, but I was moved by a colleague's
request motivated by cases like the following :
let's assume we have an instrument with separate sub-units, like for isntance
7 PMTs (photo-multiplier tubes), of which we want to store the high voltage
setting.
My approach was to have 7 keywords PMTHV1 .... PMTHV7, but he insisted in
having instead a single keyword PMTHV, which is a 7-element array.
I told him I could support this, but regard it as private within his own
software. The main reason is that I do not know of an unique convention
of mapping them to FITS.
For demo purposes at the moment I am proceeding as follows :
I map my PMTHV keyword array to 8 (n+1) FITS keywords

  PMTHV  is an ASCII comment string saying the keyword is an arrau from
         PMTHV1 to PMTHV7
  PMTHV1 is the first element of the array
  ...
  PMTHV7 is the last element of the array

I know for isntance that MIDAS uses array descriptors (the equivalent of
header keywords) and uses a different convention when storing them to FITS.
I am unsure about IRAF.

I wonder whether some convention could be agreed for the representation of
array quantities within FITS headers.

-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |
Bitnet:   EXOSAT at IMISIAM                   |
-----------------------------------------------------------------------------