Subject: re: TSORTKEY convention

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 5 Dec 1994 17:39:48 -0500

Regarding the TSORTKEY convention posted here last week, several people
questioned the use of the column name rather than the column index to
specify the sort column(s).  Our FITS working group did consider this
issue  but in the end decided to use the column label (as given by the
TTYPEn keyword) for several reasons:

  - It is easier for humans to read and decode (contrary
    to Francois  Ochsenbein's assertion that only software reads FITS headers
    we find that users and programmers also frequently need to read 
    the FITS headers).  

  - It is more in the 'object oriented' spirit to refer to a column by
    its name rather than absolute position within the table.  This
    helps to insulate the creator and user of the TSORTKEY keyword
    from having to know about the internal structure of the table.
    This is similar to relational database systems that generally
    refer to table columns by name rather than position.

  - If the columns in the table are rearranged, or if columns are added
    or deleted in the table, then the TSORTKEY does not have to be
    modified.   

The main concern about using the column name seems to be "what does one
do if the table does not have a TTYPEn keyword for the sort column?".
There is a simple solution: just make up a default name for the column and
add the appropriate TTYPEn keyword along with the TSORTKEY keyword.
For instance if one wants to indicate the table is sorted on the 3rd
column then add the following 2 keywords:

TTYPE3  = 'COL_3'
TSORTKEY= 'COL_3'

The other concern is what to do if the sort column name begins with a
minus sign or the column name is not unique.  The best anwser is
probably that the user should modify the FITS file to give the columns
sensible names.  If this is not possible though, then  one is simply
out of luck and cannot use the TSORTKEY convention, which after all, is
not required.  If someone is going to ignore the NOST recommendations
on the choice of column names (i.e., use only alphanumeric characters
and the underscore) then they should not be surprised to encounter some
difficulties later in using that FITS file in certain software
systems.

Another way to state this is that the TSORTKEY convention is layered on
top of another well established convention on the choice of column
names, and this was spelled out in section 5 of the definition of the
TSORTKEY convention.  As an aside, the OGIP is currently drafting some
specific guidelines for column names in all the FITS tables that we
produce.  This will follow the NOST recommendations plus possibly
include additional restrictions, such as that the names must be unique
within the first 8 characters to be compatible with IRAF.

Finally, Francois Ochsenbein asked:

> 2) NULL values: does the definition state that, if a TNULL value is defined
>    as "-999.99", these values will come AFTER a valid value like "999." ?
 
Yes.  The particular value used to indicate a null pixel is not really
relevant.  All that matters is that these elements are undefined, and
thus will appear at the bottom of the sorted table.  

-The OGIP FITS Working Group

---

From: Preben Grosbol <pgrosbol at eso.org>
Date: Tue, 06 Dec 1994 12:28:36 +0100

I fully agree that the usage of column labels is nice and more
readable. The disturbing point is to have a convention which
may or may not work. Just to add one more marginal issue, FITS
recommend that string values are unique to 8 character (due to
old outdated readers). Now take column labels which only differ
at the 8th character. By adding the minus sign they can not
any longer be distinguished within the 8 first characters.

As the usage of names in the TSORTKEY is better that numbers but
has its problem, one could consider to allow both i.e. demand
that label names used with the TSORTKEY convention do not
start with a '-' or a digit. If in the TSORTKEY field a label
is found starting with either a digit or -digit it is interpreted
as a field number.

Preben Grosbol

---

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Thu, 8 Dec 1994 20:19:32 +0100

>     we find that users and programmers also frequently need to read 
It is (unfortunately) true that humans occasionaly have to look at the
header, but human readable information is best conveyed through
comments (ever tried to make sense of an RA in degrees, or remember
the observing hour from a UT in seconds...).  If human readibility is
the main concern, an easy fix would be to use column numbers and
strongly suggest to add the column name in the comment field.

The points about the 'object oriented spirit' and rearrangements are 
however strong enough alone to shift my vote towards column names.

One point I don't like in the present state of the proposal is its
use of the 'sign' of the column name to convey the ordering. This
strikes me as an adhoc fix to store two informations in the same
field. Of course FITS contains a number of adhoc historical patches
(such as BITPIX=-32 for floats), but why add one when we can avoid
it. Storing the ordering as a separate keyword (say TSORTORDR)
would be cleaner, and would eliminate potential problems with column
names which are unique over 8 chars but not 7).

	Thierry Forveille
	Observatoire de Grenoble
	forveill at gag.observ-gr.fr

---

From: "Jonathan McDowell" <jcm at urania.harvard.edu>
Date: Thu, 08 Dec 94 14:35:30 EST

> names which are unique over 8 chars but not 7).

This works for the single sort key case, but it'll get messy if you
want to use secondary keys with a diferent ordering. I think Bill's
proposal is pretty elegant, and not that difficult to parse. The minus
sign does have a real arithmetical interpretation in this case - we
have reversed the direction of the sort -
unlike the BITPIX example where the minus sign is used to mean
"bad bad not integer nasty horrible real" (at least that's how I read it :-))
  - Jonathan McDowell