Subject: single dish data and FITS format

From: (Bob Hanisch) <[email-masked]>
Date: Thu, 5 Oct 89 16:50:51 EST

Some Comments About Single-Dish FITS Format
                        and Related FITS Issues

                          Bob Hanisch, ST ScI
                              October 1989

  In preparation for the meeting in Green Bank later this month, I have
written down some ideas concerning how to handle the special case of single
dish radio astronomy data within the FITS format.  This is being distributed
in advance of the meeting so that attendees can think about the issues prior
to arrival.

  Harvey Liszt distributed an e-mail message to all of us which contained a
careful analysis of the header structures proposed by Stobie and Morgan. 
Unfortunately I think this analysis does not really address the general
problem, i.e., how to manage the keyword space unique to a certain class of
instruments, detectors, or a particular project.  From the FITS point of view
this is the critical issue, and most of the items discussed in the memo are the
details that need to be worked out once the more general problem has been
addressed. 

  What Stobie and Morgan described in their original memo is a simple
hierarchical keyword space.  The presence of the hierarchy is denoted by the
use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL,
NRAO-12M, etc.) which further defines the scope of the keyword definitions.
The AIPS system has already implemented a hierarchical scheme of sorts, in
that AIPS uses the FITS COMMENT and HISTORY records for storing information
unique to AIPS.  Non-AIPS FITS readers have no obligation to parse the
contents of these cards beyond simply recording them with the other header
information, or even ignoring them if they wish.  The point here is to have a
keyword space which can be hidden from FITS reading programs that do not
understand or have no interest in some specialized set of keywords.  The
advantages of such a scheme include the fact that we need not have any
community-wide consensus on the definitions of ALL FITS keywords.  Rather,
subgroups such as the single-dish radio astronomy community can agree upon
their own conventions and be largely independent of other subdisciplines. 

  If we agree that having some sort of hierarchical keyword naming scheme is
worthwhile (and I think that this is NOT a foregone conclusion -- more comments
on this later), the first thing to do is determine the best syntax for its
implementation.  I see several difficulties with the proposal of Stobie and
Margon (and implicitly with the AIPS technique of hiding similar information
behind COMMENT and HISTORY records):

o  The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH)
and a secondary domain (GENERAL, NRAO-12M, etc.).  This seems unnecessarily
restrictive, although it is clearly simple to generalize to several more levels.

o  The keyword hierarchy takes up a lot of space on the header line, leaving
little space for values or comments.  This situation is exacerbated if deeper
hierarchies are to be permitted.

o  FITS readers require special code to parse the header and retrieve the
data values.

o  Everytime a new domain is introduced, FITS readers everywhere will have to
be updated to recognize (and most often ignore) the new domain name.

  Some time ago I proposed implementing hierarchical keywords by REQUIRING that
each hierarchical declaration actually begin with the keyword HIERARCH (this was
during a FITS planning meeting held in Charlottesville in Jan. 1988, as I 
recall).  The reasoning was that this is such an ugly keyword that no one else
in their right mind would use it, thus it would serve as a flag to indicate
that a special parser would have to go into action in order to retrieve the
actual keyword name and value buried further to the right on the card.  In
retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for
the last one; by reserving a special keyword to introduce a hierarchical
keyword domain, the problem of the arbitrary introduction of new domains is
circumvented. However, the other problems, especially the lack of space on the
header card, are not resolved. 

  An alternative scheme has come to mind (based on an idea proposed by Doug
Tody at the same meeting) which looks much more like simple nested do-loops in
Fortran, or structure definition statements in any modern computer language.
It would be based on begin/end statements for various keyword domains, and
could be nested arbitrarily deep, accommodating things like
SINGLDSH/GENERAL/NRAO/12METER/... or whatever.  I propose to use keywords of
the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy;
I don't really care what the particular names are so long as we can agree on
something unique and representative).  For example, the syntax might be
something like this (paying little attention to the actual column spacing):

---

From: [email-masked] (Don Wells)
Date: Wed, 11 Oct 89 17:38:05 EDT

From [email-masked] Fri Oct  6 11:24:52 1989
	Subject: my memo seems to have been chopped off for some people

Don,
  My memo on hierarchical keywords seems to have been truncated for some
recipients.  Bill Cotton reported to me that his copy had been chopped, and
I just checked mine and it also is chopped, at the same place.  Could I ask
you to check on the mail exploder and see if it might be the culprit?  Here's
the text.  Please send it out to the exploder, if you would.  Have to head off
to the airport in the next few minutes.  Thanks, Bob.

[see technical explanation at end -- DWells]

                          Bob Hanisch, ST ScI
                              October 1989

	.
	.  (standard FITS header)
	.
	.
KHBEGIN = 'SINGLDSH'		/ begin a keyword hierarchy called SINGLDSH
STATION = 'NRAO-12M'		/ one or more keywords unique to this
UNIQKW2 = 1.2345678		/   hierarachy
	.
	.
KHBEGIN = 'GENERAL'		/ second level of hierarchy
UNIQKW3 = 9.8765432		/ one or more keywords unique to this
	.				/   second level of the hierarchy
	.
	.	
	.  (other KHBEGIN keywords define additional levels of the hierarchy,
	.   as needed)
KHEND   = 'GENERAL'		/ close out the GENERAL keyword list
KHEND   = 'SINGLDSH'		/ close out the SINGLDSH keyword list
	.
	.

Now, there are some things that I haven't thought through that carefully yet,
like whether or not the KHBEGIN and KHEND keywords should have a numeral
attached (minimizing the number of keywords with the same name and perhaps
making the parsing of the hierarchy a bit more fool-proof).  Also, there's the
question of whether you would want to allow the full flexibility of going in
and out of a particular domain.  That is, should this sort of thing be
permitted:?

KHBEGIN = 'SINGLDSH'		/ begin a keyword hierarchy called SINGLDSH
	.
	.
KHBEGIN = 'GENERAL'		/ second level of hierarchy
	.
	.
KHEND   = 'GENERAL'		/ close out the GENERAL keyword list
	.
	.
KHBEGIN = 'NRAO-12M'		/ another second level hierarchy
	.
	.
KHEND   = 'NRAO-12M'
	.
	.
KHBEGIN = 'GENERAL'		/ reopen the GENERAL list
	.
	.
KHEND   = 'GENERAL'
KHEND   = 'SINGLDSH'		/ close out the SINGLDSH keyword list

This is not a big deal, and maybe it would simply be a matter of style whether
or not this would be allowed or just discouraged.  However, there are a couple
of major advantages of this approach to managing the keyword space:

o  The keywords within the hierarchy can all be parsed with standard FITS
readers.

o  There is no wasted space on the individual keyword cards, allowing more
complete use of the comment field.  If there are enough `private' keywords to
warrant setting up a hierarchy, the overhead of the BEGIN and END cards is
small.

o  The logic for skipping over the `private' keyword definitions is
straightforward:  upon encountering a KHBEGIN card, the reader would be free
to skip parsing other cards until the corresponding KHEND card was found.
In order to simplify this somewhat, we may wish to limit the depth of the
hierarchy, say to 9 levels (that should be enough for just about everyone,
and would allow the formation of numbered hierarchies, KHBEGIN1, KHBEGIN2,
etc., as described above).  There is clearly redundant information in using
both the numbering scheme AND the value field of the BEGIN/END cards.

o  The definition of new keyword hierarchies (SINGLDSH, INTERF, AIPS, CCD,
HST, etc.) does not require FITS readers to be updated.  A one-time update
is all that is necessary, and even this is not essential for existing FITS
readers to work since they could simply ignore all of this stuff, with no
harm done.

  Thus, if the need for private keyword definition spaces is genuine, a 
scheme such as that proposed here might be a more viable one for implementation
than that previously proposed.  However, the question remains as to whether
such private keyword domains are even required.  Just to review this situation
a bit, the prime motivation for having a hierarchical keyword space of some
sort is so that various projects, disciplines, etc., can define any set of new
keywords without having to worry about name collisions with other such groups. 
But is this really a concern?  Does it really matter if a CCD data set and a
single-dish radio data set have inconsistent definitions of BIAS?  or REFPT? 
or DELTAX?  I guess it depends on what level of interoperability we expect to
achieve with FITS.  In a practical sense I think we all recognize that the idea
of establishing some sort of global dictionary for FITS keywords is impossible.
The hierarchical scheme says, ok, we can't do it so let's hide all of our
private definitions.  The alternative says, ok, we can't do it but it doesn't
really matter because the types of data we're transferring are so heterogeneous.

  (By the way, I wish to point out something about the practice of defining
new FITS keywords.  There seems to have arisen some idea that it is forbidden
to define keywords other than those mentioned in the series of FITS standards
papers.  Nonsense.  Reread Paper I, p. 366, middle of the 2nd column: 

    "Note that the two long keywords, INSTRUMEN and TELESCOP are names which 
    have been truncated after 8 characters.  We suggest this as a good 
    convention for the creation of new keywords."

Whether they intended it or not, Don, Eric, and Ron left the door wide open for
the creation of new keywords and even recommended how it should be done.) 

  What does matter, I think, is to have a small set of reserved keywords that
all FITS readers might reasonably be expected to understand, or at least not
interfere with.  FITS Paper I defines the fundamental set (SIMPLE, BITPIX,
NAXIS, NAXISi (i=1,NAXIS), and END) and an optional set (BSCALE, BZERO,
BUNIT, CRVALi, etc.).  One solution to this problem would be for the various
FITS committees to review Paper I and agree upon such a set of reserved
keywords, and then let various projects use the rest of the keyword space to
their hearts' desires.  This is, after all, the situation we have now and
maybe it's not so bad.  Anyway, wait until you all see HST data!  In any case,
I think that a clarification and standardization of some small set of keywords
is desirable regardless of whether we adopt a hierarchical naming convention
project or discipline dependent keywords.

  Well, those are my thoughts at the moment.  I've gone on a bit longer than
I thought, but I hope that those of you who've stuck with me to this point
have got some food for thought and that we can have a lively discussion about
this in Green Bank.  See you all there.

======================= non-FITS technical discussion below: =============

Why did Bob's message on 05 October get truncated?

The SMTP (Simple Mail Transfer Protocol) of the Internet terminates a
message by a line that has *only* one period (.) on it. This rule allows
the mail protocol to work the same way on top of a variety of lower
protocols, character sets and OSes.  So, what does an SMTP mailer do if a
message contains a text line with only a period? Why, double it, of course!
And, of course, a mail daemon which receives such a message will change the
double period back to a single period before delivery to the customer
mailbox.

Messages containing single-period lines are rare. Bob's message at 1755EDT
last Thursday 05 October contains the first such lines I have seen in
several years. His message was truncated *exactly* at the first occurrence
of such a line, and so we know that the problem may be that some mail daemon
somewhere failed to double the period on transmission.

At present I am unable to say with certainty which of three different
computers, one at STScI and two at NRAO, was the guilty party. 

I have added tabs before the periods in this copy to defeat the RFC822
rule.

-Don