Subject: Summary: Storing long keywords in FITS files

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Tue, 14 Jul 1992 15:39:00 GMT

A while back I had asked the following question

	Is there any kind of defacto way to store keywords longer than eight
	characters in a FITS header?  Someone suggested using the COMMENT
	keyword followed by the non-standard keyword, although another
	possibility is using the null keyword (eight blanks).  Is this what
	other people do?

	I'm not talking about keywords essential to reading and scientifically
	processing the data.  I'm more interested in storing engineering
	information about the state of the instrument.  It would be desirable
	to retain the original mnemonic names used by the engineers.  Important
	information could also be replicated using more standard FITS keywords.

	Although it's obviously legal to put anything in a COMMENT field, I'm
	interested in knowing what other people do.

I think that the problem of storing and manipulating keywords of greater than
eight characters in length is important for the future growth of FITS,
particularly if one wants to be able to move data back and forth between FITS
and other scientific file formats such as SFDU/PVL.

The functionality I was looking for was not simply to store the information,
but to be able to retrieve and manipulate that information as well.

Of the responses I got back, I was attracted to one in particular.  First,
because it is already in use at at least one major site, and second because it
seems to offer the kind of functionality and expandability I need.  That
response, from Eric Greisen at NRAO, is as follows:

	The solution we have used at NRAO (at it is even quietly present
	in the original FITS paper!) is to have HISTORY cards.  As the
	first word in the HISTORY we put an identifier that says that
	we have written the card, e.g. NRAO or AIPS.  Then we put one
	or more keyword=value pairs, with any old arbitrary choice of
	keywords.  Our FITS readers check each HISTORY card's next word
	and if it is one of a few magic ones they parse the rest of the
	card.  Otherwise, they ignore the card as non-NRAO readers are
	expected to do with all our cards as well.

I like this idea.  However, I think I would make the magic work something
unusual so that it wouldn't be confused with some ordinary text message.  For
instance, I think I would preface it with "K_" to represent special keyword.
Thus, I could have something like

	HISTORY K_SOHO MYSTRINGKEY = "This is a string" /Comment
	HISTORY K_SOHO MYINTEGERKEY = 3			/Another comment

I don't know if HAO does it or not, but I would even allow for FITS-like
comments, delimited by slashes, as shown above.  I wouldn't be picky about the
number of spaces between the various elements, or exactly which columns the
values occur in.  The only trouble I would anticipate would be with complex
values, because the FITS standard has the real and imaginary parts occuring in
specific columnar regions, which wouldn't be possible here.

A somewhat similar solution, from David Robinson, is shown below.  However, I'm
less attracted to it because it doesn't seem to be oriented toward recovering
and manipulating the data electronically.

	It seems clear to me that the keywords should be reserved for storing
	information that is essential for the processing of the data.
	Information that records how the data was generated (such as instrument
	settings, options given to the processing software etc.) should
	properly be stored in HISTORY records.

	For example, here is the FITS header for an output file from an image
	processing program that I have written:

	SIMPLE  =                    T
	BITPIX  =                  -32
	NAXIS   =                    2
	NAXIS1  =                  512
	NAXIS2  =                    1
	DATE    = '29/05/92'
	HISTORY Written by TOY3
	HISTORY set nrand 1
	HISTORY set tol 0.1
	HISTORY set background 5
	HISTORY set entropy standard
	HISTORY set image "line.dat"
	HISTORY set psf "psf1.dat"
	HISTORY var chisq 196.25
	HISTORY var iter 6
	HISTORY var sigma 2.78645
	HISTORY command write image "out.dat"
	END

	The HISTORY cards record the program used, the parameters given by the
	user (set..), the variables that the program found, (var...) and the
	command given to write out the file.

Simon Ellingsen writes:

	Instead of putting this type of none standard information in the
	header, why don't you use a binary extension table.  Then you
	effectively store the keywords in the TFORMn value which doesn't have
	the eight character restriction.

I agree, and intend to do just that when appropriate.  However, some data would
apply to the table as a whole, or to a primary FITS HDU.

A couple of people suggested converting the long keywords to an eight-character
equivalent, and then storing the real name in the comment.  Edward J. Groth
>from Princeton writes:

	How about

	KKKKKKKK=value  / Engineering keyword here

And Thierry Forveille of the Observatoire de Grenoble writes:

	Althought I never had to face that specific problem, I would suggest
	you store the information with a standard keyword and append the
	original mnemonic as an end of line comment (after the /). This is
	standard too, and the information will be easier to extract for a
	general FITS handling program if you later want to plot this
	engineering information. Extracting the information from a COMMENT or
	blank keyword would probably need ad hoc (though trivial) modifications
	of the program. Of course the standard keyword need not be particularly
	informative and something line ENGIN001 to ENGIN999 would work.

I thank everybody for their input.

Bill Thompson