Subject: OGIP Long String Keywords

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 1 Aug 1994 15:26:20 -0400

The OGIP Long String Keyword Convention

The OGIP FITS Working Group met recently to decide how to handle long
string keyword values in FITS files.  We urgently need such a
convention for our soon to be launched mission, so given that there is
currently no consensus within the FITS community on a standard way to
deal with this problem, we have adopted a local FITS convention for use
within our own processing environment.  Since this is not yet an agreed
upon standard, we will not use this convention for any keywords that we
would expect any external software systems would need to read or
interpret.

The following example illustrates the convention that we have adopted:

BIGSTRNG= 'This is a long string value that is continued &' / Any comments
CONTINUE  'over three keywords in the&   '                 / may be appended
CONTINUE  ' FITS header.'                           / after the quoted value.

The '&' character is used as the last non-blank character of the string
to indicate that the string is (probably) continued on the following
keyword (if the next keyword name = 'CONTINUE' as described below).  We
adopted the '&' character instead of the '\' that had been previously
proposed because of Barry Schlesinger's observation that the '\' is one
of the 10 'National Use' positions which may not be interpreted as the
same symbol by all nationalities.  Also, some Fortran compilers treat
the '\' character in a non-standard way making it difficult to write
machine-independent code that uses this character.  The '&'  seems to
be a reasonable alternative, especially since it is used as a
continuation character in Fortran 90.

Each continuation line has the keyword name = 'CONTINUE'.  We adopted
this keyword name, rather than the previous suggestion of an all blank
keyword name, in order to provide an unambiguous way of identifying any
continuation keywords even if they get separated from their parent
keyword, and to try to avoid conflicts with other FITS software that
may treat keywords with an all-blank name in a unique or special way.
Since there is no equal sign in column 9, this keyword belongs to the
same class of keywords as the COMMENT and HISTORY keywords that do not
have a defined value.   Under our convention the continuation of the
character string value is enclosed in quotes starting in column 11 (or
higher) of the 80-character record.  Any other characters (e.g. a
comment field) may optionally follow the closing quote character on any
or all of the keywords.

We also considered, but decided against, implementing a convention that
would explicitly link together all the keywords so that the correct
continuation sequence could be reconstructed even if the order of the
keywords in the header were rearranged. (Several such schemes were
suggested here on FITSBITS in the past month).  Since there is even
less consensus on the best way to implement such a scheme, or if it is
even needed, we decided to simply rely on the physical order of the
keywords in the header, which meets our own needs.  (We did confirm
with Barry Schlesinger that this reliance on the order of the keywords
in the header does conform to the FITS Standard and hence is legal to
use in a local FITS convention).  If the FITS community later decides
that a full forward and backward keyword linking capability is required
to make this convention acceptable as a Standard FITS convention, then
this can in principle be added with only minor modifications to our
convention.

Finally, we plan to add the following keywords in the header of any
FITS extension  that uses this long string convention (the exact
wording may change slightly):

LONGSTRN= 'OGIP 1.0'           / The OGIP long string convention may be used.
COMMENT   This FITS file may contain long string keyword values that are
COMMENT   continued over multiple keywords.  This convention uses the  '&'
COMMENT   character at the end of a string which is then continued
COMMENT   on subsequent keywords whose name = 'CONTINUE'.

The presence of the LONGSTRN keyword serves to indicate that long
string keywords may be present in the FITS file.  The value of this
keyword gives the name and version number of the specific convention
that is used, which in our case is the OGIP long string convention,
version 1.0.  Note that the value of this keyword is a string so that
it may be used to give the name of any other convention that the FITS
community might eventually adopt.

Support for this long string convention will be built into the next
release of the FITSIO library for anyone else who may wish to
experiment with using it.  The FITSIO routines that read or delete
string keywords will transparently support this convention, but the
string keyword writing subroutines will not, by default, write strings
longer than will fit on a single keyword.  FITSIO programmers will have
to invoke a special FITSIO subroutine to write these long strings, to
prevent unsuspecting users or programmers from creating these long
string keywords unintentionally.  Details on this will be provided in
the next version of the FITSIO library  which is due to be released in
the next month.

                             - The OGIP FITS Working Group

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 4 Aug 1994 11:52 EDT

In article <199408011926.PAA20800 at tetra.gsfc.nasa.gov>, William Pence <pence at tetra.gsfc.nasa.gov> writes...
>The OGIP FITS Working Group ....
>We urgently need such a convention ... given that there is
>currently no consensus within the FITS community on ... 
>this problem, we have adopted a local FITS convention .... 
>Since this is not yet an agreed
>keyword (if the next keyword name = 'CONTINUE' as described below)....  
> 
>this keyword name... 
>in order to provide an unambiguous way of identifying any
>have a defined value.... 
>keywords in the header were rearranged....  Since there is even
>use in a local FITS convention).  

I want to be sure that what I said is interpreted with the proper
connotation.  What OGIP plans is not a violation of the rules of FITS.
I would say that it is analogous to the use of a local extension, one
that conforms to the rules of the Generalized Extensions paper and
whose name is properly registered with the IAU Working Group.  The
presence of such a FITS file will not prevent an properly written FITS
reader from reading the primary HDU and standard extensions on such a
file. It will be able to skip the unknown extension using the size 
information in the header.  But if the unknown extension contains 
information important to the scientific interpretation of the 
remainder of the file, then the readers will have difficulty in using 
the data they can read.  So the file will have failed in that sense in 
transporting information, the primary purpose of FITS.  For that 
reason, it is not wise to use local extensions, and by conventions, in 
data sets designed primarily for transport, and, by extensions, 
archival.

>version 1.0.  

In this context, we should recall some comments from the Generalized 
Extensions paper (Grosbol et al. 1988, Astron. Ap. Suppl., 73,359.

"Implementors of FITS writing programs should alsways be aware of the  
 limitations of recipients.  The primary objective of the FITS standard  
 remains the reliable, unambiguous transmission of data to recipients. 
 Esoteric, complex data structures should be avoided as much as  
 possible.  The watchword of the implementor should be: keep it 
 simple."

				Barry Schlesinger
				NOST FITS Support Office