Subject: Continuation Keywords

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Fri, 14 May 93 17:14:59 EDT

A Proposal for Continuation Keywords in FITS Headers

                   William Pence and Arnold Rots
                            NASA/GSFC
                           14 May 1993

The maximum allowed length of a single character string keyword value
in a FITS header is 68 characters which is too short for some
applications.  In order to overcome this length restriction, we propose
here several possible conventions for allowing a character string value
to be specified over multiple keywords in the FITS header.   We
prefer the first option, but would welcome comments on any of
these, or on any other possible option.

Option 1.   Blank Continuation Keyword Convention

A continued string value is recognized by a backslash (\) 
as the last character in the first keyword string, followed
by a keyword with a blank name followed by the continuation
of the keyword value enclosed in quotes.  Example:

KEYNAME = 'This is a very long keyword string value which \'
        'continues over several lines\'
        ' of the FITS header.'

Option 2.   'CONTINUE' Keyword Convention

A continued string value is recognized by a backslash (\) as the last
character in the first keyword string, followed by a keyword with the
name 'CONTINUE' containing the continuation of the keyword string value.
Example:

KEYNAME = 'This is a very long keyword string value which \'
CONTINUE= 'continues over several lines\'
CONTINUE= ' of the FITS header.'

Option 3.    A Completely General Continuation Keyword Convention

A continued string value is recognized if the name of the continuation
keyword is present at the end of the string enclosed in backslashes.
Example:
 
KEYNAMEA= 'This is a very long keyword string value which \KEYWORDB\'
KEYWORDB= 'continues over several lines\KEYWORDC\'
KEYWORDC= ' of the FITS header.'

This will only be interpreted as a continuation if the referenced
keyword exists and if the backslash is the last character of the string.

Discussion:

We prefer the first option because it should cause minimal confusion
for basic FITS readers (they could simply treat the continuation lines
as comments), and it is more compact and more efficient to parse than
Option 3.   Some FITS readers may complain or lose information with
Option 2 if there are multiple CONTINUE keywords in a FITS header.  The
FITS standard does not prohibit having multiple keywords with the same
name, but this is generally discouraged to avoid confusion.   Option 3
is the most general, however, it is wasteful of header space (since it
requires 10 characters for the continuation keyword name and
backslashes) and it would be the least efficient to implement since
there is no requirement that the continuation keywords are sequentially
ordered in the FITS header.

It should be pointed out that Option 3, although it might seem the
most robust since it does not repeat keywords and is resistent to
FITS readers/writers that throw away blank keyword lines, has the
disadvantage that it allows recursion.

As shown in the examples, this convention could be repeated on 
multiple lines to express arbitrarily long strings.  It would 
probably be prudent, however, to place an upper limit on the
total length of the concatenated string in order to simplify the
task of parsing these strings, especially on systems with
limited resources.  We suggest that an upper limit somewhere in the
range of 256 to 1024 characters would be appropriate.  Since one
should never skimp too much on these things, 1024 is probably the
more prudent value.

---

From: sla at umbra.UCSC.EDU (Steve Allen)
Date: 14 May 1993 22:28:21 GMT

In article <9305142114.AA06620 at tetra.gsfc.nasa.gov>
>        'continues over several lines\'

Is there any restriction that forces a FITS reader not to
reorder the FITS cards?  I believe I've met at least one
data reduction package which did not guarantee that output sets of
FITS cards would happen in the same sequence as input sets of FITS cards.

Nonetheless, I prefer option 1 as a strategy for the future.
_______________________________________________________________________________
Steve Allen          | That was the equation!         |   sla at lick.ucsc.edu
UCO/Lick Observatory | Existence!...Survival must     | If the UC were opining,
Santa Cruz, CA 95064 | cancel out programming! -- Ruk | it wouldn't use me.

---

From: bhill at stars.gsfc.nasa.gov (Robert S. Hill)
Date: 17 May 1993 12:42 EST

In article <9305142114.AA06620 at tetra.gsfc.nasa.gov>, 
pence at tetra.gsfc.nasa.gov (William Pence) writes...

>applications.  In order to overcome this length restriction, we propose

Could you give us an example of a real application?  I, for one,
am very leery of adding new syntax rules to FITS, particularly ones
that depend on the ordering of header records, or which encourage
(as opposed to merely tolerating) repetition of keywords.

          Robert S. Hill
----------------------------------------------------------------
bhill at stars.gsfc.nasa.gov
Hughes STX Corp.
Code 681, NASA Goddard Space Flight Center, Greenbelt, MD 20771

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Mon, 17 May 93 14:50:45 EDT

Robert Hill wrote:

>Could you give us an example of a real application? 

68 characters is really quite a small limit, so there are 
many examples that could be cited.  One example would be
to specify the full path and filename for a calibration file.
Our file names are about 30 characters long;  after you
add the full subdirectory path, it is not hard to exceed 68 characters.

Another example is the column descriptor notation that we
will probably be using for the XTE X-ray satelite data.
The organization of this data is very complex, thus we need
a rather complex string to describe the contents of each
column of a binary table, which represents a multidimensional array of 
data.  An example is:

TDDES2  = 'E(X1L|X1R|X2L|X2R|X3L|X3R) & C("TDDMAC1(0:14)") &
           T([" at TIME",1024,0.00390625])'

In principle this string could be hundreds of characters long. 

-Bill Pence

---

From: tody at noao.edu (Doug Tody)
Date: Mon, 17 May 1993 19:39:05 GMT

There are many cases in IRAF image headers where a long string is stored in
multiple FITS keywords.  Here is an example:

  WAT2_001= 'wtype=multispec spec1 = "1 127 0 4953.94775390625 0.0714096948...
  WAT2_002= '9 256 0. 23.28 31.32" spec2 = "2 126 0 4998.365722656251 0.071...
  WAT2_003= '15742111 256 0. 46.07 58.42" spec3 = "3 125 0 5043.49462890624...

Each card contains a 68 character string, and the software will simply
concatenate these strings to generate the original long string.

The advantage of this approach is that it uses only standard FITS, and does
not assume that the cards occur in the header in any particular order.  Any
FITS reader can access the individual cards.

	- Doug

---

From: thompson at serts.gsfc.nasa.gov (William Thompson)
Date: 17 May 93 21:17:50 GMT

>                   William Pence and Arnold Rots

	(text deleted)

Of three options shown, I also prefer the first.  I don't see a whole lot of
practical difference between Option 1, i.e.

and Option 2, i.e.

Option 3, i.e.

just seems way too complex to me.

Another real world application: for the SOHO project we'll be using a
combination of FITS and CDF files, and will be converting some files back and
forth between the two formats.  The CDF files are much less restricted than
FITS files in how long character strings can be (among other things).  It would
be nice to be able to convert to and from FITS without loss of information.

Bill Thompson

---

From: bhill at stars.gsfc.nasa.gov (Robert S. Hill)
Date: 17 May 1993 17:53 EST

In article <thompson.737673470 at serts.gsfc.nasa.gov>, 
thompson at serts.gsfc.nasa.gov (William Thompson) writes...

Thanks to Pence and to Thompson for the application examples.

Of the three alternatives proposed, I also prefer the first, simplest one.
However, the FITS User's Guide should then warn against having FITS
processors re-order records.

I imagine that most don't, anyhow.

          Robert S. Hill

---

From: tody at noao.edu (Doug Tody)
Date: Mon, 17 May 1993 23:45:32 GMT

>From an abstract point of view I agree with the rest of you - the nicest
solution to this problem is to have something like backslash continuation to
allow arbitrarily long header lines.  However, I think this is a more
serious change to FITS than has been admitted here.  Whether you admit it or
not, the above example is a single keyword=value statement which is spread
over three lines of the header.  In affect you are changing the lexical form
of the header.  For example, to delete this keyword, or copy it to another
header, one would have to treat the three lines as a unit or the result
would be a munged FITS header (compare this to KEYW_nnn, where the pattern
KEYW_* input to a generic FITS program will automatically deal with the
logical set of related keywords as a group).  As has already been pointed
out, there is no requirement that the ordering of FITS header lines be
preserved.  There are existing examples of FITS programs which change the
order of header lines, e.g. to separate classes of keywords or resolve cases
of redefined keywords.

Hence, general FITS programs would have to recognize the proposed convention
to be able to deal properly with headers containing this construct.  Basic
FITS is keyword=value in 80 chars.  It is best to stick with something which
is consistent with this, and perform logical grouping of related keywords at
a higher level, outside of the basic header mechanism.

The existing FITS header is pretty limited - someday it would be nice to
have a comprehensive redesign which provides things such as longer keywords,
variable length lines, backslash continuation, elimination of the
inefficient blank fill, and hierarchical keywords (a way of structuring
large headers into distinct name spaces).  On the other hand we have a
requirement to remain backwards compatible so that existing FITS archives
can still be read.  Maybe we will eventually find a way to add some of
these desirable features while still remaining backwards compatible.

Oh well.  So long as what one does is compatible with the basic standard,
one is free to solve problems such as this any way one wants.

	- Doug

---

From: bhill at stars.gsfc.nasa.gov (Robert S. Hill)
Date: 17 May 1993 20:42 EST

In article <1993May17.234532.18859 at noao.edu>, tody at noao.edu writes...
>of the header.  For example, to delete this keyword, or copy it to another

This is what bothered me as well, and is the reason I wanted to see
examples.  I think Doug has a real point here.  Of course, any of the
three proposals can be done, strictly speaking, without altering the
standard; the problem is that none of them is very robust with respect 
to perfectly legal manipulations.  

Doug's practice of using numbered keywords is more robust and accords
with existing FITS style.  The disadvantage is that each instance
requires special processing and a modest amount of additional
documentation.  

>The existing FITS header is pretty limited - someday it would be nice to

Yup.  The usual device for overcoming the limitations of FITS is to
invent a new extension.  Maybe someone should propose an 
`extended-header extension':

          XTENSION= 'XTNDHEDR'
          BITPIX  =                    8
          NAXIS   =                    n
          END
          .... ASCII text in following form:

          ARBITRARILY-LONG-KEYWORD='ANYTHING AT ALL, DELIMITED BY
          BACKSLASH'\ANOTHER-ARBITRARILY-LONG-KEYWORD='MORE AND M
          ORE AND MORE AND MORE STUFF, EXCEEDING THE 68-CHARACTER
          LIMIT WITH GREAT GLEE AND GOOD FEELING'\

Actually, I'm kidding.  But you could do it, right? 

          Robert S. Hill

---

From: GROTH at pupgg.princeton.edu (Edward J. Groth 609-258-4361)
Date: Tue, 18 May 1993 14:25:53 GMT

>one is free to solve problems such as this any way one wants.

Well - here's my 2 cents: 

Don't mess with the basic keyword=value in 80 columns!!!!!!!!!!!!!!!!!!!

Aside from the basic keywords there is no requirement on ordering
- reordering to optimize particular applications is perfectly
reasonable.  

What's wrong with 
KEY_1   = '  '
KEY_2   = '  '
....
KEY_N   = '  '

?

The only problem I see with this is you don't necessarily know
when the string ends, so precede the whole mess with

KEY_0   = Integer giving length of string or number of KEY_n
lines to follow.

					- Ed

/----------------------------------------------------------------------\
| Edward J. Groth            | Phone: 609-258-4361                     |
| Physics Dept., Jadwin Hall | Fax:   609-258-6853 <- Changed 1-Feb-93 |
| Princeton University       | SPAN/HEPNET:  PUPGG::GROTH=44117::GROTH |
| Princeton, NJ 08544        | Internet:     groth at pupgg.princeton.edu |
\----------------------------------------------------------------------/

---

From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Date: 19 May 93 16:11:28 GMT

I don't have much to say about the proposed methods for providing continued
keyword values over multiple card images, except that I don't like it much.  I
prefer a method that has multiple ordered keywords (i.e. KEY1, KEY2, ... KEYn)
as opposed to a new general rule for parsing *all* character-valued keywords.

Why should I blow CPU cycles checking the last character of each and every
string for a backslash on the off-chance one or two may be continued?

With ordered keywords, I install a callback function for that specific keyword
that knows how to deal with continuations (if any) and leave it at that -- or
simply ignore the keyword, which I am free to do under the standard.  If the
keyword is (possibly) continued, I am not free to ignore it, but must parse
forward in the input stream to skip over the continuations.

I have to admit, though, that I do enjoy seeing people jump through hoops to
try to get from FITS some of the descriptive power other disciplines have had
for years.  Rather than shoe-horning little conveniences into a wholly outdated
exchange format, I'd much prefer to see a 'constitutional convention' called for
a total redesign to make FITS actually fit in to modern networked data 
processing environments.

But that's just me.

--
-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 19 May 1993 13:38 EDT

In article <17MAY199320421274 at stars.gsfc.nasa.gov>, bhill at stars.gsfc.nasa.gov (Robert S. Hill) writes...
> ...  The usual device for overcoming the limitations of FITS is to

No, because the rules for keywords apply to all keywords, even the 
user-supplied keywords, and even those in user-defined extensions. 
This rule is one of those that cannot be violated in user designs

"The keyword is an 8-character ASCII string left justified in columns 
1-8..."
				-- FITS Paper I

5.1.2.1 Keyword (bytes 1-8)
"The keyword shall be a left justified, 8-character, blank filled 
ASCII string with no embedded blanks.  All digits (hexadecimal 30 to 
39,``0123456789'') and upper case alphabetic characters (hexadecimal 
41 to 5A, ``ABCDEFG HIJKLMN OPQRST UVWXYZ'' are permitted; no lower 
case characters shall be used.  The underscore (hexadecimal 5F, ``_'') 
and hyphen (hexadecimal 2D, ``-'' are also permitted. No other 
characters are permitted."

Actually, for long strings of text, probably the way to go is with the 
commentary keywords: COMMENT, HISTORY, and keyword field blank.  No 
special continuation is needed.

				Barry Schlesinger
				NSSDC/NOST FITS Support Office

---

From: jwp at sal.wisc.edu (Jeffrey W Percival)
Date: Wed, 19 May 1993 21:19:52 GMT

In article <leb.737827888 at Hypatia> leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes:
>I have to admit, though, that I do enjoy seeing people jump through
>hoops to try to get from FITS some of the descriptive power other
>disciplines have had for years.  Rather than shoe-horning little
>conveniences into a wholly outdated exchange format, I'd much prefer to
>see a 'constitutional convention' called for a total redesign to make
>FITS actually fit in to modern networked data processing environments.
>But that's just me.

I was wondering when someone would say this.

If we used a BNF-style specification for a header grammar, with
a Lex-style tokenizer, all these issues of parsing and lookahead,
not to mention the subtle "interpretation" conflicts that arise,
would simply vanish.

A lawyer-like combing of a possibly self-inconsistent plain-language
specification, a-la the tax code, is hardly a substitute for a yes/no
answer from a grammar-driven state machine!

And if you change the grammar, you get your code remade for free.

To paraphrase a saying around here (and not to derogate the seminal
contibution of its creators), it would be nice to drag FITS kicking and
screaming into the Eighties!
-- 
Jeffrey W Percival (jwp at larry.sal.wisc.edu) (608)262-8686

---

From: bhill at stars.gsfc.nasa.gov (Robert S. Hill)
Date: 19 May 1993 15:39 EST

In article <19MAY199313384908 at nssdca.gsfc.nasa.gov>, 
bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER) writes...
>In article <17MAY199320421274 at stars.gsfc.nasa.gov>, 
bhill at stars.gsfc.nasa.gov (Robert S. Hill) [that's me] writes...

>>`extended-header extension':

[... see below, maybe I wasn't very clear...]

>"The keyword shall be 

[...yada yada yada...]

Barry, the idea of my thought experiment is that you can put any
information you want into the *data* portion of an HDU.

The point I was making is that the only way to get really broad new
powers out of FITS is to find a way of encoding the desired information
into an extension.  But once you decide to do so, almost any result
can be forced.  I mean, the following is perfectly legal, as
far as I can tell:

Main header:
          SIMPLE  =                    T
          BITPIX  =                    8
          NAXIS   =                    0
          EXTEND  =                    T
          END
Extension header:
          XTENSION= 'IMAGE   '
          BITPIX  =                    8
          NAXIS   =                    1
          NAXIS1  =             10000000
          END
Extension data:
          [...insert complete works of Shakespeare in TeX format
          here...]

Okay, so why not the following?

Main header:
          SIMPLE  =                    T
          BITPIX  =                    8
          NAXIS   =                    0
          EXTEND  =                    T
          END
Header of first extension:
          XTENSION= 'XTNDHEDR'
          BITPIX  =                    8
          NAXIS   =                    1
          NAXIS1  =                15000 
          END
Data of first extension:
          [...insert 15000 bytes of description, structured in some
          definite way, with *its own agreed syntax* distinct from
          FITS header syntax ...]
Header of second extension:
          XTENSION= 'IMAGE   '
          BITPIX  =                  -32
          NAXIS   =                    3
          NAXIS1  =                 2048
          NAXIS2  =                  256
          NAXIS3  =                   32
          END
Data of second extension:
          [... the actual binary data ...]

I hate to take so much space with this, but I was determined to make
the faux pas of explaining my joke.

Lee Brotzman's remark about a constitutional convention is well taken,
because the extension mechanism is so clunky.  (One would rather have
had a richer format to start with.)  But perhaps we are using extensions
too conservatively.  *Would* it in fact be possible to generate some
sort of purely documentation-oriented extension that would meet Lee's 
criteria for modernity?

          Bob Hill

---

From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock)
Date: 20 May 93 15:04:47 GMT

>But that's just me.

No, it isn't just you. <g>

And it doesn't take much of a visionary to imagine "Grep-able FITS".
WHAT?  You want _DELIMITERS_???  In FITS???

But, in the meantime, we got what we got, and it works where it works
(and I don't think Casey Stengel ever said that).

--
_______________________________________________________________________
-- Archie Warnock              Internet:  warnock at hypatia.gsfc.nasa.gov
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC                   Project STELAR: WAIS to do science