Subject: FITS continuation keywords

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Tue, 5 Jul 1994 12:45:51 -0400

Last summer there was considerable debate in this newgroup about a
proposed convention for encoding very long character strings as the
value of FITS keywords.  We will soon need to begin using this
convention (or some variant on it) for the FITS files generated by the
XTE mission.

The main purpose of this posting is to see if there is any concensus
within the FITS communitiy as to which of 2 possible variations on this
convention we should use.  Our orginal (and still preferred) proposal
is to use a backslash character (\) at the end of a string to indicate
that is continued on the next keyword, and to have columns 1=10 of the
following keyword(s) be filled with blanks.  FITS readers that do not
understand this continuation convention would just treat the
continuation lines as comment keywords (since the keyword name is
blank).  For example:

LONGSTR = 'This is a very long string keyword value that \'
          'is continued over three\'
          'keywords in the FITS header.'   

One objection that was raised to this convention was that some FITS
readers apparently treat comment keywords differently from the other
keywords that have a value, and the keywords might get separated, thus
destroying the order of the keywords that is needed to reconstruct the
long string value.  One way to solve this particular problem (if it
really is a problem) is to replace the blank keyword name with a
reserved keyword such as 'CONTINUE' followd by an equal sign, as in:

LONGSTR = 'This is a very long string keyword value that \'
CONTINUE= 'is continued over three\'
CONTINUE= 'keywords in the FITS header.'   

Is there any preference as to which of these 2 conventions we use?
If not we will probably use the original proposal, with blank continuation
keyword names.

Incidentally, to preempt a common suggestion, it is not possible
to use a numbered sequence of keywords for this convention, as in:

LONGS1  = 'This is a very long string keyword value that \'
LONGS2  = 'is continued over three\'
LONGS3  = 'keywords in the FITS header.'   

because we need to use this convention in FITS tables where the keyword
name already has a sequence number attached to it to indicate which
column of the table it refers to.  So the convention that we adopt must
place no restrictions on the name of the long string keyword itself.

Finally, we realize that this is a controversial proposal, and thus will
restrict its use to rather mission-specific keywords that should minimize
the impact on other data systems that don't understand this convention.
In most cases other data systems would not know what to do with these
long string values even if they could read them.

-Bill Pence
OGIP, NASA/GSFC

---

From: Nikolai Piskunov <piskunov at phobos.astro.uwo.ca>
Date: Tue, 5 Jul 1994 14:55:06 -0400 (DST)

On Tue, 5 Jul 1994, William Pence wrote:

> ... 
> value of FITS keywords.
> ...
> For example:
> ... 
> or to replace the blank keyword name with a

I feel that the idea of using \' combination may cause problems for
FITS readers written in C. It may also result in confusion, if the text
of the comment is interpreted after extraction with a shell script.

On the other hand second example in WP mail shows certain redundancy.
So what if we keep the keyword CONTINUE and forget the back slash?
In other words, the example above will look like:

 LONGSTR = 'This is a very long string keyword value that '
 CONTINUE= 'is continued over three '
 CONTINUE= 'keywords in the FITS header.'

                    Dr. Nikolai Piskunov
                    Department of Astronomy
                    University of Western Ontario
                    London, Ontario N6A 3K7, Canada

---

From: seaman at noao.edu (Rob Seaman)
Date: Tue, 5 Jul 1994 20:37:51 GMT

>  CONTINUE= 'keywords in the FITS header.'

This is perhaps a better suggestion - or, at least, the alternative
offered of using a backslash (\) in a literal string as a continuation
character seems liable to break a lot of software.

However, nowhere in the FITS standard is there a mandate that the order
of header keywords must be preserved.  All of the options offered so
far assume that the order of the header is unchangeable.  Even the
simple addition of a new keyword, let alone a wholesale header
reordering or sorting, could disrupt a continuation card sequence.

Any continuation convention must either overtly preserve the continuation
order (within the context of the individual continuation cards), or the
FITS standard must be amended to require more stringent limitations on
header keyword operations.

Are continuation cards really needed?

Rob Seaman
NOAO/IRAF Group
seaman at noao.edu

---

From: mcs at goblin.caltech.edu (Martin Shepherd)
Date: 06 Jul 1994 09:39:18 GMT

In article <Pine.3.88.9407051430.A24934-0100000 at phobos.astro.uwo.ca> Nikolai Piskunov <piskunov at phobos.astro.uwo.ca> writes:
:On Tue, 5 Jul 1994, William Pence wrote:
:> ...
:> For example:
:> 
:> LONGSTR = 'This is a very long string keyword value that \'
:>           'is continued over three\'
:>           'keywords in the FITS header.'   
:> ... 
:> or to replace the blank keyword name with a
:> reserved keyword such as 'CONTINUE' followd by an equal sign, as in:
:> 
:> LONGSTR = 'This is a very long string keyword value that \'
:> CONTINUE= 'is continued over three\'
:> CONTINUE= 'keywords in the FITS header.'   
:> 
:> Is there any preference as to which of these 2 conventions we use?
:>
:
:I feel that the idea of using \' combination may cause problems for
:FITS readers written in C.

No. Back slashes and quotes are not significant in C programs except
when they appear in the text of a C file at compilation time.

:...
:On the other hand second example in WP mail shows certain redundancy.
:So what if we keep the keyword CONTINUE and forget the back slash?
:In other words, the example above will look like:
:
: LONGSTR = 'This is a very long string keyword value that '
: CONTINUE= 'is continued over three '
: CONTINUE= 'keywords in the FITS header.'

This seems logical, but I can see the value of having some indication
in the string being continued that the current keyword is incomplete,
rather than having to look ahead at the next keyword.

IMHO it would be better to side-step the whole issue of continuation
lines in FITS headers, and place such strings in tables.

---

From: gtr at mail.ast.cam.ac.uk (Guy Rixon)
Date: 6 Jul 1994 09:41:02 GMT

I for one would prefer the CONTINUE keyword to blank keywords in continuation lines.  If it's done this way,
the FITS reader can detect a continuation line without having to check the value field, which simplifies the
code.  I'd also prefer to keep the value field free of continuation symbols.  How about putting a sort key
in the comment field?

LONGSTR1= 'This string is going to run ' / 0001 <rest of comment>
CONTINUE= 'on to two lines...          ' / 0002 <foo>
LONGSTR2= 'This one is too, although   ' / 0003 <bar>
CONTINUE= 'I'm forcing the case...     ' / 0004 <bletch>

Since the sort keys are unique within a header the text can be reassembled however the descriptors have been
shuffled.  An unenlightened FITS reader will still get all the data, and will read meaningful text in the
value field, but won't do the reassembly for you.

Is this any help to you?

Guy.

---

From: drtr at mail.ast.cam.ac.uk (David Robinson)
Date: 6 Jul 1994 10:05:30 GMT

>If not we will probably use the original proposal, with blank continuation

Unfortunately, a FITS file with multiple CONTINUE keywords does not conform
to the NOST standard, and could be rejected by a fits reader.

Your first suggestion is the only way of doing this, (apart from numbered
keywords); however, I would suggest making your fits headers independent of
the keyword order, e.g.

LONGSTR = 'This is a very long string keyword value that \'
          LONGSTR 2 'is continued over three\'
          LONGSTR 3 'keywords in the FITS header.'

The keyword name and a seqence number appear in every record.

C programmers should have no problem with strings containinf '\'.

 David Robinson. (drtr at mail.ast.cam.ac.uk)

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Wed, 6 Jul 1994 14:12:19 +0200 (MET DST)

> The keyword name and a seqence number appear in every record.

  IF the need for long keyword really exists (which I feel questionable
  besides the case of HISTORY keywords), mainly for descriptive and
  mission specific purposes, why not just treat them as comments ?:

> COMMENT = 'LONGSTR This is a very long string keyword value that \'
> COMMENT = 'LONGSTR 2 is continued over three\'
> COMMENT = 'LONGSTR 3 keywords in the FITS header.'

---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
----------------------------------------------------------------------------

---

From: prep at yarrow.wt.uwa.edu.au (Paul Repacholi)
Date: 06 Jul 1994 13:46:50 GMT

> value of FITS keywords.  We will soon need to begin using this
					^^^^
>           'keywords in the FITS header.'   

This means interpreting the value of 'LONGSTR' as part of inputting
it. I would rather see
LONGSTR =    ".... that'\
LONGSTR =   'is cont...'\
LONGSTR =   'to a final un-escaped line'
type syntax.

Thus each line is a 'LONGSTR' to old reader, and the semantics are cleaned
up, and could extend to other keywords as well.

> If not we will probably use the original proposal, with blank continuation

Hum try this for parsing...

LONGSTR = 'This is a long string\'
CONTINUE= ' with\'
COMMENT = 'This is a comment with\'
CONTINUE= 'MORE THAN ONE LINE!! We think'

FITS seems to have a problem with lack of clear interfaces and
'levels' of function. Putting the continue char into the value
is an example of this. If the '\' is the LAST char, the reader
can strip it, the closing ' and everything upto the next value
start, then append the input to the buffer. Higher level code
need not ever know there has been a change. Except for the size
of the strings it will now be fed of course. ;-)

The word "need" is the real worry though...

--
~Paul
                                                       +61 (09) 257-1001
prep at yarrow.wt.uwa.edu.au ( preferred )                1 Crescent Rd,
zrepachol at cc.curtin.edu.au                             Kalamunda,
                                                       West Aust    6076

---

From: cur at star.rl.ac.uk
Date: Wed, 6 Jul 1994 15:54:54 +0100

The first question to address is "Are long strings necessary?".  Even
if you are prepared to lose the comments (undesirable IMHO) you are
limited to 68 characters, and 18 characters if you have a comment
starting at the normal column.  I can think of many instances where a
value in our data structures might exceed 68 characters, albeit not a
common occurrence, and where quite frequently 18 characters would be
inadequate.  If the OGIP people did not have this need also, I do not
think they would have gone to the trouble of making the proposal.  So
let's take it for granted, we need a way of storing header values
comprising up to a few hundred characters.

I do not like having such medium-length header-type information tucked
away in tables as Martin Shepherd suggests.  It has to be dealt with
separately, possibly being written to a separate file (what is it
called?) or not at all by some readers; thus the information can all
too readily become detached from the dataset it is describing.

So the question boils down to which ad-hoc convention does the job
with minimal disruption to existing software and does not limit
further developments.

Starting with the OGIP proposal.  The continuation-character
convention restricts the content of the string, albeit in a minimal
way, in that you cannot use a nominated character in a literal sense
at the end of a string.  In practice this is unlikely to be a problem
except perhaps for strings written by users ignorant of the convention
or in values written in existing files.  Since readers would have to
look at the next keyword to see whether or not it is a blank string or
CONTINUE, it can be handled.  Should readers complain if there is a
continuation character, but no continuation line, and treat the final
character as part of the keyword's value?  The above would suggest
not, but the following damning point against the OGIP proposal

RS> However, nowhere in the FITS standard is there a mandate that the order
RS> of header keywords must be preserved.  All of the options offered so
RS> far assume that the order of the header is unchangeable.  Even the
RS> simple addition of a new keyword, let alone a wholesale header
RS> reordering or sorting, could disrupt a continuation card sequence.

argues the contrary.  It is fine for systems written by observatories
and satellite missions, but not for those where users can input or
relocate new header cards.  Given header-validation software how
will it know if the continuation character was intended to be literal
or not?  I have probably laboured this point more than it merits from a
practical sense.  Whatever is decided the documentation of the convention
should be clarified accordingly.

WDP> LONGSTR = 'This is a very long string keyword value that \'
WDP> CONTINUE= 'is continued over three\'
WDP> CONTINUE= 'keywords in the FITS header.'

The redundancy question raised by Nikolai occurred to me too.  Why is
there a continuation character and a CONTINUE keyword?  Is it that the
continuation character flags to the reader that it has to look at the
next header card?

The CONTINUE card may have been used already for another purpose.
The fact that it is not part of the current standard, is less of
a problem for me.  We would have to modify the standard to allow
multiple CONTINUE cards.

Coming back to Rob Seaman's posting,

RS> Any continuation convention must either overtly preserve the continuation
RS> order (within the context of the individual continuation cards), or the
RS> FITS standard must be amended to require more stringent limitations on
RS> header keyword operations.

there have been some suggestions for the former.  I am not keen on
numbering schemes and ones that require looking at the comments
(shades of UNIX shell scripts) as Guy Rixon suggests.  I should rather
have the number in the string if it could be done unambiguously.  The
trouble with numbering is that you have to know which numbers already
exist, or at least the highest index, before inserting a new card.
That is not a big deal, just a little messy to code offering more
opportunity for violations, and is less efficient.  If you want to
connect keywords why not have a linked list by keyword name rather
than number? (-: I am probably prejudiced against David Robinson's
suggestion because it reminds me too much of Isaac Newton Group's
hierarchical keywords.  Given an arbitrary blank keyword (or COMMENT
in Lucio Chiappetti's proposal) you will not be able to have the first
word being another keyword followed by a number.  Probably not a
frequent occurrence, but it is possible.  The disadvantage of using
COMMENT keywords is that certain readers may detach the information
from the data the long string describes. I do not think it
unreasonable that a dataset has some ancillary information longer than
68 characters.

>From a selfish point of view in Starlink we can cope with either OGIP
proposal as we store the FITS header verbatim in our standard data
structures, and so order is preserved.  Presumably FITSIO users will
also be able to handle whatever is agreed without pain.  Of the
suggestions made so far, I would plump for 

WDP> LONGSTR = 'This is a very long string keyword value that \'
WDP>           'is continued over three\'
WDP>           'keywords in the FITS header.'

being the least undesirable.  However, I would be interested in other
proposals that avoid a specific ordering of header cards to if we can
improve on the OGIP proposal.

Malcolm Currie
Starlink Project

---

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Wed, 6 Jul 1994 17:22:49 +0200

Since all proposed continuation syntaxes violate the present FITS rule that
the header ordering is irrelevant, I'd like to see a better case for the
need of >68 characters in header values. Contrary to a statement in one
of the follow-ups to the original posting, some FITS readers (including
ours...) do use this feature of the present standard. The disruption
would be relatively minor since OGIP intends to use continuation lines for
mission specific information only (and our users will presumably not look
at any XTE product anyway), but this opens the door to a more general
use and potential problems. The modification will put a small but non-zero
burden on a number of people, so its proposers need to convince the 
community through a worked out example that their purpose cannot be met 
otherwise. 
	Thierry Forveille
	Observatoire de Grenoble

---

From: valdes at tucana.tuc.noao.edu (Frank Valdes)
Date: Wed, 6 Jul 1994 17:32:09 GMT

My personal preference is numbered keywords.  All parts of the string
should be preserved by any reader/writer and the order can be recovered
by software interested in it that knows the root keyword.  This is also
an existing practice.

However, the statement has been made that this is not possible for a
certain dataset (which was a design decision that might have been
different).  I would suggest that the same keyword be used for each part
and the index be part of the value field (with some variants following).
Before giving the argument an example would be:

LONGSTR = 1 'First part of this keyword'
LONGSTR = 3 'Third part of this keyword'
ANOTHER = 2 ' very long keyword'
LONGSTR = 2 'Second part of this keyword'
ANOTHER = 1 'Here is another'

The argument for using the same keyword is that all parts of a long
value are associated.  Use of blank, COMMENT, HISTORY, etc. can become
confused.  The index allows the order to be reconstructed if some
reader/writer does not preserve the original order.  Finally existing
readers will likely take one of four actions; just save all keywords,
save the first with the same name, save the last with the same name,
give a warning or error.  Also the value will either be preserved
completely or just the first part, whether it is the string or the index.

The main problem is having two parts to the value field.  A variant
would be: 'Second part of this keyword\2' (or some other delimiter).
In this case, unlike some other proposals, the reader doesn't necessarily 
have to scan the value.  If the reader is prepared for this syntax it
would only need to scan the value for ordering when a second instance
of a keyword is found.

A caveat is that I don't know without researching it whether there is
any mandate in the standard against multiple occurances of the same
keyword though I know that blank and HISTORY were specifically allowed.
So in a sense there is a precedent.

Frank Valdes
IRAF Group/NOAO

---

From: tennant at alph.msfc.nasa.gov (Allyn Tennant)
Date: 6 Jul 1994 23:25:15 GMT

In article <199407051645.MAA00598 at tetra.gsfc.nasa.gov> William Pence  
<pence at tetra.gsfc.nasa.gov> writes:
> (Do you prefer:)
> (or)

I vote for neither.  I object to the use of the backslash to denote
a continuation line.  This is for two main reasons:

1) For hundreds of years publishers have used a trailing - (hyphen)
to denote a continuation line.  Why create a new symbol to do the
same job?

2) In UNIX, C and many other parsers, backslash is an escape character
which means that the OS is to pass the following character without
interpretation.  Thus in C the backslash at the end of the line means
don't treat the following linefeed as an end of line marker but pass
it through.  In the two examples above the backslash precedes a single
quote character.  Thus someone who knows C et al. should naturally
read the line to mean that the single quote is part of the string.
I believe that using backslash in the proposed manner will make it
very difficult to use backslash in it's more normal meaning as an
escape character in the future.

In other words, using backslash as a continuation character only makes
sense if it is followed by a linefeed character, and this condition does
not arise in FITS.

						Allyn

---

From: leb at clark.net (Lee E. Brotzman)
Date: 6 Jul 1994 21:51:48 -0400

>when they appear in the text of a C file at compilation time.

Exactly right.  Parsing for backslashes poses no difficulty at all for C
programs.  Shells, which are written in C, do it all the time.  I prefer a
backslash because it does not appear in normal text (which can not be said for
the amersand, &, suggested by a previous poster).

>rather than having to look ahead at the next keyword.

I definitely DO NOT want a CONTINUE keyword.  In order to generalize the parser
I would have to change the algorithm to use look ahead for all keywords.  Yuch.
That is a very significant coding change.  If continuations are needed at all
(see below), then the keyword being continued must indicate that there is more
information to follow.  This can be coded much more simply as an exceptional, 
as opposed to a general, rule.

>lines in FITS headers, and place such strings in tables.

This I agree most strongly with, but in a different way.  My impression is that
these keywords would hold "one-time" descriptive information for an entire
image or table column.  Putting it into the table would introduce a lot of
redundancy.  However, I tend to doubt that there is a need here that can only
be met by breaking the standard, or otherwise introducing new conventions to
write code around.  I have been pretty successful using conventions in standard
COMMENT keywords to do exactly the same thing (kind of a mini-heiarchical
keyword scheme... works great for me at least).

Unless I can see a really compelling argument for this convention, I most
certainly would vote against it (if I'm still on the E-mail voting body, that
is, I haven't heard from Don in quite a while).

-- 
-- Lee E. Brotzman                    E-mail: leb at clark.net
-- Advanced Data Solutions            Phone : 301-776-0324
-- "Sometimes I think the surest sign that intelligent life exists elsewhere in
--  the Universe is that none of it has tried to contact us."  --- Calvin, 1990

---

From: bhill at trifle.gsfc.nasa.gov (Robert Hill)
Date: 7 Jul 1994 14:27:48 GMT

I don't like the idea of continuations, myself.  If I had this
need in my project, I think I would try to find some way to embed
the whole structure in history or comment cards, with some extra
code to break it out of there.  Alternatively, it seems to me that
a long string of mission-specific information must have some
substructure, so that it could be parsed into separate parameters
before being written to the header.

Of the schemes proposed, however, I prefer the backslash
continuation onto 'blank' comment cards as having the minimum
impact on FITS readers outside the project.  

I really dislike the CONTINUE= keyword.  I think what really
rankles me is the equals sign, which implies that CONTINUE is
somehow a parameter of the data set, like NAXIS or CRPIX1.
Really, CONTINUE is another keyword of the HISTORY/COMMENT/blank
class, and the equals sign is there only because FITS allows us to
invent new keywords, but not new history/comment indicators.
The invention of such an indicator specifically for continuing
strings would require the standard to be amended, and nobody wants
to get into that.

So, if you must, backslash + blank-comment card.

--
Robert.S.Hill at gsfc.nasa.gov  Not speaking for any of these:
Hughes STX Corp., Code 681, NASA/GSFC, Greenbelt, MD 20771
"There's nothing you can't prove if your outlook is 
only sufficiently limited." -- Lord Peter Wimsey

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 7 Jul 1994 17:30 EDT

I'm commenting on a number of points from different posts, and VNEWS 
doesn't have an obvious way of keeping the attribution straight from 
multiple non-sequential posts.  

On the question of repeated keywords...
	The NOST standard does not specifically address the general 
issue of keyword repetition.  The order and placement of most 
mandatory keywords is prescribed.  Presence in the header of more than 
one HISTORY, COMMENT, or [blank] is specifically permitted.  But there 
is no specification on reserved or user-defined keywords.  The issue 
of repeating this kind of keyword was left to the User's Guide for
discussion, rather than being prescribed in the standard.  The
relevant text in the User's Guide: 

	For keywords that have no value and are used to transfer text, 
	such as HISTORY and COMMENT, the same keyword may be used 
	repeatedly to indicate that the remainder of the card image
	contains text or comments.  However, repetition of a keyword 
	with conflicting values may cause confusion.  There is no
	standard interpretation in this case as to which value is to 
	be retained as the true value of the keyword.  Different FITS 
	readers may interpret the header differently.  Do not repeat a 
	keyword on different card images if the values conflict.

So multiple CONTINUE keywords would not be a violation of the 
standard.  

On distinguishing keywords without values....
	Except for COMMENT, HISTORY, and [blank], the contents of
columns 9 and 10 distinguish whether or not the keyword has a value. 
Except for those three keywords, the keyword has a value if and only
if columns 9 and 10 contain '= '.  Otherwise, columns 9-80 are comment
and there is no value. The reason for the exceptions in the cases of
COMMENT, HISTORY and [blank] is historical; the original FITS papers
did not forbid '= ' in columns 9 and 10, and this freedom must be
preserved, following the "once FITS, always FITS" rule.  The keyword
name tells the reader that there is no value.  For user-defined
keywords, though some way of distinguishing whether there is a value
is needed. 

One point to consider is that some readers may treat multiple
occurrences of keywords without values differently from multiple
occurrences of keywords with values.  

On the order of keywords in a header...
	To say that the order of optional keywords in a header is
irrelevant is perhaps too strong a statement.   The order is not
specified.  Because the order is not specified, FITS readers are not
required to preserve it.  Some don't.  This potential for card
shuffling is the reason that it is unwise to rely on card image order
in a FITS header. 

Finally, whatever convention is accepted should not break or even bend
FITS readers.  An example of how to construct such a convention is
the NRAO use of hidden keywords behind the HISTORY keyword -- also
used in the HIERARCH convention at ESO -- illustrated in Example 1 of
the User's Guide.  A reader that is familiar with the convention will
find the "keywords" that start in the later columns, and be able to
process them.  However, a reader that is unfamiliar with the
convention still has no problem; it simply treats the card images in
question as containing comments, and the information can be easily
obtained by the human reader by reading a listing of the header.  We
must keep the basic purpose of FITS in mind -- to allow the community
access to data without the necessity of constructing specialized
software for each data set read.  If we are to achieve this goal, we
must avoid creating conventions that will require patches to a
significant fraction of the FITS readers in use. 

 				Barry Schlesinger
				NOST FITS Support Office

---

From: leb at clark.net (Lee E. Brotzman)
Date: 8 Jul 1994 23:06:51 -0400

>up, and could extend to other keywords as well.

BINGO!  This is it. Except the syntax should be:

LONGSTR = 'This is the beginning of the string...'
LONGSTR = 'followed by even more of this string...'
LONGSTR = 'summed up by this string'

When one defines a new keyword, one must also define the keyword's value, either
integer, floating point, string, etc.  Therefore, to have "long strings" one
simply defines the keyword syntax as a string and that the keyword can appear
more than once, like COMMENT.  There is no specific prohibition against
repeated occurences of a keyword, so as far as I can tell, this is a legal FITS
solution to the "problem" (which, of course, I still do not believe really
exists).

It may even be better to define the keyword without the '=', thereby creating a
new form of COMMENT/HISTORY.  I'll leave that to Barry to see if it is legal
(frankly, his message talking about 'no value' for keywords without '=' was a
little confusing to me, just what does 'no value' mean?).

As a FITS programmer, how would I deal with this?  Basically, my code ignores
any keyword it is not specifically looking for; as is normal for FITS.  So to
begin with, the 'LONGSTR' keywords would have no effect whatsoever.
Continuation characters and blank keywords would, however, since my code scans
for blank keywords followed by text and interprets them as comments.  This has
the effect of all the blank keywords *following* the 'LONGSTR' keyword being
put into my comment structure, thereby rendering it incomplete and invalid.
If I actually *want* to scan for the 'LONGSTR' keywords, and if they are
repeating keywords, all I have to do is add something like this in my parser:

	Comment* LONGSTR;
	enum keywords {
		... , longstr, ...
	}
	struct keyentry {
		... , { "LONGSTR ", longstr}, ...
	...
	switch (keytype) {
	...
	longstr: ParseComment(value, LONGSTR);

A recompile, a relink, and voila!  The strings are read in and stored in total.
This does not qualify, in my mind, as writing new code, since I would have to
do the same thing for *any* new keyword I want to scan for.  Maybe it
isn't the prettiest solution, but it has worked quite well for me.

So, to sum up.  If indeed such a need exists (which I doubt) the best legal
solution is repeating keywords based on COMMENT syntax.  There.  We can all go
home now.  No wait, I'm already there!  Oh well, you know what I mean.

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Tue, 12 Jul 1994 16:17:16 -0400

We've been following the discussion of the continuation proposal with
interest and we appreciate the comments. Many variations on our
proposed convention have been suggested, (both recently, and during the
prior discusion of this issue in May 1993) but it appears to us that
there is still no consensus that any of them is significantly better
than our original proposal,  So, at this point we are inclined to stick
with our simple convention using blank continuation lines (although we
could be persuaded to use the CONTINUE= alternative; see point 3
below).  To be specific, this convention for encoding long string
values is:

    a.  Insert a backslash character (\) immediately before the
	closing quote character in the string value that is to
	be continued.

    b.  Continue the string value, enclosed in single quotes,
	on the next keyword in the header

    c.  The continuation keyword must contain ASCII blanks
	in columns 1-10 (i.e., the keyword name must be blank
	and there must be no equal sign in column 9).  

FITS readers that do not understand this continuation convention will
simply treat the continuation lines as COMMENT keywords.

A somewhat frivolous example of this usage is:

LONGSTR = 'This is a very long string keyword value that \'
          'is continued over three\'
          ' keywords in the FITS header.'   

or to give a more realistic example of the 'array descriptor' syntax 
to be used in the XTE FITS files:

TDDES2  = 'E(X1L|X1R|X2L|X2R|X3L|X3R) & C("TDDMAC1(0:14)") & \'
          'T([" at TIME",1024,0.00390625])'

      We don't wish to needlessly prolong the debate on this, but would like
to comment on a few issues that have been raised:

1.  Is the order of keywords in a FITS header significant?

The only statement made in the NOST FITS Standard about keyword order is:

     "Except where specifically stated otherwise in this standard, 
      keywords may appear in any order."

and the only places that the order is specifically stated is for the
required keywords at the beginning of each Header-Data Unit.  We
interpret this to mean that the creators of a FITS file are free to
decide on the most appropriate order for the keywords, but it does not
mean, in our view, that readers may come along later and arbitrarily
rearrange the order of the keywords without risking the loss of some
information.  Clearly, if comment keywords have been inserted in the
header at strategic places to explain the meaning of the preceding or
following keywords, or if history keywords are inserted to document the
order of the data processing operations, then rearranging the header
keywords will lead to loss of information.  Also, keywords may be put
in a specific order to make them easier for humans to understand, or to
make it more efficient for software to read them.  So we believe that
our convention that requires that the order of the continuation lines
be preserved is not unprecedented nor violates the FITS Standard.

2.  On the use of backslash (\) as a continuation character.

We prefer to use the backslash because it is a rarely used ASCII
character and is thus unlikely to be confused with a literal backslash
within the string value.  One can still write string keyword values
that end in a backslash, and these will not be confused with our
continuation convention as long as one of the 3 conditions listed above
is violated (e.g., insert a space character between the backslash and
the closing quote character, or make sure that the following keyword
does not have a blank name).

Some people objected to having any sort of continuation character at
the end of the line, but we think it is important to have some sort of
indicator to software that it should check the following keyword to see
if it is a continuation.  Otherwise the software would have to always
check the next keyword every time it reads a string valued keyword.

As was stated by several other people, the fact that the backslash has
a different meaning in other contexts (e.g., C source code and Unix
shell scripts) we believe is not a real source of confusion here.

Some people suggested putting the continuation character outside the
value field in the comment area at the end of the keyword.  But this
would require imposing new conventions on the format of the comment
field in keywords that we would rather avoid.  Also, some FITS readers
simply ignore the comment field entirely, so putting significant
information there could cause difficulties for them.  If the
continuation character is put in the value field itself then every FITS
reader will be able to get at it easily.

3.  Should the continuation keywords have a blank name, or some other
name.  

This in our minds is the biggest open issue about this proposal.  The
blank keyword names will simply be treated as comments by FITS readers
that don't understand this convention which we felt was advantagous
(i.e. is the least harmful way to deal with an unknown convention).
But if certain data systems strip comments from the header and
segregate them from the other keyword=value keywords, then this is not
so desirable.  This is why we suggested, as an alternative, using a
special keyword name for the continuation keywords (such as CONTINUE=).
While a number of people did to prefer this alternative, the votes
were not overwhelming in favor.  IF ANYONE ELSE HAS A STRONG PREFERENCE
BETWEEN THESE 2 OPTIONS, PLEASE LET US KNOW.

We, on the other hand, do not like the idea of repeating the original
keyword name on all the continuation cards.  This could cause
unpredicable results for FITS readers that do not understand this
convention (would they return the value of the first keyword, or of the
last?), and could be confusing if a FITS reader tried to delete the
keyword, but didn't delete all the continuations.  Under our
convention, if a FITS reader just deleted the first keyword of a
continuation sequence then the header would just be left with some
orphaned continuation lines that are interpreted as comments and do no
harm.

4.  Why not use a more complicated 'keyword referencing' scheme to point
to the next keyword in the continued string?

Some of the alternative proposals are completely independent of the
keyword order and still allow unlimited continuation lines.  We had
suggested such a scheme last year:

	KEYNAMEA= 'This is a very long keyword string value which \KEYWORDB\'
	KEYWORDC= ' of the FITS header.'
	KEYWORDB= 'continues over several lines\KEYWORDC\'

Some other variations have been offered in the past week.  But no one
we've talked to has much interest in implementing such a complicated
scheme, and neither do we.  Our convention is much simplier and still
serves our needs, so why introduce an even more complicated convention
into FITS that very few people are interested in using?

Regards,

Bill Pence
Arnold Rots

---

From: tody at lepus.tuc.noao.edu (Doug Tody)
Date: Wed, 13 Jul 1994 04:53:49 GMT

> into FITS that very few people are interested in using?

Bill - The above is more or less the convention used in IRAF, except that
we use numbers instead of A B C.  It works fine and is independent of the
keyword order.  The only disadvantage is that it is hard to squeeze both
the root keyword name and sequence number into an 8 character keyword.
But we don't find we need to do this very often (there are only a few cases
in all of IRAF), so it is not a big problem.

Note that in IRAF, only certain applications are expected to understand this
convention; generic FITS programs see only what appear to be separate
keywords.  Hence if we want to delete a "multiline keyword" we do something
like delete keywords "wsv*" to delete all the wsvNNNN keywords with a
generic image or FITS tool.  The same operation applied to an image using
your convention will delete only the initial header card, leaving ambiguous,
nameless continuation cards in the header.  Ditto for any other operation
such as copying or printing a keyword.  To properly deal with your headers,
all general FITS tools would have to understand your special convention.
This is not the case with the keyword based convention we use.

You are free to use any convention you choose for your data so long as it
is legal FITS - so long as you don't expect everyone else's FITS code to
understand it.  Anyhow, the point I wanted to make is that we don't agree
that your convention is "much simpler", or that the numbered continuation
approach is "something more complicated which very few people are interested
in using".      - Doug

-- 
Doug Tody	National Optical Astronomy Observatories    IRAF project
tody at noao.edu   P.O. Box 26732, Tucson, Arizona, 85726      iraf at noao.edu

---

From: Arnold Rots <arots at xebec.gsfc.nasa.gov>
Date: Wed, 13 Jul 1994 07:14:03 -0400

> in all of IRAF), so it is not a big problem.

The big problem with using numbers (and, hence, the general
applicability of your scheme) is, as pointed out in an earlier
message, that it does not work for column-related keywords - or any
other keyword that *has* to end in a digit.  This, by the way, is a
situation where one really gets into a crunch: a mandatory/customary
"T", three digits for the column number, at least one character for a
sequence indicator - and there is very little left to indicate what
the keyword is really about, especially if it follows some kind of
convention that prescribes yet another mandatory character.

  - Arnold Rots

---

From: tody at lepus.tuc.noao.edu (Doug Tody)
Date: Wed, 13 Jul 1994 13:14:01 GMT

> convention that prescribes yet another mandatory character.

Arnold, this is a good point.  I was not proposing this as a fully general
mechanism however.  Extending FITS to allow any string keyword to be
continued arbitrarily is a pretty major change and I was not proposing
anything of the sort.  We use this mechanism in IRAF only for a few special
cases and in these cases the 8 character keyword limit, while a nusiance, is
not a serious problem.  Frankly, in the cases where we do this there can be
a lot of data and we would be better off using a table - we use the multiple
keyword technique only because we don't do this often enough to warrant the
more complex and general approach (a table).  For most normal header strings
we live within the 68 character limit.

	- Doug

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 13 Jul 1994 16:59 EDT

In article <2vl48b$k7k at explorer.clark.net>, leb at clark.net (Lee E. Brotzman) writes...

>solution to the "problem" (which, of course, I still do not believe really

There is no specific prohibition against repeated occurrences of a 
keyword.

>new form of COMMENT/HISTORY.  I'll leave that to Barry to see if it is legal

It is permitted.  The meaning is that columns 9-80 of the card image 
are comment.  Users may define their own keywords that have only 
comment.  The controversial HIERARCH is an example.

>little confusing to me, just what does 'no value' mean?).

The operational definition is that a value field, following an "= " in
columns 9-10 must be of one of the specified types and follow the
format prescriptions for the type, for example, character strings 
enclosed in single right quotes.  Comment fields are not limited.  

A conceptual definition has not been stated and would require some 
careful thought.

				Barry Schlesinger
				NOST FITS Support Office

---

From: leb at clark.net (Lee E. Brotzman)
Date: 14 Jul 1994 22:51:14 -0400

William Pence <pence at tetra.gsfc.nasa.gov> writes:
>                                            ...it appears to us that
>with our simple convention using blank continuation lines 

A lack of consensus should make you step back and rethink the situation, not go
ahead with it. 

>simply treat the continuation lines as COMMENT keywords.

Right, and corrupt their comment structures by introducing incomplete and
confusing into them.  This is not a Good Thing (TM).

>          'T([" at TIME",1024,0.00390625])'

This looks like it could factor into more than one keyword and avoid the
problem altogether, but I won't belabor the point.

>         ...  Clearly, if comment keywords have been inserted in the
>keywords will lead to loss of information.

The "order of the keywords" pertains more to the relative order of *all*
keywords, I think, than the order of one kind of keyword (like COMMENT).
Putting COMMENT cards next to a keyword to describe it without clearly
labelling the COMMENT is asking for trouble later on, e.g.

TTYPE1   = 'SpType  '
COMMENT Types are arranged by color.

is poor style.  If a reader extracts information from this header, it may read
comment cards into a single structure and then spew them out as one, like so:

COMMENT Names are sorted alphabetically
COMMENT Types are arranged by color
COMMENT Velocities are relative to Sol

What goes with what?  The convention I follow at the ADC is this

TTYPE1   = 'SpType  '
COMMENT SpType: Types are arranged by color

The name of the field is in the comment.  The comment explicitly names the
field it is describing.  It doesn't matter (as much) if the comments get
shuffled to the top or bottom of the header.  This is a simple, non-binding
convention that works real well for our data.  My code will even detect this
construction and put the comments back adjacent to the appropriate keyword.
You may wish to look into something similar:

COMMENT TDDES2: E(X1L|X1R|X2L|X2R|X3L|X3R) & C("TDDMAC1(0:14)") & 
COMMENT TDDES2: T([" at TIME",1024,0.00390625])

This is readable, parseable, and best of all it conforms to the current
standard in every way.

>make it more efficient for software to read them.  

This is an aside from the main topic, but does FITSIO still destroy the
contents of inline comments following a slash character '/'?  I actually use
inline comments and it perturbs me to see FITSIO wipe out that information.  
It's important for making the header "easier for humans to understand".

>2.  On the use of backslash (\) as a continuation character.

No real argument here, except that it is unnecessary and unhealthy (for reading
software) to use your convention at all.

>(i.e. is the least harmful way to deal with an unknown convention).

The reason I object to your convention is because it is the *most* harmful way 
to introduce the information to a FITS reader that does not understand the 
convention.  You are partially using previously defined conventions, thereby
confusing just about all readers.

>keyword, but didn't delete all the continuations.

A reader that doesn't understand the convention almost certainly won't read the
keyword at all.  You are not suggesting using this convention for reserved
keywords like TTYPE are you?  I didn't think so, but if you are *please don't*.
The value returned by a reader that doesn't know the convention is a moot
point.  It will return incomplete (and supposedly useless) information in any
case, following any of the suggested courses.  That's why we've always been
told to just ignore keywords we don't understand.  I don't want to ignore the
blank comment keyword, though, although under your system, I won't have a
choice.

> Under our

I'm going to beat this horse into the ground.  Orphaned, incomplete, nonsense 
comments most certainly do harm.  They make headers more difficult to read 
and interpret correctly later on in the archival life of the data.

>to the next keyword in the continued string?

No argument here.  

To sum up.  Please do not follow the convention you initially proposed with
blank comment keywords to hold continuations.  It will not be nearly as
harmless as you think it is, and it defies the current standard.  It is bad
medicine.

---

From: leb at clark.net (Lee E. Brotzman)
Date: 14 Jul 1994 23:04:55 -0400

>enclosed in single right quotes.  Comment fields are not limited.  

Aahh.  Now I get it.  Thanks for the clarification.  This confirms then my
belief that using multiple instances of the same keyword and a comment (i.e.
'no value') format is the most appropriate form for "continuing" long strings.
For example:

TTDES1   This is the beginning of the structured TTDES1 field..
TTDES1   This is the end of the structured TTDES1 field.

Bill, I hope that you and Arnold will reconsider this and move away from
adopting a new interpretation convention for an established keyword (the
"blank" comment keyword in this case.)  The format above is indeed considered
legal under the current standard and will provoke the least problem for readers
that do not adhere to the convention.

---

From: bhill at trifle.gsfc.nasa.gov (Robert Hill)
Date: 15 Jul 1994 15:47:50 GMT

>A lack of consensus should make you step back and rethink the situation, not go

I second Lee's comment here.  In an earlier post, I was willing to
go along with the scheme of backslashes and blank comments, but I
believe I said `if you must' in there somewhere.  I still hope you
don't.  (Yet another alternative suggestion is made at the end of
this posting.)

>problem altogether, but I won't belabor the point.

On the face of it, I have to agree with this.  Why can't the E(),
C(), and T() constructs be made into separate strings?  In this
example, they aren't too long.  Could they be?

All right, suppose there is a compelling reason the value can't be
pre-parsed like that.  Now here's my concrete suggestion.  It's
really a combination of Lee's idea with something like IRAF-style
keyword references:

TDDES2  = 'COMMENT TDDES2:'      / Value exists in comment 
COMMENT TDDES2:  [the rest as Lee had it]
COMMENT TDDES2:  [etc.]
COMMENT TDDES2:  [etc.]

Your design constraints appear to be these:  You want the existence
of the parameter to be indicated at the level of regular FITS
keywords.  But you are willing to do a little extra work to get at
the actual string value.

OK, this scheme lets you tell immediately that the value is
present (because keyword TDDES2 exists).  It tells you that it
isn't in the keyword value itself (value starts with COMMENT), and
it tells you which comments to look in.  And just like the
backslash + blank-comment scheme, it could be applied arbitrarily
to any locally-defined string parameter.  If the word COMMENT
occurring at the beginning of a string value is a problem, you
could use something else, of course.

Best of all, it avoids the stylistic problem of values split
across two different *kinds* of FITS records.

-- Bob Hill

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 15 Jul 1994 16:33 EDT

In article <199407122017.QAA16849 at tetra.gsfc.nasa.gov>, William Pence
<pence at tetra.gsfc.nasa.gov> (and Arnold Rots) write... 
>... of the continuation proposal .... Many variations on our
>proposed convention have been suggested, ...  but it appears to us that
>with our simple convention ...

I concur with Lee Brotzman and Robert Hill.  Failure to reach 
consensus is not a justification for moving ahead; on the contrary it 
is a reason for not taking any action and discussing the issue 
further. 

Consensus is the established process by which standards are adopted. 
The NOST standard underwent three review cycles, during which the
community was given opportunity to comment.  The Technical Panel
considered all comments, either incorporating the substance into the
standard or determining the reason the response should not be
incorporated.  NOST also has an Accreditation Panel, whose role is to
be certain that in the development of any standard, the Technical
Panel makes all reasonable effort to notify the affected community of
the opportunity to review the standard, and that all reviewers
comments were addressed, either through revision, or through
explanation why there was no revision.  Those who followed the
development of Fortran-90 will remember the extensive review cycles,
-- so extensive that the language originally started as Fortran 8x --
and the significant changes made during the process. 

Standards are effective only when observed by the great majority of 
people in the affected area.  They will do so only if they believe 
that the standard is a good standard.  Hence the requirement for 
general, consensus acceptance.  If there is no consensus, the rules 
will not be followed.  Proceeding to define a convention in the 
absence of consensus will not produce a standard, only a local 
convention that is at least not understood elsewhere, and, very 
likely, misinterpreted by FITS readers.  

>      We ... would like
>decide on the most appropriate order for the keywords, 

Yes.

>but it does not
>information. 

But, as it does not forbid reorganizations, it does not mean that 
writers can rely on keyword order.

>Clearly, if comment keywords have been inserted in the
One reason for the failure of the KHBEGIN/KHEND hierarchy convention
(discussed in the User's Guide) to fail to gain wide acceptance was
the problem of keyword rearrangement.  What we're dealing with here is 
a requirement for readers to recognize a connection between different 
keywords, as opposed to recognizing a comment block.  In that sense 
the requirement is unprecedented.  And not everything that is 
permitted by the rules of FITS is wise. 

>shell scripts) we believe is not a real source of confusion here.

ANSI X3.4-1977 (code for information interchange) states, "Their [ISO 
and CCITT] recommendations, however, permit national standardization 
by the various countries in seven code table positions."

One of these National Use positions is 5/12 (hexadecimal 5C), which,
in the U. S. is \ (reverse slant or backslash). So the symbol is not 
necessarily internationally portable.

>3.  Should the continuation keywords have a blank name, or some other

There have been some suggestions by others in this thread that the 
continuation have the first eight columns blank, followed by '= ' in 
9-10.  While such a practice is permitted, the reason is basically 
historical: it was allowed in the FITS papers.  Except for the three 
original keywords, '= ' is not permitted in columns 9-10 of comment 
keywords, and using it in HISTORY, COMMENT, and [blank] is not 
encouraged.

				Barry Schlesinger
				NSSDC/NOST FITS Support Office