Subject: HEASARC Continuation Keyword Convention

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Fri, 21 May 93 17:07:46 EDT

Thanks to everyone that commented on our proposals on 'Continuation
Keywords' in FITS headers.   Nearly everyone who expressed a preference
for one of our 3 proposals preferred the first option, thus we intend to
start using this, when necessary, in the FITS files produced by the
HEASARC.   Under this convention, a string keyword value may be
continued over multiple keywords in a header by terminating the first
line of the string with a backslash, and continuing it on the next
keyword record which has a blank keyword name (the full definition of
this convention and an example are given below).

In reply to some other comments that were made:

Some suggested that since FITS headers have such a rigid (and
primative?) format, it would be better to completely redesign FITS
rather than try to add patches to the existing system to support new
features.  This approach may have its merits, but in the meantime, we
have to live with the current implementation of FITS and need to
continue to find ways to adapt it to new situations.

Doug Tody rightly pointed out that this proposal has significant
implications when one needs to delete, copy, or otherwise manipulate
string keywords.  When performing such operations, one must treat the
entire set of continuation keywords as a unit, otherwise the header may
get messed up.  This is very true, and I intend to modify the FITSIO
software, in particular, to handle these situations transparently for
the user.  FITISO will then automatically write a string value over
multiple keywords using this convention if it is more than 68
characters long.  Similarly FITSIO will automatically concatenate the
long string together when reading a FITS header. (There are also lower
level FITSIO routines that may be used to read or write each individual
header record, if desired).

Contrary to Lee Brotzman's concern that this will eat up
too many CPU cycle testing for continuation lines for every string
keyword, I don't think this will really be very expensive.  One has to
parse the keyword record anyway to find the position of the closing
single quote character marking the end of the string, so it is pretty
trivial then to test the previous character to see if it is a
backslash.

Finally, several people expressed a preference for a scheme 
where continuation keywords are numbered as in

KEY_1    =  'This is a long string tha\'
KEY_2    =  't is continued on two lines of the header.'

This is fine in simple cases, but in many of the cases where we
envisage needing this facility, the keyword name already ends with a
sequence number, (e.g.,signifying the binary table column to which it
applies).  It would be complicated to append a second continuation
sequence number to these keywords.  In addition, in many cases we are
already hard pressed to encode all the desired information into an
8-character name, so we would not want to have to use some of these
precious characters as a continuation sequence indicator.  Also, it
would be useful if the continuation convention was general enough such
that it could also be applied to any previously defined  FITS keywords
such as 'TELESCOP', or 'OBSERVER' which could not be done if it was
required to add a sequence number to the keyword name.

The following is the precise definition of this continuation
convention:

  The HEASARC Convention for the Continuation of FITS String Keyword Values

                   William Pence and Arnold Rots
                            NASA/GSFC
                           21 May 1993

This convention defines how to specify the value of a character string
keyword which may be too long to fit within the 68-character limit of a
single keyword value.  This is done by inserting a backslash character
immediately before the closing single quote character of the first
keyword, and by then continuing the value string, again enclosed in
single quote characters, on the next keyword in the header which must
have a blank keyword name.   The value string may be continued over
multiple blank keywords as long as each previous string ends with a
backslash character.  Optionally, a comment string may follow the
quoted value string on any or all of the continuation keywords.

Example:

OBSERVER= 'Sir Nathanual Hawthorn Jacobs \'      / proposer's full name
        'Mitchael Fitzpatrick David Millheart\'  / this string starts in col. 9
             ' Rosebud Washington, III'          / proposer's name (cont.)
END

---

From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Date: 24 May 93 13:23:41 GMT

>trivial then to test the previous character to see if it is a

I may not have worded my comment correctly, but really I was complaining about
wasting *any* time checking *every* string for continuation.  Not just too much,
but any, including the time it will take to rewrite existing code.  Your 
convention will dictate that *all* FITS readers must be rewritten in order to 
correctly interpret your headers.  This is very costly, and I wish you would 
take a hard look at other, more FITS-like ways to solve your problem.

Also, I was being flip about the "constitutional convention".  FITS was designed
in another era, and the fact that you feel you must use a convention that, 
although it doesn't break the letter of the standard, certainly breaks the 
"spirit" of the standard, provokes me to goad the powers-that-be that FITS 
isn't everything it is cracked up to be.  We wouldn't want them to get too 
smug, now would we?  :-)

I know my opinion doesn't count for much against the FITS heavyweights, but
please, take another look at what it is you are doing and look hard at what
effect you will have on other software groups.  Not everyone has a full-time
staff of programmers to modify their code when you hit a syntactical
stumbling block.

-- 
-- Lee Brotzman

P.S.  *Now* I wish I was going to the Berkeley AAS meeting for WGAS.  I'll wind
      up missing one just when it might get interesting.  That figures.

--
-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS

---

From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Date: 24 May 93 14:09:24 GMT

>quoted value string on any or all of the continuation keywords.

If you are going to do this (and I still prefer you don't), you should
establish an upper limit for the length of the string.  Buffers need to be
allocated to concatenate each piece of the string, and programmers will need
guidance on what a safe limit is.  My suggestion is 68.  :-)

No really, pick a reasonable limit like 500 or 1000 characters.

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Mon, 24 May 93 16:33:11 EDT

Lee Brotzman wrote (regarding our proposal for continuation keywords):

> ... (stuff deleted) Your convention will dictate that *all* FITS
>readers must be rewritten in order to correctly interpret your
>headers.  This is very costly, and I wish you would take a hard look at
>other, more FITS-like ways to solve your problem.

We decided to go with the current proposal because it seems like
best alternative we could think of, and because no one else
suggested a viable alternative.

The only other suggestion we've seen is to use a numbered sequence
of keywords (KEY1, KEY2, KEY3 ...) but this is unacceptable to us
for several reasons:

1.  This doesn't work if the keyword name already ends with a sequence
number.  In our applications we use FITS binary tables almost exclusively
and many of our keywords are applicable to a specific column, so the
keyword names end with the column sequence number (as in TDIMnnn).
It is simply not practical to try to encode a continuation sequence
number into these keyword names.  There are only 5 characters left
in the name to play with and we don't want to reserve 1 or 2 of
them for a continuation character.

2.  Using a numbered set of keyword names is a very application
specific solution for individual cases, and is not a general
solution to the continuation problem.  Basically, you have
to decide ahead of time whether a particular string keyword
value is going to exceed 68 characters and if so, define a
numbered naming sequence for it.  Then all the software that
reads this keyword has to know that it must read in an array
of keywords to construct the whole string.  But if you
define a non-numbered keyword for an application, and then
only later discover that the value will sometimes exceed 68
characters, then you are stuck.  You would have to define a
whole new set of numerical indexed keywords, and would have
to change all your software to look for this new set of
keywords.  

In the long run, I think our proposal will actually involve
less change to existing software than the case-by-case 
numerically indexed keyword solution.  To implement
our general solution you just have to modify a single 
keyword parsing subroutine (as I will do in FITSIO) to handle
the continuation convention.  Then just relink your application
software and that is about it.

3.  While not essential, our proposal is retroactively applicable
to any existing defined string keywords (e.g., TELESCOP).   Otherwise
one would have to define a new set of seqential keywords (TELEnnn) for
this use, which would probably not be recognized by any existing
FITS readers.

Finally, we are certainly not requiring that all other groups
immediately support this convention.   Ideally, we would hope that
this convention becomes endorsed by the wider FITS community.
In the meantime, we have to implement some solution to this problem
for immediate use within the HEASARC and this seems like the best 
alternative.

>No really, pick a reasonable limit like 500 or 1000 characters.

We didn't set a specific limit because it seemed like a completely
arbitrary decision.  Is there really a need for an upper limit?
If so, is there a objective way of deciding what the limit 
should be?

-Bill Pence
HEASARC, NASA/GSFC

---

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Mon, 24 May 1993 22:57:49 +0200

I am jumping somewhat late into this continuation line discussion, so sorry
if my point has already been addressed and I have missed that part of the
discussion.

I tend to agree that the continuation character is the least disturbing option
to existing FITS readers (which tend to ignore blank keywords and will thus
truncate the continued string) if continuation lines need to be implemented.

I have however a slight impression that we are putting our cart before the 
horses since I haven't yet seen on the list a justification for this need.
Most of the data I have actually came across only use a rather small fraction
of the 68 allowed characters. Aren't we going to use header keywords when
we should use something else?

	Thierry Forveille
	Observatoire de Grenoble

---

From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Date: 24 May 93 21:19:22 GMT

>If so, is there a objective way of deciding what the limit 

There's nothing wrong with being arbitrary.  FITS is full of arbitrary.

Pick a size that will be sufficient for all time, or at least for a few
years.  Something that I and other programmers can set and depend on (at least
in a #define).  I *could* set aside a 10,000 character temporary buffer to
read in strings, but that's wasteful.  Pick 500 or 1000 (I generally dislike
using powers of two just for the sake of it, a philosophy that I picked up
from being a fan of REXX).

I'll quote a response I penned earlier today to Bruce O'Neel when he asked a 
similar question about using dynamically allocated arrays for these continued 
strings:

	"My original point, ... was that in order to dynamically allocate an 
	array, I have to give [malloc] a size, and in order to determine the 
	size I have to concatenate all the pieces, and in order to concatenate 
	all the pieces I have to have an array large enough to hold them all, 
	and in order to have one large enough I have to have an upper limit.  
	The alternative would be to scan through all of the continued strings, 
	counting lengths, then allocate the string I need and *rescan* the 
	continued keyword.  Yuch.  Even worse than having continuations in the 
	first place."

Basically I have to hold onto the continued string somewhere before I can store
it in the structure where it belongs, which will now have to be dynamically
allocated in every case, since I can't predict the length of strings, which
means more guard code against memory leaks, etc. etc. ad nauseum.  68 was such
a nice, arbitrary limit.  Sigh.  Those were the days.

If you have an alternative algorithm, please let's hear it.  I could very well
be missing something simple.  Otherwise, just pick a number.  I'll vote for
1000 (unless 68 is already taken  :-).

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Tue, 25 May 93 16:37:00 EDT

>we should use something else?

This is not meant to be a flame, but I am rather surprised at how the
question 'Why would you ever need more than 68 characters?' keeps
getting raised.  I would turn this around an ask 'How have FITS users
been able to cope so long with this restriction?'.  With just a little
thought it is easy to come up with many  examples of long strings
that one might want to put into header keywords:

  - file names, including the full node:user:disk:direcory:subdirectory path
  - a bibibliographic reference, including the title of an article
  - postal address
  - a caption for a figure or plot
  - a long array of numbers (given as a quoted string)
  - Within the HEASARC we are running into a number of more specific cases
with respect to fully describing the contents of various FITS binary
tables which in general could require quite long strings.  

In the past it seems that FITS users have either just learned to live
within this limit, or have developed rather adhoc solutions to cover a
few specific cases where long strings could not be avoided.
What we are proposing by this new 'continuation convention' is a
general solution to this problem which will be applicable in all
situations.  This is in keeping with the philosophy that FITS
must continue to evolve to meet the growing and changing needs of
the users.

-Bill Pence

---

From: jwp at sal.wisc.edu (Jeffrey W Percival)
Date: Tue, 25 May 1993 21:19:15 GMT

In article <leb.738278362 at Hypatia> leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes:
:I'll quote a response I penned earlier today to Bruce O'Neel when he asked a 
:similar question about using dynamically allocated arrays for these continued 
:strings:
:
:	"My original point, ... was that in order to dynamically allocate an 
:	array, I have to give [malloc] a size, and in order to determine the 
:	size I have to concatenate all the pieces, and in order to concatenate 
:	all the pieces I have to have an array large enough to hold them all, 
:	and in order to have one large enough I have to have an upper limit.  
:	The alternative would be to scan through all of the continued strings, 
:	counting lengths, then allocate the string I need and *rescan* the 
:	continued keyword.  Yuch.  Even worse than having continuations in the 
:	first place."

Use realloc(3) as you find each continuation.  No upper limits need be known.
-- 
Jeffrey W Percival (jwp at larry.sal.wisc.edu) (608)262-8686

---

From: bhill at stars.gsfc.nasa.gov (Robert S. Hill)
Date: 25 May 1993 18:25 EST

In article <9305252037.AA17717 at tetra.gsfc.nasa.gov>, 
pence at tetra.gsfc.nasa.gov (William Pence) writes...
>been able to cope so long with this restriction?'.  With just a little

This is also not meant to be a flame, but since you asked...

>  - file names, including the full node:user:disk:direcory:subdirectory path

Gee, this bugs me, but I can't quite say exactly why.  My own
approach to this sort of thing is not to specify a filename, but to have
a list of named resources that gets translated to filenames somewhere
outside the FITS realm -- e.g., a bunch of flatfields for the different
detectors and filters on our telescope.  If I specify these things by
actual filenames, the difficulty of porting the batch processing system
across platforms is increased.

I don't have a real strong objection to putting filenames in
FITS headers, but I have always tried to avoid having a reason to
do it.

>  - a bibliographic reference, including the title of an article
>  - postal address

Why wouldn't these go into history cards?  To me, keywords are like
a parameter space describing your data, and stuff that is really just
plain ol' text goes more naturally into history/comments.

>  - a caption for a figure or plot

Seems a bit outside FITS's bailiwick to me.

>  - a long array of numbers (given as a quoted string)

Better in this case to bite the bullet and make a keyword array with the
XXXXX001, XXXXX002 method. People have invented huge arrays, e.g., HST
Guide Star Selection Survey astrometry coefficients (not the CDm_n
keywords, the other ones that go with GSSS images themselves; there must
be a couple of dozen of them).

>tables which in general could require quite long strings.  

To me, this is your best case by far.

IMHO, people's expectations are starting to stretch the boundaries of
FITS a bit, even factoring in extensibility.  It's fine to ask FITS to
expand to fill the users' needs, but design aesthetics matter, too.  In
fact, they matter a lot. Following one's sense of what's clean and
consistent with the past is just a way of looking into the crystal ball
to anticipate and forestall  future snarls.

I hope you understand that we are not trying to keep you from doing what
your project requires.  It's just that the continuation proposal has
obviously rung a mild, low-volume sort of ugly design alarm for some of
us.  (Not that FITS has never done that before :-)

Like I said before, I don't really *hate* it to where I'd fight tooth
and nail, and I say if you must do it, then do it, but it still makes
me feel funny.

          Bob Hill
----------------------------------------------------------------
bhill at stars.gsfc.nasa.gov
Hughes STX Corp.
Code 681, NASA Goddard Space Flight Center, Greenbelt, MD 20771

---

From: tody at noao.edu (Doug Tody)
Date: Wed, 26 May 1993 00:04:56 GMT

>   - a long array of numbers (given as a quoted string)

Gee Bill, maybe you ought to use an auxiliary table for such information?
It is often simplest to put the occasional long string in a header using
multiple keywords, but if you find yourself doing this often you may be
using the wrong representation for your data.  A table can easily accomodate
large strings or other vector elements.  A simple table with 3 or 4 columns
(keyword name, value string, comment, and maybe a datatype code) would be
logically equivalent to the FITS header, but would avoid most of the
limitations, i.e., keyword names, value strings, and comments can be as long
as you want.  Sure it is a nusiance to have another table but it does avoid
all these problems.  If you find your data continually running into the
limitations of the simple FITS header (such as you suggest) I think you
ought to seriously consider this option.             - Doug

---

From: gsh7w at fermi.clas.Virginia.EDU (Greg Hennessy)
Date: Tue, 25 May 1993 23:42:03 GMT

Jeffrey W Percival writes:
#Use realloc(3) as you find each continuation.  No upper limits need be known.

And what about those who use computers that don't have realloc? Not
very flexible it seems.

--
-Greg Hennessy, University of Virginia
 USPS Mail:     Astronomy Department, Charlottesville, VA 22903-2475 USA
 Internet:      gsh7w at virginia.edu  
 UUCP:		...!uunet!virginia!gsh7w

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Wed, 26 May 1993 15:42:58 +0200 (MET DST)

I apologize for coming late into this discussion, but I was away for most
of the last two weeks. I picked up the various messages last Friday as
"travel reading material", and I am now reading the forthcoming discussion.
Most of my comments have already been raised, I reiterate them here mainly
to give further support.

1) REAL need for string > 68 characters

   1.1) is it REALLY necessary ?

   This has been questioned, and I would say that I agree with most of
   the questions raised (i.e. I generally do not see a need for this,
   beside some special cases).
   In fact I have been recently defining a data format for our internal
   use (this format is not FITS, but was designed to be 1-to-1 mappable
   to FITS for transport). In this format I store keywords in a header
   (more properly a trailer, to allow extendability) in binary format,
   essentially as :

     1 byte :   code indicating the data type (CHAR, I*4, R*4, R*8)
  *  1 byte :   length of the data field in bytes (n)
     8 char :   NAME of the kewyord (8 char for FITS compatibility)
     n char :   value(s) of the keyword (if the keyword is CHAR this is
                a string of length n, if the keyword is numeric, it is
                an array of n/4 I*4 or R*4 or n/8 R*8 ... in general the
                array is actually a single scalar)

  From the fact that I use one byte for the length (field marked as *) a
  nominal limit of 255 characters can be derived for string keywords. I
  originally wanted to limit this to 246 (so that 1+1+8+n = 256), but I
  soon decided to impose a further limit, and use 68 JUST IN ORDER TO BE
  COMPATIBLE WITH FITS.

  This is somewhat the opposite approach to HEASARC. I started with a
  non-FITS format for storage, and accepted to put limitations in it in
  order to be compatible with FITS for transport. HEASARC started with
  the choice to use FITS also for storage, and is now finding some
  intrinsic limitations in it ...

   1.2) in which cases it might be necessary ?

   Some persons asked for examples. Most of the examples provided are
   of "documentary" nature. I would in general agree with the comments
   supplied by Robert Hill. Namely :

   - filenames including a full path : ARE THESE OF ANY RELEVANCE OUTSIDE
     THE SITE where the files were produced ? My personal approach is to
     store in the header only the name part (e.g. in the case of a file
     names /xas/calib/xmm/epic/pinco.matrix, I would store in the header
     only "pinco" ; the extension ".matrix" should be implicit in the file
     format, and the path should be site dependent (e.g. set according to
     environment variables)

   - information like bibliography, addresses, captions look to me to be
     there FOR DOCUMENTATION ONLY, therefore they belong to the class of
     "comments". See point 4 below.

   - an example of a complex string descriptor for XTE was made. This 
     appeared to me really awkward !! Is not really possible to arrange
     things to be simpler ?

   - finally, the case was made of a keyword describing an array of
     numbers. This is something I find sometimes useful. For instance if
     an instrument has seven PMTs, it might be useful to record their HV  
     setting as an "array kewyord" PMTHVS of 7 numbers. This is possible
     in the scheme I described above (but I use binary representation),
     but is quite FITS-unlike ... or un-FITS-like ?). I had the problem
     of writing this into FITS for export and PROVISIONALLY used an
     arrangement as follows : 

        in the FITS file I put a character string PMTHVS which says
        'this kewyord is an array from PMTHV1 to PMTHV7' 
        and I follow with seven numbered keywords

     I am not completely happy with this arrangement, and so far it is
     not reversible (I mean I can write my file into FITS for documentary
     purpose, but cannot read such "array kewyords" back yet), and, what
     matters more, is NOT INTEROPERABLE. That is none of the readers I
     use follows that particular convention.

     I know by looking at MIDAS-produced FITS files that MIDAS writes
     array descriptors (how keywords are named in MIDAS) as strings, e.g.
     PMTHVS = '105 104 106 107 93 100 102', and this convention is
     reversible within MIDAS, but I am not aware of any related documen-
     tation.
     I am also not aware if IRAF has any similar facility.
     Maybe fitsbits subscribers directly involved in MIDAS and IRAF might
     comment on that.

*****
     I presume that HERE there is a VALID case for defining an
     INTEROPERABILITY standard. But this should be called "a standard
     for array of numbers in FITS kewyords" and not a "standard for long 
     character strings"
*****

2) Compatibility with the past

I regard the issue of compatibility with the past (and present) of paramount
importance (the so-called "once FITS forever FITS" principle), and I agree
with the comments by Lee Brotzman on this.

Defining a standard which no other reader can use defies interoperability
(which is the modern way to interpret the "T for Transport" in FITS.
                                                                 ^

Of course it may be legitimate instead to use a local convention which
is useful at one site, or within one package, but this convention shall
apply only in such a way not to hinder anybody else to use the data with
other packages. In the present case this means that the convention for
continuation keywords should allow readers to skip them, and that their
interpretation is not essential for the analysis. 

3) Maximum length

I feel that a maximum length should be defined, if reconstructing the
entire value is necessary for the analysis (if is is just of "documetnary"
nature, it might be truncated in a harmless way). This is said from the
point of view of the Fortran programmer.

My *personal* preference is for this length to be no more than 255
character, for the very simple reason that my own software may be already
compatible with it, in the case I'd decide to support the new convention.
(but this is obviously not a strong reason).

4) Suggestions and conclusions

My strong preference is NOT to define any new standard which forces ALL
FITS readers and dependent software (i.e. analysis packages) to be 
rewritten in order to allow use of keyword values longer than 68.

This also implies that the information written in such keywords is NOT
to be used outside the site where the files are produced or with other
packages than those used there. I.e. that all other software can SKIP
such information (treat it as comments) in a harmless way.

In this sense it is ACCEPTABLE to follow a convention in which one has

1234567890
KEYWORDX='value blah blah \'
         ' continues \'
         ' etc. etc. up to the end'

provided "columns 1 to 8 are left blank" in order to comply with the
standard about comments (NOST 100.03b pag. 19)

IF however the information to be written in strings longer than 68 is
essentially of "documentary nature", that is is there just to remind an
human of how the file was produced, it might be treated as a COMMENT
or HISTORY and split on several lines without any mandatory indication
of continuation.

I am personally using the following convention in my files. I do not
allow a given named keyword to appear more than once in an header, but
for HISTORY, COMMENT and PARENTS (this is a local usage to keep trace
of the ancestor files to a given file). In general a file header looks
like :

some keywords
freely interspersed with COMMENT keywords
one or more HISTORY keywords -------------------------------------
some more keywords
one or more HISTORY keyowrds -------------------------------------
etc.  

Each set of HISTORY keyowrds records the run string of the command which
generated or modified the file. In general a command may also modify a
file in place, altering some keywords, and adding some new ones, but will
add a new history section. The HISTORY case is the only case in which I
ever had to write things longer than 68 characters (command run strings),
and I was happy to do this splitting them in more lines, with continuation
lines starting with a plus sign. 

As an example (this is taken from an header lister output, so it is not
in FITS, but it should be obvious how it could look in FITS):

 (J) BITPIX       =            8
 (J) NAXIS1       =           16
 (J) NAXIS2       =        12232
 (J) TFIELDS      =            6
 ......
 (C) HISTORY      = oapxas targetcrab  900.0 target_crablike
 (C) COMMENT      = Converted to focal plane units
 (C) COMMENT      = Original file modified in place
 (C) COMMENT      = THETA PHI converted into X Y
 ......
 (C) HISTORY      = focalplane targetcrab 7500.0
 (C) COMMENT      = X Y shifted by DELTAX DELTAY
 (J) DELTAX       =        -5000
 (J) DELTAY       =         5000
 (C) HISTORY      = relocate targetcrab    -5000     5000
 (C) COMMENT      = Converted to single chip pixels
 (C) COMMENT      = X,Y converted to X,YPOSITN
 ......
 (C) HISTORY      = chipsize targetcrab targetcrab_eev1_c  27.  768 1024   0.
 (C) HISTORY      =  + MOS_EEV_framestore_top_chip  0.100

The last HISTORY kewyord is continued on two lines. This is used for visual
inspection only, and the fact that, being FITS-like, it fits (sic!) into
80 characters allows easy display on any terminal.

I am NOT advocating this latter (the "plus" convention) as a proposed
solution, it is just an example to indicate that, IF "long" keyowrds are
needed ONLY for documentation they CAN be handled as "comments"

-----------------------------------------------------------------------------   
       A member of  G.ASS : Group for Astronomical Software Support             
-----------------------------------------------------------------------------   
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E          
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager           
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE            
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |                                    
Bitnet:   <please do not use any more>     |                                    
-----------------------------------------------------------------------------

---

From: pmurphy at cv3.CV.NRAO.EDU (Pat Murphy)
Date: Wed, 26 May 1993 19:33:41 GMT

"GH" == Greg Hennessy <gsh7w at fermi.clas.Virginia.EDU> writes:

GH> Jeffrey W Percival writes:

JP> Use realloc(3) as you find each continuation.  No upper limits need
JP> be known.

GH> And what about those who use computers that don't have realloc? Not
GH> very flexible it seems.

realloc is part of posix.1 so it's available on at least a fairly
reasonable subset of systems.  At least unix systems...  and no, this
doesn't answer the question.
				- Pat
--
==========================================================================
| Patrick P. Murphy, Ph.D.                Scientific Programming Analyst |
| National Radio Astronomy Observatory    Net:       pmurphy at nrao.edu    |
| 520 Edgemont Road                       Phone:     (804) 296-0372      |
| Charlottesville, VA 22903-2475          VoiceMail: (804) 980-5889      |
|      "I don't believe in the no-win scenario"  --- James T. Kirk       |
==========================================================================

---

From: jwp at sal.wisc.edu (Jeffrey W Percival)
Date: Wed, 26 May 1993 16:20:55 GMT

In article <C7LwI4.Eo5 at murdoch.acc.Virginia.EDU> gsh7w at fermi.clas.Virginia.EDU (Greg Hennessy) writes:
>And what about those who use computers that don't have realloc?

The original comment referred to a call to malloc(), hence my reference
to realloc().  Are you referring to computers that have one but not the other?

I realized that many development environments have neither, but realloc()
exists in both Ultrix and Suns, so it seems my assumption that malloc()
implies realloc() is not too bad.

I take your point, however, that there are implied system dependencies
involved.
-- 
Jeffrey W Percival (jwp at larry.sal.wisc.edu) (608)262-8686

---

From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Date: 26 May 93 19:21:30 GMT

>  - file names, including the full node:user:disk:direcory:subdirectory path

HOSTNAME= 'MyHost'
DOMAIN  = 'anywhere.com'
SUBDIRS = 3
SUBDIR1 = 'Topmost'
SUBDIR2 = 'nextdir'
SUBDIR3 = 'lastdir'
FILENAME= 'Thisis.fits'

This eliminates OS-specific file name conventions, which is important for data
interchange.  It can translate to 
	"MYHOST::TOPMOST:[NEXTDIR.LASTDIR]THISIS.FITS" or
	'MyHost.anywhere.com:/Topmost/nextdir/lastdir/Thisis.fits' 
as appropriate.  I've mentioned this convention before, but I don't think it 
will catch on simply because there isn't much need for fully qualified file-
names in FITS.  The 'T' is for 'transport' and the filenames are generally 
useless at someone else's site.  But since we are in the networking age, we 
would do well to avoid *any* OS dependancies in the names since we are not all 
running the same operating system (yet).  Now that I think of it, I can see 
having keywords like:

NETWORK = 'INTERNET'
PROTOCOL= 'TCP     '
ACCESSBY= 'ANON_FTP'

And my FITS reader can crank up FTP to log in to HOSTNAME.DOMAIN, follow the 
path from SUBDIRn, and retrieve FILENAME.  Slick.  I'll have to think about 
that a little more.

>  - a bibliographic reference, including the title of an article

REFERENC= 'Pence & Brotzman 1999, AAS Bull., 199, 235'
COMMENT This FITS file conforms to the FITS-99 standard as defined in Pence &
COMMENT Brotzman 1999, "FITS for the next century: a standard remarkably 
COMMENT similar to the old one except for the racing stripes and mag wheels."

Seriously, I never had a problem citing references using the ApJ style in the
REFERENC keyword and then embellishing them in COMMENTs.  The only snag I hit 
was data that stemmed from more than one paper, but it isn't very hard.

>  - postal address
Use COMMENT, or make the address parseable by using keywords like ADDRESSn,
CITY, STATE, ZIP.  A raw address that I can't parse is basically worthless from
a data processing point-of-view.

(Actually, I can't see any need for a non-COMMENT keyword for a postal address 
unless your FITS reader can lick stamps.  :-)

>  - a caption for a figure or plot
Hmmm.  There's a new one.  I presume the figure or plot would be some sort of
bitmap, right?  I'd put the caption in one field of a BINTABLE and the plot in
another.  The caption is data more than metadata in this case because it's 
something intrinsic that goes with the plot.

>  - a long array of numbers (given as a quoted string)
TABLE or BINTABLE.

>tables which in general could require quite long strings.  

A specific example would be helpful here.  Are the descriptions for MACHINEs or
PEOPLE?  Machines need keywords, people need comments.

My philosophy was always that field names and other meta-data keywords, like
TUNIT, are akin to variable names.  In that sense you need uniqueness and some
sense of content more than complete descriptions. If further explanations of
contents are needed, use comments.  On the ADC CD-ROM I used a simple variant
of the heirarchical keywords proposed some time back to 'tag' comments to a
specific field, for example, from the Strasbourg Planetary Nebulae catalogs:

TTYPE9  = 'Morph_Flag'         / Morphology flag
TBCOL9  =                   34 / Start column
TFORM9  = 'A1      '           / Fortran format
TNULL9  = '        '           / Null value
COMMENT Morph_Flag: '<' for star-like appearance, '>' for fainter spherical
COMMENT Morph_Flag: envelope, or else blank.

How does this differ from your proposed convention?  Well, it follows standard
practice, uses well-defined keywords (COMMENT) for their original purpose, 
and maintains the semantic content even if the COMMENTs "float" to another 
place in the header.  Granted it will still suffer if the ordering gets 
shuffled, like any multi-line comment, but for me it has a better "look and 
feel".

>situations.

Which is precisely why I'm leery of it.  Should the needs of the few [keywords]
dictate the syntax of all?  I'm not so sure, especially when a more
conservative approach actually results in more transportable data (like avoiding
fully-qualified filenames).  Sometimes a straight jacket is a useful tool.

I don't want to drag this out any further.  Your convention doesn't break the
letter of the standard, only the "spirit" of the standard and you are free to
do as you please.  I happen to think you are making a mistake.  Since one of my
goals is to build BINTABLE browsing software with a sharp eye to HEASARC data,
I guess I'll have to learn to live with that mistake.

>This is in keeping with the philosophy that FITS

The time I served on the FITS Technical Panel for NOST made me very sensitive
to design changes.  I would much prefer a complete overhaul rather than a few 
piddly tweaks here and there.  I'm realistic enough to know that an overhaul 
ain't gonna happen, but that doesn't make me more amenable to tweaking.  FITS 
is the MS-DOS of astronomical data interchange.  It isn't the best solution by 
far, but it's the one that won the acceptance battle.  That's a shame, but one 
we must live with.
--
-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS

---

From: cflatter at cv3.CV.NRAO.EDU (Chris Flatters)
Date: Wed, 26 May 93 21:29:25 GMT

>doesn't answer the question.

Correction: realloc() is part of the ANSI C library so if you have a C
development environment that doesn't have it, it's time you threw it
away and got a new one.  Of course, realloc() is of no earthly use to
you if you are reading FITS from FORTRAN 77.

Remember that many FITS readers are FORTRAN-based for historical
reasons and there is no way to dynamically allocate or reallocate an
array or character string in FORTRAN 77.

	Chris Flatters
	cflatter at nrao.edu

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Fri, 28 May 93 10:31:49 EDT

I hope this is about my last reply on this subject;  life's too
short to dwell on this indefinitely. :-)  Anyway, here's a
few parting comments:

Some people raised objections to my examples of possible uses of long
strings, which I think was missing the point.  Even if you don't like
some of my examples, just about anyone who works with data files for
very long can come up with legitimate examples in their own areas of work
where it was necessary to somehow store the value of a long string in
the data (several people cited such examples).   After all, the
68-character limit is completely arbitrary,  so surely nobody can
really argue that there is never a good case for representing a longer
string in a FITS header.  It seems people have spent a lot of creative
effort to come up with more or less elaborate schemes to work within
the 68 character FITS limit.  To my way of thinking, all of these
alternative work arounds ('jumping through hoops' as someone said) just
helps to prove my point that we really need a simple scheme for
allowing longer character strings in FITS headers so that everyone can
use the same convention, rather than having to deal with a multitude of
local home-brewed conventions for dealing with this general problem.

To perhaps alleviate some of the concerns that this issue has raised,
the HEASARC doesn't intend to immediately start using this convention
everywhere in all of the FITS files we produce.  We certainly realize
that most other sites will not support this convention, at least not
immediately, so we will use it sparingly, mainly for mission specific
keywords that only our local software systems will know how to
interprete and only in cases where there is no good alternative.
Eventually, I would hope that the various FITS committies will endorse
this, or a similar scheme for handling long strings, but in the meantime
we have to adopt something to use for now.

Below is our latest draft of this convention, including the proposed
upper limit to the string length.  We felt that limiting the length
to 256 characters, as some preferred, was not enough of an increase
over the current limit and could still be too restrictive for some
applications.  Therefore we decided on a limit of 1023 characters
(plus an optional hidden end-of-string character, as
used in C and SPP, which would then require a 1024 byte memory buffer).
This is not a strict upper limit however, and groups may encode
arbitrarily long strings using this convention for their own local
use, on the understanding that other FITS readers are free to
ignore anything after the 1023rd character (treating any additional
lines simply as comments).

-Bill Pence

-------------------------------------------------------------------------

                           28 May 1993

There is no explicit upper limit to the length of a continued string
under this convention, however, FITS readers in general will not be
expected to be able to read strings more than 1023 characters long.
Thus, one may use arbitrarily long continued strings in FITS headers
for special purposes, but it is understood that other FITS readers may
ignore any characters beyond the 1023 character limit and treat
any addition blank continuation keywords as 'COMMENT' keywords.

Example:

-------------------------------------------------------------------------

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Fri, 28 May 1993 16:38:37 +0200 (MET DST)

> Below is our latest draft of this convention, including the proposed

  Still, I'd like to see in the convention an EXPLICIT statement that the
  continuation lines have the first 8 characters blank, so that they can be
  treated as comments by most readers ...
  ... and perhaps also a statement that the information coded in such
  keyword is not necessary to be interpreted for generic software to
  access the data (although this is required by site-specific software)