Subject: TSORTKEY and continuation proposals

From: Clive Page <cgp at star.le.ac.uk>
Date: Wed, 6 Jul 1994 09:07:00 +0000 (GMT)

(1) Sort order
 
I agree that we need a way of specifying the sort order of FITS tables. 
Many existing tables are actually sorted on some column or columns; for
example the astronomical catalogues distributed by NSSDC on CD-ROM mostly
seem to be sorted on right ascension, but at present you have to check all
through the table to discover this.  Many applications will be able to
access such tables much more efficiently if the sorting order is declared. 
The proposal by Bill Pence seems simple and effective and I'd like to
support it. 

(2) Strings that are so long that they need to be continued.

I'd prefer the second option, that is to have the continuation lines
labelled with 'CONTINUE= '.  This is because lines with a blank keyword
may not be processed properly by current systems.  It is true that the
FITS Standard does not require the order of keywords to be preserved, so
any system of continuations is liable to fail, but in practice lots of
existing FITS files have a sequence of keywords containing COMMENTs that
only makes sense if they can be read in sequence; all the systems that I
know of seem to preserve their order in such cases. If so then maybe
the backslash character at the end of the string being continued is not
necessary.  The backslash can be hard to handle in C programs, and many
Fortran systems running on Unix also treat the backslash as an escape
character.  If we can do without it, so much the better. If we do need a 
continuation character then I'd prefer the ampersand "&" as this is 
consistent with Fortran 90.

! Clive Page,  Dept of Physics & Astronomy, University of Leicester, UK.
! e-mail:  cgp at star.le.ac.uk
! phone:   +44 533 523551           Fax: +44 533 550182

---

From: prep at yarrow.wt.uwa.edu.au (Paul Repacholi)
Date: 08 Jul 1994 18:43:39 GMT

In article <Pine.3.87.9407060900.A11900-0100000 at ltsun6> Clive Page <cgp at star.le.ac.uk> writes:
...
> know of seem to preserve their order in such cases.

In the interest of the golden rule "once FITS, always FITS" it seems
that preserving order should become a REQUIRMENT in FITS.

1) No existing files will be affected.

2) Software packages will have two less free variables to worry about.

3) Any multi-line form of extention will reqire that ordering and
  grouping be preserved, so this will be needed as part of Bill's
   proposal.

I am still not happy with a syntax form that embeds packing info
into a value. The problem is not that it is \ or & or -, but
that it apeares inside the value. So is the \ part of the string?
Inband signaling is very seldom a good idea!
--
~Paul
                                                       +61 (09) 257-1001
prep at yarrow.wt.uwa.edu.au ( preferred )                1 Crescent Rd,
zrepachol at cc.curtin.edu.au                             Kalamunda,
                                                       West Aust    6076

---

From: tody at lepus.tuc.noao.edu (Doug Tody)
Date: Sat, 9 Jul 1994 05:47:26 GMT

>   grouping be preserved, so this will be needed as part of Bill's

This is easy to say but it is not that simple.  The possibility of
reordering header keywords occurs when the FITS header is edited.  Often the
header contains related groups of keywords.  For example, the header may
contain a world coordinate system (WCS) specification stored in a number of
related keywords (any number of other examples are possible).  If the WCS
associated with the image changes it may be that the number of keywords
required to describe the WCS changes.  One must delete all the old WCS
keywords and insert the new WCS keywords.  It is possible to insert or
append the new group of keywords as a sequential block, but there is no
guarantee that all the original WCS keywords will be consecutive.  Hence one
deletes all the old WCS keywords wherever they may appear in the header, and
inserts or appends the new keywords somewhere else, the logical place being
near the beginning of the header since these are standard keywords.  This
may mean that the order of the keywords changes.  Preserving the original
order could be difficult or impossible.

The situation can get even worse.  It may be that the header is ambiguous,
containing multiple copies of the same keyword, e.g. two WCS
specifications.  Which is correct?  If the WCS is modified do we delete both
sets?  If one thinks of the FITS header as time ordered the last entry would
take precedence, but the defacto FITS practice is to order the header with
the "standard" keywords near the top, followed by an arbitrary number of
application specific keywords, comments, history etc. in the latter part of
the header.  Hence to preserve the logical order of the FITS header one
wants to consolidate the system keywords near the top.  If the input header
is poorly structured it may need to be reordered when written back out by
the software which edits the header.

Another example would be combining two images and writing a single new
output image.  How do we combine the two headers without reordering at least
some of the keywords, and probably removing some duplicate keywords?

Obviously one wants to preserve the original order of the header keywords
where possible, but the point is that this is not always possible or
desirable.

Note that this problem only arises when using FITS as a runtime format where
headers are subject to dynamic updates.  Since FITS is intended as a static
archival and exchange format issues such as dynamic updates or reordering
keywords are not officially an issue.  Nonetheless it is an issue for real
data systems, since we all ingest FITS files, generate new images, and write
out new FITS files, and one wants to preserve the information in the input
headers so far as possible.

-- 
Doug Tody	National Optical Astronomy Observatories    IRAF project
tody at noao.edu   P.O. Box 26732, Tucson, Arizona, 85726      iraf at noao.edu

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Sun, 10 Jul 1994 17:20:35 +0200 (MET DST)

...
  ...
  
  The point is that a FITS header is somewhat an awkward structure to be
  handled in memory, and on disk, since it is an almost fixed (once written) 
  number of card images in a fixed number of 2880-blocks. Extending the
  header will require writing a new file (a reason not to use FITS as
  working format, see below).

  Hence the point is : 

    either read all the header in memory, and REPLACE the values of
    changed kewyords, or EXTEND the header allocating more memory and
    finally FLUSH the header to disk

    or read "wanted" keywords by name from the input file and write them
    to the output file eventually with a modified value. What to do then
    with "unwanted" or "unexpected" keywords ? Ignore them ? or do a
    further pass and write out those not already written (which implies
    they are flagged) ?

> specifications.  Which is correct?  If the WCS is modified do we delete both

  Is this a "legal" FITS file ?
  I'd assumed that, besides COMMENTS and HISTORY, duplication of keywords
  was not allowed or at least discouraged. 

> some of the keywords, and probably removing some duplicate keywords?
   Yes indeed

> archival and exchange format issues such as dynamic updates or reordering
  ^^^^^^^^^^^^^^^^^^^^^ 

  YES INDEED

---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
----------------------------------------------------------------------------