Subject: Re: OFSP Proposal on the use of EXTNAME,EXTCLASS & EXTTYPE keywords

From: CORCORAN at ndadsa.gsfc.nasa.gov
Date: Fri, 18 Jun 93 09:50:05 EDT

[This message was sent to "fitsbits-request" rather than to "fitsbits". -Don]
------- Start of forwarded message -------
Message-Id: <930618094224.22800397 at ndadsa.gsfc.nasa.gov>
From: CORCORAN at ndadsa.gsfc.nasa.gov
To: fitsbits-request at fits.CV.NRAO.EDU
Subject: Re: OFSP Proposal on the use of EXTNAME,EXTCLASS & EXTTYPE keywords
Date: Fri, 18 Jun 1993 9:42:24 -0400 (EDT)

Maureen Conroy at CFA writes:

>First, let me correct the statement that IRAF can only handle
>6 character keyword values.  This is NOT correct, though
>it is recommended in the FITS standards, due to limitations on
>many systems, that KEYWORD values be UNIQUE in the first 8 characters.
>For ROSAT FITS, I had recommended a guideline of 6 because most of
>our previously used extensions were 3 characters, and then combinations
>would naturally be 6 characters.  Also, in PROS, these keywords are then
>used as suffixes on the DATASET name to produce the final unique
>filenames; keeping all these as short as possible seems best for the user.

Sorry Maureen, I think I'm responsible for this nasty rumor.  But at
present for ROSAT FITS, as you point out, it is a requirement that EXTNAMEs
be restricted to 6 characters, without blanks.

>An easy way to provide automatic filenaming for FITS readers is to
>name the file according to the EXTNAME.  The FITS proposal
>for BINTABLE encourages this, and suggests that EXTNAME can be used
>to give a name to the extension file to distinguish it from similar files.
>This of course doesn't work, if there is more than 1 extension with the 
>same EXTNAME in a file.  Also with this in mind, I prefer EXTNAMEs
>that do not contain blanks.

Since some groups are using the EXTNAME to construct names for files, it
seems to me that use of blanks in EXTNAMES should be STRONGLY discouraged. 
In addition, the uniqueness issue argues against using EXTNAME to describe
the  "type" of file, hence the need for the EXTTYPE, etc., set of keywords.

>	Personally, I am concerned by the recent proliferation of proposed
>keywords.  I would prefer to see these issues handled
>within the existing definitions.  I think it would be
>more flexible to make separate FITS files for the different CLASSES and
>TYPES that are discussed in this proposal.  Then simple HISTORY or
>COMMENT keywords would be sufficient to record the processing history
>or derivation of the data sets.  This has the additional advantage
>that users and/or readers don't have to wade through long lists of
>events, in which they may not be interested, to get to the list of
>choice.

I wholeheartedly that existing keywords/structures should be used whenever
appropriate.   But I still think there is a need for a keyword in the
header which documents the "type" of extension the user is dealing with,
and I DON'T think that the EXTNAME keyword fills the bill.  

>	In this way, one can have for the current HEA EVENT lists:
>	a)  STANDARD event list FITS file containing extensions
>		EVENTS
>		GTI
>		TSI
>		QLIM
>		etc...
>	b)  REJECTED event list FITS file containing extensions
>		EVENTS
>		GTI
>		...
>	c)  ALL ( or unscreened ) event list FITS file containing extensions
>		EVENTS
>		GTI
>		...
>FITS readers have no need to know the processing history of these files.
>The user can choose which sets of processed data are to be read.
 
As you are aware (but probably most aren't) the ROSAT FITS files will
include the STANDARD events (i.e. those that have been accepted by standard
processing) and the REJECTED events (i.e. "bad" photons as determined by
standard processing) as separate extensions in the SAME file.  In such a
case the EXTTYPE = 'EVENTS' keyword is very useful to tell the user that
the file is an EVENT-type file.  The EXTNAMES for the 2 extensions are
different (STDEVT and REJEVT) which allows PROS to construct tables having
different names.  If we used EXTNAME = 'EVENTS' for both tables (as
originally suggested) then the PROS-constructed name for each table would
be the same.  I think the same problem would occur if the standard and
rejected photons were put in separate files, given that PROS (as I
understand it) constructs the table name by basically appending the EXTNAME
to the sequence number (which is preceeded by rp, rh or rf for US data). 
For example, suppose we had the rejected photons in a file
rp123456_rej.fits with an EXTNAME = 'EVENTS'  and the accepted photons in a
separate file rp123456_std.fits (also with EXTNAME = 'EVENTS').  My
understanding is that for each case pros would generate a photon events
list table called rp123456_events.tab, which I think would be confusing to
the user.  Using different EXTNAMES gets around this problem, but at the
same time it means that you CANNOT use EXTNAME to designate the generic
"type" of the file (which is the same for both extensions) thus the need
for EXTTYPE, EXTCLASS, etc.   

Mike Corcoran
ROSAT/OGIP FITS group
------- End of forwarded message -------

---

From: CORCORAN at ndadsa.gsfc.nasa.gov
Date: Fri, 18 Jun 1993 10:33:32 -0400 (EDT)

On 18-JUN-1993 Lucio Chiappetti writes:

>  I am really perplexed by this proposal, and I fail to see a clear
>  distinction between what is used for inter-package data exchange (i.e.
>  to make the file readable, or at least understandable to a "foreign"
>  package) and what is used exclusively within a GIVEN package.
>
>  My approach would be to use EXTNAME to classify in a broad sense the
>  kind of file. In particular in the high energy astrophysics context
>  the "kinds of files" I am using are :>
>
>     'SPECTRUM'               count/s (or photon/s) spectra
>     'RATE'                   time profile (anything vs time)
>     'PHOTON LIST'            list of photons or events
>
>      in addition to these I consider images and response matrices
>      (but in my view they are images, not bintables, therefore do
>      not use EXTNAME)
>
>  If each one of those supports variations, these are either in the
>  form (e.g. number and names of columns; and therefore are self-docu-
>  mented by other standard keywords like TTYPEn TFORMn etc.), or are
>  in the semantics (which is relevant within a particular package
>  only, and can be described by whatever optional keyword).>

I think there are 2 problems using EXTNAME in the way you've described:

1) Having an EXTNAME = 'SPECTRUM' does not tell
the user anything about the file contents, while EXTCLASS can be used to
give the user information about what standard the file was constructed to
follow.  For example, if EXTCLASS = 'SPECTRUM', this tells the user that
the file was constructed according to the OGIP spectrum extension document,
so that for example, the data are given by a TTYPEN = 'COUNTS  ' or TTYPEn = 
'RATE    ' and not 'SOURCE_CTS' or something else.  This standardization
is necessary in order to construct software that's multi-mission.

2) construction of unique filenames is not possible if EXTNAMEs are used to
construct the filenames and if a given "class" of file occurs more than
once in a dataset.

>  Being my primary interest in the possibility of EXCHANGING data
>  (i.e. importing/exporting them from/to my own package to/from other
>  packages) I would as far as possible avoid the use of "specialized"
>  convention, which may force to have a dedicated conversion utility
>  for each package, instead of a generic reader. 

The need for EXTCLASS, EXTTYPE etc. keywords was prompted so that a generic
reader could be constructed for files of the same "class".  This is not
possible if we rely on EXTNAME to tell us the "class" of the file, since
different groups have been using the same EXTNAMEs to mean different
things.  

>  As a conclusion, I do not see why you cannot use EXTNAME (instead
>  of EXTCLASS) for the top-level classification (to be kept to a FEW
>  classes, essentially the ones I listed above ... the exact names 
>  can be agreed if you are concerned with imbedded blanks, or more 
>  than 6 or 8 characters), and whatever YOU like for the bottom level
>  classification. 
>  Still, I perceive that the top-level classification (EXTNAME) is
>  of relevance within the X-ray (HEAFITS) community (but not necessarily
>  within a larger - fitsbits - community), while the lower level
>  classification is only "private" to your software.

I can't see how the use of EXTNAME to denote the top-level classification
will allow the construction of a generic file reader for each class of
file, which is I think a universally agreed-upon goal, given the fact that
an EXTNAME='SPECTRUM' means different things to different groups.  Even if
we get all groups to agree that a file with EXTNAME = 'SPECTRUM' has
certain required fields and keywords (a difficult goal to attain), this
does not ensure that all files with EXTNAME = 'SPECTRUM' follow the adopted
specifications, since a file with the EXTNAME = 'SPECTRUM' may have been
constructed before the specifications were adopted.  This argues for the
creation of the EXTCLASS, EXTTYPE keywords.

Cheers,

Mike Corcoran
ROSAT/OGIP FITS group

---

From: Lucio Chiappetti <lucio at IFCTR.MI.CNR.IT>
Date: Wed, 23 Jun 1993 10:45:50 +0200 (MET DST)

I'd like to continue the discussion about EXTNAME etc.
I'll enclose below some comments to Mike Corcoran's posting.

I'd also like some of the guys who proposed the BINTABLE extension and/or
who are mantaining the standards to comment on the issue.

From the binary table document (I'm using Appendix A of the NOST document)
I see there are already three (optional reserved) keywords to define
extensions (EXTNAME, EXTVER, EXTLEVEL), although their intended usage is
not altogether clear (otherwise we won't be here discussing :-) ) and I'm
afraid of proliferation of different conventions.

> construct the filenames and if a given "class" of file occurs more than

  I (at least) never said or intended to use EXTNAME as "filetype".
  By filetype I mean the component of a full file name in a form like :
  /path/path/filename.filetype or {path.path]filename.filetype;ver
  Also from the NOST document, Appendix A, it is not obvious that EXTNAME
  was intended for that.

  Concerning the habit of sticking more extensions in the same FITS file,
  I regard this as something to be avoided (however that should be irre-
  levant to you since you read the FITS extensions directly and do not
  read them into separate "native" (IRAF, MIDAS or whatever) files).

  I'll use that only for TRANSPORT and not for native storage, in one
  particular case (and anyhow I use the IMAGE extension in such case).

  The conventions to give a filename.filetype are by definition dependent
  on whatever package one is using. 
  I personally use a FILENAME keyword which contains the file name without
  path (site-dependent), and without filetype (package-dependent).
  If a filetype is needed (as a recommended value), one could add a
  FILETYPE keyword.

  It is however true that I construct a default filetype based on the
  "content" of the file. This occurs when I read back from FITS to my\
  native format. My default filetypes are ... guess what :

      .image
      .spectrum
      .rate
      .photon
      .matrix   (and .histo)

  My native format is not FITS, but is one-to-one mappable. When reading
  a file, I use whatever filetype the user supplied (if he did, and if
  he wants to call a spectrum pinco.panco or pinco.photon instead of
  pinco.spectrum that is his damn business), or if he did not supply
  one (as in the majority of the cases, jsut said "pinco") I construct
  it from "context" (a fitting program will expect .spectrum, an image
  display one .image etc.). Then when I have constructed the file name,
  I open the file and check it is what expected (if not error exir).

  The check uses a magic number in the file header which is structured
  in four parts (none of this is FITS, but some of this is used to
  generate FITS keywords when I convert the file to FITS ... most of my
  header keywords have the same name of FITS keywords, but a few FITS
  keywords get generated on the fly) :

    XAS\001    identifies that the file conforms to my convention
               (this is so far ignored when converting to FITS)

    xxx\002    where xxx=IMG or BIN, indicating the file is an image
               or a binary table (this controls conversion to FITS
               as a primary HDU, or an XTENSION='BINTABLE')

    yyy\003    where (for xxx=BIN) yyy=SPE,PHO,TIM ... indicating the
               file is a spectrum, photon list, time profile ....
               (in FITS this generates EXTNAME)

    zzz\004    is a code indicating the operating system on which the
               file was written (this is obviously irrelevant to FITS)

    So my hierarchy is : is the file according to my convention (yes); 
                         is the file an image or a bintable ? 
                         and if a bintable what is it ?
    I do not feel the need of any further specialization, since I kept
    the definition of different types at minimum, according to Ockham's
    razor ("file structures non sunt multiplicanda praeter necessitatem")  

> is necessary in order to construct software that's multi-mission.
  what do you mean exactly by multimission ?

  i)  designed to support any future mission which will be compliant to the
      standard data format, and any other past and future mission which is
      not compliant by means of converters
  OR
  ii) designed to support directly the many different data formats and
      conventions already supported by past missions ?

  Me, I'd go for i) 

  The goal is : make sure that the files contain ALL NECESSARY information
  and the conventions used are documented (if any spectra contain for
  instance low and high channel boundary energy, cts/s and errors, AND the
  name of the columns are documented, a converter manipulating table
  columns can always be written ... in Fortran, MIDAS, IRAF, Ftools or
  whatever ....)   

> .... could be constructed for files of the same "class".  This is not

> I can't see how ....
> ... EXTNAME='SPECTRUM' means different things to different groups. Even if

  As I said, get rid of the past (hide them behind converters), and try to
  have at least all high energy groups to agree they mean the same thing !
  
+-------------------------------------------------------------------------+
| I feel that what is needed here is NOT a convention about EXTNAME, or   | 
| any lower level convention, but just a keyword which allows the user to |
| readily identify ... the conventions according to which the file has    |
| been written (the equivalent of the first field in my magic number).    |
|                                                                         |
| This could be in your case ORIGIN='OGIP' or EXTNAME='OGIP' or           |
| CONVENTN='OGIP'.                                                        |
+-------------------------------------------------------------------------+

Is this an acceptable and useful thing to do ?

After this (since it is not covered by the full standard) you could do
whatever you like with lower level keywords, I could do whatever I like,
anybody else can do whatever one likes ... and one can always read the
file as a generic binary table and manipulate it within one's package of
choice ... or write a converter from your convention to mine (for some
cases I have a prototype), or from your to somebody else's etc. 

-----------------------------------------------------------------------------   
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E          
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager           
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE            
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |                                    
Bitnet:   <please do not use any more>     |                                    
-----------------------------------------------------------------------------

---

From: thompson at serts.gsfc.nasa.gov (William Thompson)
Date: 23 Jun 93 14:13:39 GMT

Just to put my 2 cents worth in.  For the SOHO/CDS FITS data files, we were
planning to use EXTNAME to uniquely identify the data stored within the
extension, not what kind of data it is--all our data is of the same kind.  We
currently envision only using one extension (binary table) per file, with the
primary HDU being empty.  However, it does seem advantageous to retain the
capability of merging several related datasets in separate extensions within a
single FITS file.

Bill Thompson