Subject: FITS Image Extension support in IRAF

From: perry at eclipse.stsci.edu (Perry Greenfield)
Date: 15 Aug 1995 16:36:26 GMT

The Science Software Group (i.e., the STSDAS group) at the Space
Telescope Science Institute has been involved in a cooperative
effort with NOAO to modify the IRAF imio kernel to support the
FITS Image Extensions within IRAF as a native disk format. We
have tentatively settled on a specification for how we see the
interface working and have been implementing a kernel to this
specification. The implementation we come up with may be modified
as a result of comments from NOAO or may later be enhanced or
modified by NOAO before distribution as part of IRAF v2.11.
This posting is intended to serve the purpose of notifying the
FITS community about our intended use and interpretation of the
FITS format as well as solicit comments about its compliance with
the standard.

Before discussing the details, some brief background. The calibration
pipelines being developed for the two new instruments being installed
during the 97 servicing mission, STIS and NICMOS, will produce
significantly more datasets than are now being produced for the
existing instruments. This is a result of generating associated
data arrays for all science images that will contain data quality
and variance information. Furthermore, both instruments have modes
which produce multiple images for each observation (possibly more
than 50 for NICMOS). FITS Image Extensions have been chosen
as a means of reducing the number of file generated by combining
all the associated arrays for an observation into one file. The 
following issues have arisen in the design of the data files and
the ease in which a user may access an image contained in such
a file.

I) "INHERITANCE"

There are two aspects of the interface we are developing that relate
to issues with the FITS standard. The first deals with relation of
the keywords in the primary header to the extension images that
follow in the same file. This particular aspect engendered much
discussion and consideration. There is an intuitive expectation
among many that the Primary Header keywords should apply to all
extension headers, but in fact it seems that most FITS software
makes no such assumptions, i.e., all extension headers are treated
as independent of the Primary Header. The FITS standard makes no
explicit reference to this issue. After much discussion it was
decided to include a mechanism for Extension Header Units (here-
after EHU) to "inherit" Primary Header Unit (PHU) keywords. The
scheme  being adopted is that the EHU will inherit the PHU
keywords if it contains a line specifying "INHERIT=T". In the
absence of such a line (or if "INHERIT=F" exists) no inheritance
of PHU keywords will take place. In cases where the PHU and EHU
both contain the same keyword, the EHU value takes precedence.
If the EHU is propagated by the applications program to an output
FITS file, the inherited keywords will then appear in the new
EHU (or PHU if it is written as the Primary Unit).

An obvious problem with this scheme is in the effective duplication
of all PHU keywords in the output EHUs after one step of processing
or data reduction effectively defeating the usefulness of having
"global" keywords in the PHU so they wouldn't be repeated everywhere.
To avoid this needless replication, we have incorporated a mechanism
to override the inheritance when opening an Extension Unit. This
is accomplished by specifying an option as part of the filename/
extension selection specification. This is intended to be used
primarily by the applications programs that expect files in FITS
Image Extension format (e.g., the future STIS and NICMOS calibration
pipeline programs) and expect the "global" PHU keywords. In this
situation, we expect that the application program will disable
inheritance, read the PHU and EHU as separate entities, and
propagate them separately. The inheritance of keywords is accomplished
by the use of routines that search the EHU and PHU (in that order)
for a given keyword.

A possibility being considered (but certainly more difficult to
implement) is to use the output PHU to determine if the keyword
in the combined input PHU/EHU exists in the output PHU and has the
same value. If so then the keyword would not be written to the
output EHU. This scheme would also prevent the automatic duplication
of PHU keywords in output EHUs.

Since most FITS readers will have no knowledge about our local
convention regarding keyword inheritance, it will be important for us
(and any others for that matter) to ensure that all keywords that
would be reasonably expected to be useful for analysis be present
in the EHU even if they are the same for all Extensions. (This still
leaves a healthy number of keywords for the global header as far
as HST data products are concerned!)

II) EXTENSION NAMES

The second issue related to the FITS standard is the selection
of Image Extensions. The obvious and straightforward way to
specify an extension is to refer to it by number. The convention
we have adopted is to refer to the Primary Unit as extension 0,
and the first Extension Unit as 1. We have also adopted a naming
scheme so that Image Extensions can be referred to by name. For the
time being the name is defined by the combination of the values
of the EXTNAME and EXTVER keywords, if they exist (future enhancements
by NOAO may include more keywords as part of the naming scheme).
This naming convention is essentially identical to that proposed
in "A Hierarchical Grouping Convention for FITS" by Jennings et al.

The planned data products of the future instruments will incorporate
one or more "groups" of images into one FITS file. Such a group might
consist of a science image, data quality image, and error image. There
are occasions where multiple groups will be in the same file. Our
plan is to use EXTNAME to specify the image type and EXTVER to specify
the number in the sequence, e.g., SCIENCE,3 would specify the first
extension with EXTNAME=SCIENCE and EXTVER=3.

There are many more details involved in specifying the IRAF imio
interface for FITS image extensions, but they don't directly relate
to FITS issues. Anyone who is interested in those details can email
me for the specifics.

NULL IMAGES:

There is one last issue that does not involve the IRAF imio interface
directly but is related to planned HST data products and the FITS
standard. The data products plan to use "NULL" Data Units for two
distinct cases. For the first case, and intended to be used for all STIS
and NICMOS data, the Primary Data Unit will be a NULL array. This is
so that all the data products can be referred to by name, and a use of
either EXTNAME or EXTVER in the Primary Header Unit would clearly be
at odds with their definition. The second use of NULL arrays is when
certain data arrays, for example data quality arrays or error images,
will be filled with a constant value. In these cases it makes sense
to use a NULL array with a keyword in the EHU specifying the constant
value. The IRAF imio kernel will *not* automatically convert these
Extensions into constant arrays. It will be up to specific application
programs to interpret the meaning of these NULL images.

Perry Greenfield

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Tue, 22 Aug 1995 12:22:00 -0400

I have 4 suggested changes or additions to the proposed STScI local
FITS convention for the INHERIT keyword (posted by Perry Greenfield on
17 Aug).  As it stands, I think the proposal is  too general or vague,
and could cause inappropriate keywords to be inherited in some cases
which could confuse users and possibly lead to incorrect results from
the analysis of FITS files using this convention.

Here are the suggested changes to the proposal:

  1)  The required keywords in the PHU shall never be inherited.

  2)  The PHU must have NAXIS = 0 for any keywords to be inherited.

  3)  If software modifies the value of an inherited keyword, that new
      keyword is simply added to the EHU, and the value of the keyword
      in the PHU is not changed.

  4)  Any COMMENT, HISTORY, or blank keywords in the PHU shall not be
      inherited.

The following sections describe why I think the above restrictions
are needed.

1.  Inheritance of the required PHU keywords:  

There could be problems if the required FITS keywords are inherited
from the primary header unit (PHU).  Clearly the SIMPLE keyword must
never be inherited.  If the primary array has NAXIS = 3 and the
extension has NAXIS = 2, then the extension must not inherit the NAXIS3
keyword from the PHU.  Also, it should be forbidden to have an
extension header unit (EHU) with just the following keywords:

XTENSION= 'IMAGE   '
INHERIT =                 T
END

with the expectation that the other required keywords would be
inherited from the PHU.  (This would violate the requirements for a
FITS conforming extension in any case).  All of these potential
problems can be avoided simply by restricting the inheritance proposal
to state that the following required or reserved PHU keywords shall
never be inherited:  SIMPLE, NAXIS, NAXISn, EXTEND, PCOUNT, GCOUNT, and
END.

2.  Inheritance of inappropriate PHU keywords by an Image extension:

If the primary array is not null (NAXIS != 0) then there may be
keywords in the PHU that should not be inherited.  For instance, the
BSCALE, BZERO, DATAMIN, and DATAMAX keywords might well only apply to
the primary array.  One solution to this problem would be to ensure
that these keywords are always repeated in each EHU, so they will never
be inherited, but due to negligence or ignorance users may forget to do
this, potentially leading to disasterous results when analyzing the
FITS file.  There is a subtle pitfall here, too, in that often the
absence of a keyword is used to imply a default value.  BSCALE and
BZERO for example should be assumed to have values of 1.0 and 0.0 if
not present in the EHU, and one should presumably not inherit the value
from the PHU in this case.  This also applies to the CROTA2 WCS keyword
which should be assumed to be 0.0 if not present in the EHU.  I think
the cleanest solution to this class of problems would be to restrict
the inherit convention to only apply in cases where the PHU has NAXIS =
0.  If NAXIS != 0, then the PHU keywords would not be inherited even if
the EHU has the INHERIT= T keyword.  If the PHU does have NAXIS = 0,
then it is safer to assume that any keywords may be inherited.

3.  Inheritance of inappropriate PHU keywords by a Table extension:

The same problems described above in (2) also apply to TABLE or
BINTABLE extensions, only more so. There are whole classes of PHU
keywords which would be illegal to inherit into tables such as the WCS
keywords, BSCALE, BZERO, BLANK, etc.  Any implementation of this scheme
would need to filter out these keywords before inheriting them to a
table extension.  However, this would not catch any user-defined
keywords which only apply to a primary array and are meaningless, or
confusing in the context of a table extension.  One could of course
restrict this proposal to only apply to IMAGE extensions, but this
would be inconsistant since the arguments in favor of this proposal
apply just as well to tables as they do to images.  As in case (2), I
think the simplest solution to these sorts of problems is to restrict
the inheritance convention to only apply if the PHU has NAXIS = 0.  By
doing this, then it is less likely that there would be keywords in
the PHU that would be forbidden in a table.

4.  Modification of inherited keywords:  

If a program opens a particular EHU and inherits some keywords from the
PHU, what should it do if it needs to modify the values of some of the
inherited keywords?  I think it would be too dangerous to directly
modify the value in the PHU because of the unknown effect this would
have on any other EHU which may inherit this same keyword.  So I think
the software should simply add this modified keyword value to the EHU,
thus overriding the value in the PHU.  If the software really needs to
modify (or delete) the PHU keyword value then it should be required to
physically open the PHU and make the modification directly.

5.  Are HISTORY, COMMENT and blank keywords inherited?  This is not a
major problem, but the proposal should state how these keywords should
be treated.  I would think these keywords should not be inherited, but
there may be cases where it would be desireable.

 Bill Pence
 HEASARC/NASA/GSFC
 pence at tetra.gsfc.nasa.gov

---

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry M. Schlesinger)
Date: 25 Aug 1995 15:51 EDT

In article <199508221622.MAA23921 at tetra.gsfc.nasa.gov>, William Pence <pence at tetra.gsfc.nasa.gov> writes...
>the analysis of FITS files using this convention.

The general issue raised here, that of which header keywords are 
inherited, is an important one, not just the question of the
particular solution suggested.  The proposal needs to define more
precisely which keywords in the header are inherited and which are
not. 

More generally, it is important in the context of this discussion to
remember that individual FITS HDUs may be copied from one file to
another.  That possibility is one reason why most FITS software treats 
extensions independently.  Although the rule stated in the Generalized
Extensions Paper states only that keywords in the primary header are
not permitted to announce the presence of a particular kind of
extension, in general, it is not advisable to have keywords in the
primary header that mandate the structure or content of the extensions
in the file, since there is no requirement to copy the extensions 
along with the primary header.  EXTEND = T, for example means only that extensions *may*
be present, not necessarily that they are.  The discussion in the
proposal about maintaining information in the extensions, 

>> as HST data products are concerned!)

also applies to the case where the extension is moved from one file to
another by software that does not recognize the inheritance
convention. 

Perhaps the best use of inheritance would be to express information 
that is relevant to a set of HDUs considered together but not of 
particular interest when only one is read.

				Barry Schlesinger
				FITS Support Office
				GSFC/ADF

---

From: RANDY THOMPSON (301) 286-8800 <RTHOMPSON at iuedac.gsfc.nasa.gov>
Date: Tue, 29 Aug 1995 11:54:02 -0400 (EDT)

I think the issue of inheritance is important, and the question 
of how to keep data sets connected with the data descriptions has become 
more involved with the addition of FITS extensions and FITS readers which 
separate extensions into separate files. The adoption of the Image
extension may have made separating FITS extensions into separate FITS files 
more feasible since the Image extension header format is so similiar to that
of a primary header. It needs to be emphasized however that the issue 
of inheritance is not specific to the Image extension but applies equally 
to all the existing FITS extensions. For this reason, I think it's important 
that new conventions do not change the interpretation of existing FITS 
files. Obviously LOCAL conventions should not impact other projects, but
IRAF users will undoubtedly want to read FITS files from other 
institutions.
    Our project adopted the policy of storing all but the required
extension header keywords in the primary header. In the most complicated 
file format, we use a primary HDU containing a 2-dimensional image, followed
by a Binary table extension, followed by two Image extensions. All the data
in the file pertains to one observation. It did not seem necessary to us 
to repeat all the observational and processing history information (roughly
400 keywords) in each of the extension headers. The FILENAME keyword
included in the extension headers provides (some) information about the 
individual data sets and does afford a way to keep data sets connected if 
they are written to separate files. 
    In retrospect, particularly in view of the discussions on inheritance, 
perhaps we should have considered a different format for the IUE archive 
in which more information is stored in the individual extension headers. 
Unfortunately, I think its too late for us to change, and because FITS
files like ours exist, it may not be valid to make assumptions about 
how primary header information should or should not be inherited by the 
extensions when newly-defined keywords (such as INHERIT = T) are not 
present.

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Wed, 30 Aug 1995 09:01:45 +0200 (MET DST)

On 25 Aug 1995, Barry M. Schlesinger wrote:

> inherited, is an important one, ....
  [...]
> Extensions Paper states ...

Uh, my apologies to everybody, but I remain constantly puzzled by the
emphasis on the ACTUAL USAGE of multi-extension files. It is clear to me 
why the "general extension" mechanism was invented that way (to allow the 
basic "image" files to coexist with new extensions), but it appears it 
did not have a widespread usage until binary tables came out. 

And even then I cannot understand why the majority of the cases (95 % ? 
or more ?) cannot be handled either :

  - as independent "image" files (i.e. primary HDU only)

  - as independent "single table" files (i.e. a primary HDU without data
    array, and ONLY another ONE HDU with a BINTABLE)

I do not understand what are the reasons for having more tables in the 
same file (and not separate files). There is no significant gain in space 
(an argument which might perhaps be relevant for a DATA EXCHANGE format), 
and there is complication in the reading/analysis software.
This is particularly true if the tables collated together are of 
miscellaneous nature and format.

I could PERHAPS find some sense for IMAGE extensions (the 5% or less 
missing above) where all extensions are of the same and simple nature and 
layout, and only in the case the extensions are of "ancillary" nature 
w.r.t. the primary data array (like the case of the "epsilons" for IUE).

---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
----------------------------------------------------------------------------