Subject: Reposting of Hierarchical Grouping Proposal

From: jennings at tcumsh.gsfc.nasa.gov (Don Jennings)
Date: 06 Jan 1995 16:19:05 GMT

This is a reposting of the FITS hierarchical grouping proposal. We have had
several good comments/suggestions sent to us by email that I will summarize
and post at a later time (once I get permission from the authors). This
proposal is also on the agenda for the up comming WGAS forum at the AAS
meeting in Tucson.

Latex and postscript versions of this proposal are available via anonymous
ftp at ssvs.gsfc.nasa.gov in the /pub/convert directory. It is also online
at http://acadia.gsfc.nasa.gov/convert/group.html

             Don Jennings, jennings at tcumsh.gsfc.nasa.gov

------------------------------------------------------------------------------

            A Hierarchical Grouping Convention for FITS

                  Donald G. Jennings, GSFC/HSTX 
                  William D. Pence, GSFC 
                  Michael Folk, NCSA 
                  Barry M. Schlesinger, GSFC/HSTX

                  Proposal Draft, Revision 4
                     December 16, 1994 

                          Abstract
              
This paper describes a grouping convention for FITS that facilitates the 
construction of hierarchical associations of Header Data Units (HDUs). The 
grouping convention uses FITS table structures (ASCII or binary) to 
encapsulate pertinent information about the HDUs belonging to a group. Group 
members may reside in a single FITS file or be distributed in many FITS files; 
the FITS files themselves may reside on different computer systems.

                      1. Introduction  

The rules for generalized extensions in FITS (Grosbol et. al., 1988) provide 
for FITS formatted files containing more than one header data unit. By 
using combinations of ASCII tables (Harten et. al., 1988), binary tables 
(Cotton et. al., 1994) and image extensions (Ponz et. al., 1994) related data 
sets requiring different data structures may be stored in the same FITS file, 
each within its own HDU. Unfortunately, once the related data sets are 
segregated into separate HDUs the relationship between them is often lost.

The FITS standard currently allows for simple hierarchical associations of 
HDUs within a single FITS file through use of the EXTLEVEL keyword. 
However, this mechanism has several major limitations. First, 
its use is not well defined. Different organizations may use EXTLEVEL for 
widely varying purposes and still not violate the FITS standard. 
Secondly, it does not specify a mechanism for defining distinct multiple 
groups of HDUs within a FITS file. Lastly, it cannot be used to 
associate HDUs residing in different FITS files. Except for very simple 
cases, FITS contains no mechanism for creating or preserving associations 
between HDUs or groups of HDUs.

As the volume and complexity of FITS formatted data grows, the need for a 
recognized and versatile HDU grouping mechanism increases. Individuals can 
be overwhelmed trying to manage and analyze large data sets unless those 
sets are logically organized. Software tools also require data organization
in order to access all necessary components of an observation, simulation
or experimental data set.

As an example of where grouping capabilities within FITS would be useful,
consider the following. It is desirable to combine a set of observations
from a given time period into a single FITS file for transport and
archival purposes. For each observation there is an observation log,
an event list, a derived image and a set of instrument calibration data;
furthermore, several observations share a common set of calibration data.
By using a grouping mechanism each [log, event list, image, calibration] set 
could be logically tagged as an associated observation group and the 
calibration data could be made a part of many different observation groups, 
thus eliminating the need to store it more than once. Software could retrieve 
all the information about a given observation simply by extracting those 
HDUs defined in the table that identifies members of the group.  
Also, observations of the same object from different observational periods 
could be combined into a group and accessed as a unit, even though the HDU 
sets comprising the different observations reside in separate FITS files. 

The following sections describe a scheme for implementing a hierarchical 
grouping of header data units within single and multiple FITS files. Section 
2 discusses the content of table extensions used to define HDU groupings. 
Section 3 lists those keywords recommended for headers of group member 
extensions. Finally, Section 4 provides sample headers from FITS table 
extensions containing grouping structures.

                   2. Group Tables  

A group table, as defined in this convention, is a FITS table extension that 
contains a list of all the associated member HDUs in the group.  Group tables 
may be represented by either FITS ASCII tables XTENSION= 'TABLE   ' or binary 
tables XTENSION= 'BINTABLE', and are uniquely distinguished from other types 
of FITS tables by having the EXTNAME = 'GROUPING' keyword and value in the 
header. The other required or recommended keywords and columns in a group 
table are described in the following sections.

There may be zero, one, or more group tables within a given FITS file.
Each group table may reference any number of HDUs. The entire set of
HDUs referenced in a group table, along with the group table itself,
form a group . Individual HDUs referenced in a group table are
said to be members of the group or group members.

Groups can contain any type and mix of HDU. This includes all of the
IAU-endorsed extensions as well as other extensions that conform to the
requirements for generalized FITS extensions.  Note that a group may
also contain other groups as members, since a group table is itself a
FITS extension.  This feature allows for the construction of hierarchical 
structures of HDUs within a single FITS file or across many FITS files.

         2.1 Group Member Identification Methods 

Group tables specify the names and locations of FITS files containing
member HDUs as well as identifying members within their FITS files.
The name and location of each FITS file is specified by using the World-Wide 
Web (Berners-Lee, 1994) Uniform Resource Identifiers, or URIs.  All current 
and future forms of URIs, such as Uniform Resource Locators (URL) and the 
proposed Uniform Resource Names (URN), shall constitute valid names, although
the group table must specify the type of URI being used. If the group member 
resides in a different FITS file but on the same computer system then partial 
URIs (specifically partial URLs) may be used instead of absolute URIs to 
specify the member's file location.  If the group member resides in the same 
FITS file as the group table itself, then the URI field may be left blank.

The location of member HDUs within FITS files may be specified in
two different ways, either by reference or by absolute position.  The 
reference identification method uses the values of the XTENSION, EXTNAME 
and EXTVER keywords to uniquely identify the member HDU within the FITS file.
The position method uses the HDU order number to identify members, with the 
primary array having order value 0, the first extension order value 1, and so 
on.  Users may choose either or both identification methods when constructing 
a group table.  

While the reference method is not invalidated by a reordering of HDU positions 
within FITS files, it does require that each member HDU have a unique set of 
(non-FITS-required) keyword values, Thus, this method may present problems 
for FITS files whose headers cannot be easily modified, such as FITS files on 
read-only media. The position identification method provides for quick 
``random'' access to the member HDUs, since software does not have to sort 
though each extension looking for the correct set of keyword values, but will 
be affected if the order of member HDUs within their FITS files is changed
(please note: there is nothing within the current FITS standard governing
how or when HDUs may be reordered within their files).

                2.2 Group Table Keywords  
 
In addition to the standard required FITS table extension keywords, the
following keywords are required in the header of a group table:

EXTNAME (character): 
This value of the FITS reserved keyword uniquely identifies that this
FITS extension contains a group table. For group tables EXTNAME must have
the value `GROUPING'.

EXTVER (positive integer):
The value of this FITS reserved keyword serves as a group ID number
that uniquely distinguishes this group from any other groups that may
be defined in the same FITS file.  This group number may also be used
in the header of each group member to identify the group(s) to which the
member belongs (see section 3, GRPIDn keyword).

The following keyword is strongly recommended for inclusion in the
header of each group table:

GRPNAME (character):
This keyword contains the name associated with the group table. GRPNAME
values are case-insensitive and should only contain letters, digits, and
the underscore character (and not contain any embedded blank (ASCII 32)
characters).

                2.3 Group Table Columns 

The number of columns required in a group table depends on which method is 
used to identify the members (and recall that both methods may be used within 
the same group).  If the members are identified by reference then the 
following columns are required:

TTYPEn  = 'MEMBER_XTENSION' -- character field:
Contains the value of the XTENSION keyword from the group member's header.
In the case of primary HDUs where there is no required XTENSION keyword,
the value of `PRIMARY' will be used instead. Therefore, the current valid 
entries for this column are 'PRIMARY ', 'TABLE   '+, 'BINTABLE', 'IMAGE   ' 
or any other IAU  FITS Working Group registered XTENSION value. This field 
may contain the FITS null value appropriate for this column type if the value 
is unknown (e.g., if the position identification method described below is 
used to identify the member location).

TTYPEn  = 'MEMBER_NAME' -- character field:
Contains the value of the EXTNAME keyword from the group member's header. 
In the case of primary HDUs where the EXTNAME keyword is not defined or when
the member extension has no EXTNAME keyword present, this field may contain
the FITS null value appropriate for the column type. 

TTYPEn  = 'MEMBER_VERSION' -- integer field:
Contains the value of the EXTVER keyword from the group member's header.  In
the case of primary HDUs, or if the EXTVER keyword is not present in the 
member header then a value of 1 should be assumed.

If members are identified by file position then the following column is 
required:

TTYPEn  = 'MEMBER_POSITION' -- integer field:
Contains a group member's position within its FITS file. The file's primary 
header is given a position value of 0, the first extension is given a 
position value of 1, and so on. If for some reason a group member's 
`MEMBER_POSITION' value becomes invalid or undefined, then this column
field should be filled with the FITS null value appropriate for the column
format.

If some or all of the group members reside in FITS files separate from the
group table itself then the following two columns are also required:

TTYPEn  = 'MEMBER_LOCATION' -- character field: 
Contains the location of the group member's FITS file using Uniform
Resource Identifiers. If the FITS file resides on the same computer
system as the group table, then partial URIs may be used instead of
absolute URIs, and if the group member resides in the same FITS file as
the group table then the field may be filled with blanks.  If the
MEMBER_LOCATION value becomes unknown or invalid then this field may
be filled with the FITS null value appropriate for the column type.

TTYPEn  = 'MEMBER_URI_TYPE' -- character field:
Contains the mne-monic for the Uniform Resource Identifier type used in the 
corresponding MEMBER_LOCATION field. Recommended values for this column field 
are `URL' for the Uniform Resource Locator and `URN' for the Uniform Resource 
Name. As other URI types are defined their mnemonics will also become 
acceptable values for this field. In cases where the MEMBER_URI_TYPE is
undefined (such as a null or blank MEMBER_LOCATION field value) this field 
may contain the FITS null value appropriate for the column type.

Besides the table columns defined above, a group table may contain any number 
of user defined columns. Group table columns may appear in any order within 
the table and their TTYPEn values are not to be considered case-sensitive. 

           3. Keywords for Group Member Extensions 

No additional keywords are required for HDUs that are members of a group.
This rule is to ensure that all currently existing FITS files and their 
constituent HDUs may all be part of this convention. There are, however, 
several grouping related keywords whose presence is strongly recommended in 
newly created headers. The description of these keywords follow. 

EXTNAME (character): 
This keyword is the FITS reserved keyword EXTNAME. Extensions that are part 
of groups should use this keyword to allow member identification by reference.

EXTVER (integer):
This keyword is the FITS reserved keyword EXTVER. Extensions that are part 
of groups should use this keyword in their headers to allow member
identification by reference. 

GRPIDn (integer): 
A series of indexed keywords that denote the group(s) to which an HDU
belongs. The value of GRPIDn is the EXTVER value of the nth group table
that the HDU is a member of. In this sense, the EXTVER value of a group table 
defines a unique ID for the group within a FITS file. If the value of 
GRPIDn is negative, then the HDU is a member of a group defined in another 
file. In this case the absolute value of GRPIDn is the EXTVER value of the 
external group table, and the corresponding GRPLCn keyword holds the 
URI of the FITS file containing the group's table. The GRPIDn keywords (and 
their associated GRPLCn keywords) not only identify HDUs as members of groups,
 but also allow group members to ``point'' back to their group tables. Any 
software that might change the position or nature of the HDU would know that 
it was a member of a group and that the group table would require updating.

GRPLCn (character):
A series of indexed keywords that contain the  
Uniform Resource Identifiers  corresponding to the GRPIDn keyword. The
GRPLCn values follow the same syntax rules as those specified for the
group table's MEMBER_LOCATION column (see section 2). It is unnecessary to 
have a GRPLCn keyword accompany a GRPIDn keyword when the value of the GRPIDn 
keyword is positive.

              4. Example Group Table Headers 

The following are examples of valid group table headers that use different 
combinations of identification methods.

Example 1: A group containing five members all of which reside in the same
file as the group table. This group is itself a member of two other groups 
and both of those groups' tables reside in the same file as this extension. 
The member position identification method is used to locate member HDUs.

XTENSION= 'BINTABLE'        / This is a binary table
BITPIX  =                 8 / Table contains 8-bit bytes
NAXIS   =                 2 / Number of axis 
NAXIS1  =                 4 / Width of table in bytes                
NAXIS2  =                 5 / Number of member entries
GCOUNT  =                 1 / Mandatory FITS keyword
PCOUNT  =                 0 / Number of bytes in HEAP area
TFIELDS =                 1 / Number of columns in table        
EXTNAME = 'GROUPING'        / This BINTABLE contains a group
EXTVER  =                 3 / The ID number of this group
GRPID1  =                 1 / Part of group 1
GRPID2  =                 2 / Part of group 2
TTYPE1  = 'MEMBER_POSITION' / Position of member within file
TFORM1  = '1J'              / Datatype descriptor
END

Example 2: A group containing 150 members, some of which reside in FITS
files different from that of the group table. This group is not a member of
any other group, although it is the seventh group table defined in the FITS 
file. All member identification methods are used.

XTENSION= 'BINTABLE'        / This is a binary table
BITPIX  =                 8 / Table contains 8-bit bytes
NAXIS   =                 2 / Number of axis 
NAXIS1  =                76 / Width of table in bytes                
NAXIS2  =               150 / Number of member entries
GCOUNT  =                 1 / Mandatory FITS keyword
PCOUNT  =                 0 / Number of bytes in HEAP area
TFIELDS =                 6 / Number of columns in table        
EXTNAME = 'GROUPING'        / This BINTABLE contains a group
EXTVER  =                 7 / The ID number of this group
TTYPE1  = 'MEMBER_LOCATION' / URI of file containing member HDU
TFORM1  = '30A     '        / Datatype descriptor
TTYPE2  = 'MEMBER_URI_TYPE'  / URI type of MEMBER_LOCATION field
TFORM2  = '3A      '         / Datatype descriptor
TTYPE3  = 'MEMBER_POSITION' / Position of member within file
TFORM3  = '1J      '        / Datatype descriptor
TTYPE4  = 'MEMBER_XTENSION' / XTENSION keyword value of member
TFORM4  = '8A      '        / Datatype descriptor
TTYPE5  = 'MEMBER_NAME'     / EXTNAME keyword value of member
TFORM5  = '30A     '        / Datatype descriptor
TTYPE6  = 'MEMBER_VERSION'  / EXTVER keyword value of member
TFORM6  = '1J      '        / Datatype descriptor
END

Example 3: A group containing 17 members, some of which reside in FITS
files different from that of the group table. This group is a member of
six other groups, two of which are defined in FITS files on other computer
systems and one that is defined in a FITS file on the same computer system. The
member reference identification and member file location methods are used. Two 
user defined columns are also present.

XTENSION= 'BINTABLE'         / This is a binary table
BITPIX  =                  8 / Table contains 8-bit bytes
NAXIS   =                  2 / Number of axis 
NAXIS1  =                180 / Width of table in bytes                
NAXIS2  =                 17 / Number of member entries
GCOUNT  =                  1 / Mandatory FITS keyword
PCOUNT  =                  0 / Number of bytes in HEAP area
TFIELDS =                  7 / Number of columns in table        
EXTNAME = 'GROUPING'         / This BINTABLE contains a group
EXTVER  =                  7 / The ID number of this group
GRPID1  =                  3 / Member of group 3 
GRPID2  =                  6 / Member of group 6 
GRPID3  =                 18 / Member of group 18
GRPID4  =                 -1 / Member of external group 1  
GRPLC4  = 'http://fits.gsfc.nasa.gov/FITS/file1.fits' / location of
COMMENT                         FITS file containing group 
GRPID5  =                 -5 / Member of external group 5  
GRPLC5  = '/FITS/file5.fits' / Location of file containing group 
GRPID6  =                 -2 / Member of external group 2  
GRPLC6  = 'http://www.noao.edu/irafdir/file2.fits' / location of
COMMENT                         FITS file containing group 
TTYPE1  = 'USER_INFO_1'      / A user supplied column
TFORM1  = '25J     '         / Datatype descriptor
TTYPE2  = 'MEMBER_LOCATION'  / URI of file containing member HDU
TFORM2  = '30A     '         / Datatype descriptor
TTYPE3  = 'MEMBER_XTENSION'  / XTENSION keyword value of member
TFORM3  = '8A      '         / Datatype descriptor 
TTYPE4  = 'MEMBER_NAME'      / EXTNAME keyword value of member
TFORM4  = '30A     '         / Datatype descriptor
TTYPE5  = 'USER_INFO_2'      / A user supplied column
TFORM5  = '5A      '         / Datatype descriptor 
TTYPE6  = 'MEMBER_VERSION'   / EXTVER keyword value of member
TFORM6  = '1J      '         / Datatype descriptor
TTYPE7  = 'MEMBER_URI_TYPE'  / URI type of MEMBER_LOCATION field
TFORM7  = '3A      '         / Datatype descriptor
END

Example 4: A group containing 82 members, some of which reside in FITS
files different from that of the group table. This group is a member of
three other groups, and makes use of the member position and member file 
location methods. One user defined column is present. Note that in this 
example an ASCII table (as opposed to a binary table) is used to define the 
group.

XTENSION= 'TABLE   '         / This is an ASCII table
BITPIX  =                  8 / Table contains 8-bit ASCII characters
NAXIS   =                  2 / Number of axis 
NAXIS1  =                 46 / Width of table in bytes                
NAXIS2  =                 82 / Number of member entries
GCOUNT  =                  1 / Mandatory FITS keyword
PCOUNT  =                  0 / Mandatory FITS keyword
TFIELDS =                  4 / Number of columns in table        
EXTNAME = 'GROUPING'         / This TABLE contains a group
EXTVER  =                 31 / The ID number of this group
GRPID1  =                  3 / Member of group 3 
GRPID2  =                  9 / Member of group 9 
GRPID3  =                 27 / Member of group 27
TTYPE1  = 'USER_INFO_1'      / A user supplied column
TFORM1  = 'E10.3   '         / Datatype descriptor
TBCOL1  =                  1 / Starting table column for field
TTYPE2  = 'MEMBER_LOCATION'  / URI of file containing member HDU
TFORM2  = 'A30     '         / Datatype descriptor
TBCOL2  =                 11 / Starting table column for field
TTYPE3  = 'MEMBER_URI_TYPE'  / URI type of MEMBER_LOCATION field
TFORM3  = 'A3      '         / Datatype descriptor
TBCOL3  =                 41 / Starting table column for field
TTYPE4  = 'MEMBER_POSITION'  / XTENSION keyword value of member
TFORM4  = 'I3      '         / Datatype descriptor 
TBCOL4  =                 44 / Starting table column for field
END

                  5. Acknowledgments 

We gratefully acknowledge the support of the NASA Applied Information
Systems  Research Program, underwhich this effort is funded.

                    6. References 

Berners-Lee, Tim, 1994. ``World Wide Web Initiative'', CERN - European 
Particle Physics Lab. http://info.cern.ch/hypertext/WWW /TheProject.html .

Cotton, W. D., Tody, D. and Pence W., 1994. ``Binary Table Extension to FITS:
A Proposal'', version dated June 13, 1994, final form. NRAO FITS Archives. 
http://fits.cv.nrao.edu/fits/documents/standards/bintable3.ps .

Grosbol, P., Harten, R. H., Greisen, E. W., and Wells, D. C., 1988. 
``Generalized extensions and blocking factors for FITS,'' Astronomy and 
Astrophysics Suppl., 73, 359-364. 

Harten, R. H., Grosbol. P., Greisen, E. W., and Wells, D. C., 1988. ``The 
FITS tables extension'', Astronomy and Astrophysics Suppl., 73, 365-372. 

Ponz, J. D., Thompson, R. W., and Munoz, J. R., 1994. ``FITS Image Extension''
, Astronomy and Astrophysics Suppl., vol 105, pp 53-55.