Subject: OGIP Rationalized Data Files

From: [email-masked]
Date: Thu, 15 Apr 93 13:23:01 -0400

Below is a copy of the e-mail I previously sent to Lucio about the 
data file structures that the OGIP has developed (with input from CFA) 
for archiving/distribution of data from high energy astrophysics 
missions.  

Mike Corcoran

----------------------Begin Include Message----------------------------

From:	NDADS::CORCORAN     15-MAR-1993 10:52:21.73
To:	SMTP%"[email-masked]"
CC:	CORCORAN
Subj:	OGIP FITS conventions

Hello Lucio,

I received your e-mail concerning the OGIP FITS conventions and would
like to give you some background concerning the OGIP file formats and
respond to your questions.  The OGIP has begun development of a
"standard" set of file formats for distribution and archiving of data
from high energy astrophysics missions.  These formats are called
"rationalized data file" (RDF) formats.  The basic specifications for
these files are 
	1) the files follow the standard FITS conventions for images 
and extensions (due to this requirement the rationalized data files 
are sometimes called "rationalized FITS files")
	2) under the rationalized scheme, the data is grouped 
into 4 categories:
		a) BASIC data - this is the bare-bones science data, 
like good time intervals, photon event lists, and statuses.  In other 
words, BASIC data is the information a user would need to perform 
scientific analysis.
		b) ANCILLARY data - data which is not expressly needed 
for standard science analysis but which is in general useful.  This 
data includes engineering or housekeeping data, and spacecraft 
pointing (aspect) information.
		c) CALIBRATION data - data which should be available 
to the user to describe how the instrument is calibrated.  This 
information includes detector response matrices, mirror effective 
areas, and point spread functions.
		d) DERIVED data - data which is derived from standard 
processing from the BASIC science information.  This includes images, 
spectra, lightcurves, source lists, etc.
	3) The OGIP has developed keywords and FITS table column names 
for use in the rationalized files.  It is suggested that whenever 
possible projects should use defined keywords or table column names 
for description of similar quantities.  However in the interest of 
accomodating a wide variety of missions these lists are expandable.  
	4) File formats should agree with previously defined OGIP 
formats and should be backward-compatible 
	5) Files should be self-documenting as much as practical.  
	
ROSAT is the first active mission to adopt the RDF format.  Below I give 
you a rough outline of the ROSAT implementation of the RDF format:
	1) for each observation, a BASIC file consisting of a FITS 
files with a NULL primary header and 4 binary table extensions giving
		a) good time intervals
		b) time intervals when the instrument was on, 
		   regardless of the quality flag settings
		c) a photon events list
		d) status histories
	2) for each observation, an ANCILLARY file consisting of a 
FITS file with a NULL primary header and binary table extensions 
giving
		a) orbit information
		b) aspect information
		c) housekeeping information
		d) other instrument-specific information
	3) for each observation, a CALIBRATION file consisting of a 
FITS file with a NULL primary header and binary table extensions 
giving
		a) effective areas
		b) response matrices
		c) instrument maps
		d) bad pixel locations
		e) etc.
	4) for each observations, a set of DERIVED files consisting of 
FITS files with the following information
		a) FITS files with an energy selected image as the 
		   primary array and energy-selected source locations
		   as binary table extensions
		b) FITS files with energy-selected backgrounds as the 
		   primary array
		c) A FITS file giving the total-band exposure map as 
		   the primary array
		d) A FITS file giving the source list and spectra as 
		   binary table extensions
		e) A FITS file giving a history of the data processing
		f) etc.

The above is meant to give you some idea about how the data is
structured in the RDF's for ROSAT.  Individual considerations for
other missions may require that the necessary information be divided
up differently. For example, if a file grows too large it is ok to
split the information up into individual files.  One could envision
that the BASIC file consisting of the status history and the event
list may be too large, so it would be desireable to perhaps include
the events list in a separate file. 

I'll send you along a list of keywords we've created for ROSAT, along 
with the binary table column names.  If you want, I can send you 
header dumps for the files.  Test versions of the ROSAT RDF files 
are available via anonymous FTP from 
		legacy.gsfc.nasa.gov
The test versions of the PSPC files are located in 
		rosat/data/pspc/sample_files/ratfits
while those for the HRI are located in 
		rosat/data/hri/sample_files/ratfits
Please beware that these files are developmental ONLY, and as such 
have a number of known (and unknown) BUGS, but will give you the 
general idea of how ROSAT is using the RDF format.

In terms of your specific questions:

> - Are these OGIP conventions a "private" convention of GSFC, or it is
>   intended to propose them as a draft standard in the traditional 
>   FITS way ?

These files follow the FITS standard so there are no plans to propose
them as a draft standard in the traditional FITS way. 

>  - Is the related documentation restricted in access or usage ?
>    And if not is it available electronically ?
	
	The documentation is not restricted, and I can send it to you
if you so desire.  I'm developing a ROSAT RDF spec which I can mail to
you (at the moment it is not available electronically).  I can e-mail
you the basic specifications we're using to develop the files.  Or you
can ftp the files themselves from legacy. 

>  - How stable are those conventions ? I mean, are they frozen ? Are
>    they liable to change duirng development dramatically ? etc. etc.

	For ROSAT the conventions are pretty stable, since we plan to 
start distributing data in this format by the end of April.  The 
formats are meant to be flexible so as to be able to accomodate data 
from other missions, and some (hopefully minor) changes are foreseen.

>  - Are these conventions open for discussion in the context of future
>    missions in which European and Eastern countries may be involved
>    (e.g. SAX, Spektrum-RG, XMM etc.) ?

	Insofar as the impact on missions which have used older 
versions of the RDF formats is minimal, new changes can be added to 
the conventions so as to allow future missions to use the RDF.  Like 
all FITS files, changes should be backward compatible.

If you have any further questions please don't hesitate to contact me. 

Cheers,

Mike Corcoran
ROSAT Project
Code 668
Goddard Space Flight Ctr
Greenbelt, MD 20771 USA
(301) 286-5576