Subject: Re: OGIP FITS conventions (ROSAT et al.)

From: Lucio Chiappetti <[email-masked]>
Date: Tue, 27 Apr 93 08:43:12 -0400

As promised a while ago, I would address to HEAFITS a "first order" question
about what, in my opinion, may and what may not be discussed over this
exploder in the aim of converging about some multi-mission standard.
For this I take as reference Mike Corcoran's mail, of which I will quote
a few excerpts below, in order to make a common reference.

On Thu, 20 Apr 1993 [email-masked] wrote:
(forwarding to HEAFITS a message originally sent to me, of which I keep
only hilights in the text below)
> 
> .....respond to your questions.  The OGIP has begun development of a
> "standard" set of file formats for distribution and archiving of data
> from high energy astrophysics missions.  

  The fact OGIP is concerned on one hand ALSO with archiving of PAST missions,
  and on the other hand it is acting like a "filter" between a mission in
  which had no direct involvement at the level of defining the telemetry
  format or original data distribution format being this somehow ruled by
  other national agencies (e.g. ROSAT, ASTRO-D) puts it into a different
  framework than the one I'm operating in.

  I will make my points below based on the cases of SAX (a mission of the
  Italian Space Agency, with collaboration in the Netherlands and ESTEC),
  and XMM (a mission of the European Space Agency). In both cases the
  primary responsibility of data dissemination stays with the relevant
  agency (or its contractors). However there is a role of the scientists
  in giving as strict requirements as we can about how we want this to
  be done.

> These formats are called
> "rationalized data file" (RDF) formats.  The basic specifications for
> these files are
>      1) the files follow the standard FITS conventions .....

   *** see comment  # 5 below **

>      2) under the rationalized scheme, the data is grouped into 4 categories:
>                 a) BASIC data - this is the bare-bones science data,
> like good time intervals, photon event lists, and statuses.  

  I would call these "science telemetry data"  
  *** see comment # 1 below **

>                 b) ANCILLARY data - ..... engineering or housekeeping 
> data, and spacecraft pointing (aspect) information.

  I would call these "housekeeping telemetry & auxiliary data"
  *** see comment # 2 below **

>                 c) CALIBRATION data -.....
> response matrices, mirror effective areas, and point spread functions.

  for calibration data
  *** see comment # 3 below ***

>                 d) DERIVED data ..... This includes images,
> spectra, lightcurves, source lists, etc.

  I would call these "reduced data" or "data products"
  and put MOST EMPHASIS AT THIS LEVEL
  *** see comment # 4 below **

>         3) The OGIP has developed keywords and FITS table column names...

  *** see comment # 5 below **

---------------------------------------------------------------------------

In the following I'll make references to the case of SAX, which is in a
stage of advanced definition, and XMM, for which I hope to be able to
negotiate with ESA in a similar framework.

# 1)  SCIENCE "TELEMETRY" DATA

Both SAX and XMM will use for their telemetry the ESA "Packet Telemetry"
standard (actually this is a CCSDS standard, being the CCSDS a nice
acronym which looks the same both in English and French - Consultative
Committee for Space Data Systems or Comite' Consultatif pour Systemes
de Donnees Spatiaux ... apologies about my bad French - that is some
inter-agency-wide agreement). I do not know whether other space agencies
are currently recommending/enforcing it for their missions.

Being it a "packet" standard means in practice that each science instrument,
operating in a variety of modes (i.e. direct modes producing photon lists,
indirect spectral modes, indirect timing modes) produces one or more
data streams with a defined layout. The standard itself is concerned
about the way this user defined layout is encapsulated for transmission
and not about forcing rules for this layout. I won't go here into any
technical detail.
Those of you being familiar with Exosat data may recognise that Exosat
"floating telemetry" was a sort of rough, unstandardised precursor of
the Packet Telemetry.

Actually the Packet Telemetry applies both to science and HK telemetry.
The basic facts here are :

   each instrument may have a large number of operating modes (i.e.
   different packet types), and not all of them are "direct" modes
   (i.e. there are spectral and timing modes). This is as far as I can
   say unlike missions having mainly only a direct (photon list) mode.

   for spacecraft and ground segment constraints one is often forced
   to optimize the telemetry bit rate using "unnatural" arrangements
   (i.e. having 10-bits for position, or 9 for energy, or packing events in
   some floating format with time every 6 photons only, etc.)

   These data cannot be reasonably given directly to observers, specially
   if one wants to give them some truly portable software. Furthermore 
   one hand there are reasons to archive the telemetry as raw as
   possible (i.e. just sorted by packet type and observation BUT
   PRESERVING the packet arrangement : 1 packet = 1 record), on another
   hand whatever reformatting is done here must be simple, so that data
   can be reprocessed in the event of any mistake or update.
   [Finally since these things are to be done by a contractor, and contract
   requirement documents are unflexible, it is not possible to use standards
   not FULLY defined at the moment the requirements are written]

The idea being pursued for SAX is to store packets of the same type in
separate files for each observation (which is a sub-interval of a
pointing interval where all instrument commandable parameters are kept
constant), applying just a reformatting like "padding all quantities
which are 1-8 bits to 8 bits, 9-16 bits to 16 bits, 17-32 bits to 32 bits"
and "normalizing spacecraft time to a common resolution".
[again those of you familiar with Exosat will recognise the "observation"
and "observing period" distinction - I presume, given the similarity of
operations, this might apply to XMM too]

This sort of reformatted data, although not strictly telemetry, I'll
term "science telemetry data". With it go some auxiliary files like
"observation directories", "instrument configuration files" etc. which
specify when and how data can be combined for analysis.

Given the numerous mission specific issues at this level, I would personally 
NOT regard this kind of data as a subject of discussion (e.g. over HEAFITS)
for standardization.

(I will however incidentally mention here that we are thinking of having
the "accumulation" programs, i.e. those which transform "science telemetry
data" into "reduced data", driven by packet description kept in external
files. We call such files "packetcap" for analogy with Unix termcap,
printcap, graphcap etc.  In such files one has for instance the NAMES
of the fields present for each event. Being these names used later as
column names in reduced data, THERE IS AN OBVIOUS NEED OF A STANDARD
CONVENTION ON SUCH NAMES, and this CAN - MUST ? - BE MATTER FOR HEAFITS)

# 2) HK TELEMETRY AND AUXILIARY DATA

Payload and spacecraft housekeeping is handled exactly as science
telemetry (at least whenever it is possible it be of any use for some
observers - other s/c HK data are not even passed to the observer).
In this case description of the content (so called Parameter Characteristics
Files) are to be supplied by the Ground Segment contractor.

Auxiliary data concern orbit, attitude, OBT to UTC conversion and
alike, and are likewise supplied by the Ground Segment contractor, in
ASCII form, which is sufficiently portable and self documented.

Therefore these data too are NOT of concern to HEAFITS, in my opinion.

# 3) CALIBRATION DATA

I would consider here different kind of data :

- the calibration observation (telemetry) data which are of no concern
  here (they are just like other observation data, only they are analysed
  by a particular PI or Agency team). Likewise are of no concern whatever
  intermediate files are produced during such analysis. But only the
  result of this analysis, i.e. ...

- ...the calibration result files, i.e. files describing individual
  components (e.g. efficiency vs energy, optics area vs position, psf,
  gain, energy resolution, etc.). These are building blocks to be
  given to the observer. The general recommendation is that these data
  are not distributed along with the "telemetry" tapes, but made
  available for ftp or network copy (so that the observer can grab the
  latest update).
  I am unsure whether a standardization at this level is desirable or
  possible, but I will not a priori exclude related matters are 
  discussed over HEAFITS

- the ultimate files used for the analysis, like response matrices.
  These must be produced by the observer using supplied software, starting
  from the "building blocks". Such data in my view must be handled at
  all in the same way as "reduced data" (see below).

# 4) REDUCED DATA

Reduced data is what the observer produces using the accumulation programs
(images, spectra, time profiles, photon lists AND response matrices). He can
then manipulate them for background subtraction, dead time correction etc.
and ultimately feed them in into one's favourite analysis package.

One requirement at this stage is usage of a mission independent format (the
only mission dependent stuff is storing reference information about
instrument configuration somewhere in header keywords). 

The other basic requirement in my view is an economy principle, i.e.
Ockham's Razor :
FILE STRUCTURES NON SUNT MULTIPLICANDA PRAETER NECESSITATEM (are not to
be multiplied beyond need).

And also FILE LAYOUTS non sunt multiplicanda praeter necessitatem ! 
And hopefully they should be "natural looking".
All this does not (yet necessarily) mean FITS (see comment #5 below), but
means a preliminary agreement on WHAT ARE these basic file structures, and
immediately after about their basic content. I'd like users of all various
X-ray analysis packages in the world (from the general purpose ones like
EXSAS, PROS or XANADU to any site or mission specific package) to comment
on what is their feeling on current compatibility and requirements towards
it (I would assume that one should not necessarily aim to "portability", i.e.
use only one format 100% compatible with all packages, or even worse, all use
a single package, but instead to "interoperability", i.e. import a file from
one package, convert it hopefully with a SIMPLE and SINGLE operation to the
other package format, use it in the second package, and so on)

Concerning the list of basic data structures I presume (at least in X-ray
astronomy ... I'm not sure how gamma-ray people feel about this) one can
limit it to :

   - images
   - spectra
   - time profiles
   - photon lists
   - response matrices
   - plus perhaps minor ancillary files

By images I mean not only the preponderant case of photon counts vs spatial
x,y, but also any other z=f(x,y) case, like photon counts vs energy and
rise time, or chi-square vs alpha and NH, or even a Fourier transform vs
frequency and energy (the latter case are mainly for storage and display of
results).

By spectra I mean chiefly PHA spectra, but may also apply to any sort of
1-d histograms (Burst Length, Rise Time etc. ; but these can also be hanlded
as 1-d images)

By time profiles I mean any time profile, not just count rate vs time, but
also any HK parameter vs time.
 
Ancillary files are in my view things specific of some s/w package or program
(like time windows, channel grouping masks and alike) and best stored as
plain ASCII files, and anyhow outside the scope of the main discussion.

Concerning the list of basic layouts, I would group the above in :

   - "images"  (images, perhaps histograms, partly also response matrices)
   - "tables"  (spectra, time profiles, photon lists)

I would also as far as possible keep the above "physical" layout level
hidden to the user, i.e. the user says "add spectrum A + spectrum B" and
not "add column 1 of tables A and B, and quadratically add column 2 of
tables A and B for the errors".

I also note that while images are a simple univocally defined structure,
tables allow a very wide range of choices, and it is quite easy to go astray
from portability, interoperability or naturality. THIS IS WHERE CONVENTIONS
MUST BE DISCUSSED BOTH ON THE LAYOUT, THE NAMES OF COLUMNS, THE KEYWORDS
NEEDED FOR DESCRIPTION.

Also in my quest for simplicity, I'd like as far as possible that each
"logical" file (spectrum, image etc.) be a single disk file (or FITS extension
if FITS is used either for storage or transport).

I'd like to see the above things discussed, and of course I am ready to
contribute to this discussion, but I believe this mail is getting quite
long, and I'll better do that separately sometimes later.

# 5) FITS OR NON-FITS

I believe that discussion on the above topics can go on over HEAFITS
(despite of the name of the exploder) EVEN if a particular mission
decides to use a non-FITS format for its data.

I'd personally favour (following a long-known trend) to store data
in some native format, and use FITS for exchange only. However I have
designed such format ("XAS" format) in such a way to be one-to-one
mappable to FITS. The main differences are : data is kept in native
binary representation, header keywords are kept in binary form, header
keywords are located at the end of the file, in order to be able to
extend the header freely e.g. to record in HISTORY any in-place mani-
pulation to the file. Files are record-oriented with a "natural" record
length dictated case-by-case (instead of the FITS 2880 ...  however on
a non-record oriented i.e. Unix, IEEE-compatible, e.g. Sun, machine, the
data portion of a XAS file is IDENTICAL to the data part of a FITS HDU).
Prototype programs exist to produce and manipulate XAS files, to convert
them to FITS, and also to convert them between different operating 
systems without an intermediate FITS file

Given this, the basic requirement from the beginning was to keep a
one-to-one "mappability" to FITS. For this for instance I forced the
LENGTH of a kewyord NAME to 8 characters, or the LENGTH of a CHARACTER
keyword field to 68, etc.

Therefore I believe there is PLENTY of conventions which can be discussed
and agreed over HEAFITS, some of them being "first order" issues (like
the identification of "basic" data structures mentioned above) and other
being "second order" issues (like names of columns and keywords, as
mentioned in Mike Corcoran's mail) or even lower order issues (one of
these being the physical unit convention being recently discussed ...
again the wording "lower order" has no contemptuous meaning ... actually
the lowest the order, the easiest the agreement).

---

From: [email-masked] (William Pence)
Date: Wed, 28 Apr 93 16:08:04 -0400

I've been meaning to reply to some of the many issues raised in Lucio's last
message but have been short of time.  One of the problems is that there
are so many good issues raised here, that it is hard to know where to
start in replying to all of them.  As a suggestion for future posts,
it would make it much easier for me to comment on various issues if
the postings to this group are kept relatively short and focused on a
single issue.  If too many different things are brought up at once, I
get overloaded and may not be able to respond as well as I would like
to all the individual issues.  Also these issues will probably get
a fuller discussion from a wider range of people if the issues are
raised one by one over a reasonable period of time.

I hope to reply to some of the specific issues raised by Lucio over the
next few days...

-Bill Pence

---

From: Lucio Chiappetti <[email-masked]>
Date: Thu, 29 Apr 93 07:19:42 -0400

> it would make it much easier for me to comment on various issues if

  I do agree as general principle. I thought to condense most of the
  points in a single mail on one hand because maybe not all people would
  feel all points worth further discussion, so having the full list there
  may help the selection (this was just the start, if you want sort of
  an attempt to define a "charter" for HEAFITS).
  On the other hand I had also been quite busy in the previous days (and
  will be in the next ones, a common problem, isn't it ?) and I found
  easier to condense all in one.
  But for the future I strogly support Bill's recommendation.

                                             Lucio