Subject: HK telemetry formats

From: pence@tetra (William Pence)
Date: Wed, 12 May 1993 16:42:20 -0400

This is in reply to a couple of points raised by Lucio Chiappetti in his
message dated 27 April 93 concerning the formats for science telemetry
data and HK telemetry data.  Lucio argued that these data tend to be
highly mission specific, therefore they should not be the subject of
discussion for standardization between different missions.  However,
from the experience we have had at the HEASARC in dealing with ROSAT,
ASCA, and (in the near future) XTE data, we would argue that there are
significant advantages in trying to develop standard FITS formats for
telemetry data.

These three different missions (ROSAT, ASCA, XTE) all have somewhat
different requirements, and hence the resulting FITS files have
slightly different formats.  But the data files from all three missions
share many common characteristics (most importantly they all use the
BINTABLE format) which makes it easier to write multimission software
which can deal with any of these data files.

Below is a brief description of the FITS format for the HK telemetry
files for each of the missions.  Hopefully other future missions will
find that one (or more) of these formats  meets their requirements.

ROSAT HK Files

Planning for the ROSAT mission began before the HEASARC was formed,
therefore we were not involved in the early data format decisions.  
It was decided early on that only a small set of HK parameters was of
interest to the observer and this decision is being  maintained in the
new 'rationalized' FITS  formats that are now being implemented.  The
values of these HK parameters are written to a FITS binary table
extension with a separate column for each parameter.  A new row of
values is written to the table for each cycle of telemetry readouts
from the spacecraft.  Besides the set of parameter value columns, there
is also a 'TIME' column in the table which records the time of all the
other measurements on that row of the table.  (The format of this TIME
column is another big issue which should be discussed separately).  In
principle, if the values of all the parameters remained the same from
one row to the next, the latter row could be deleted from the table
without lose of information to reduce the size of the table, but this
has not been done for ROSAT.

ASCA HK Files

During the early design of the ASCA FITS data formats it was decided
that virtually all the HK parameters should be saved and given to the
observer in the HK FITS file.  Because there are about 600 different
ASCA HK parameters it was deemed not practical to use the ROSAT format
where each parameter is written to a separate column of the table.
Instead the HK data are written to a much smaller binary table which
contains only three columns:  TIME, HK_NAME, and HK_VALUE.  A new table
is written for each observation.  The initial value of each HK
parameter is written at the beginnig of each table (which requires
about 600 rows to list all the parameters), and then whenever the value
of a parameter changes from its previous state, the new value, with
its time stamp, is written to the table.  Since many of the HK
parameters do not change value very frequently,  these files are only a
fraction as large as if every parameter is written into a separate
column.

While this format has the advantage that is is quite compact, it is
often more convenient for software to read the HK data if it is in a
ROSAT-style format table.  Therefore, we have written software which
will convert FITS files between these two formats (usually after
selecting just a small subset of the total number of HK parameters to
be converted).  There is also software available which will convert the
raw telemetry value of each parameter into its true engineering units
(e.g. Volts, or Degrees C).  In general this conversion can only be
applied to the ROSAT-style format files which allows the the datatype
of each parameter to vary (e.g., integer or floating point).  In the
more compressed format HK files the HK_VALUE column is always a 4-byte
integer.

XTE

The design of the formats of the XTE FITS files has just started, but
one of the primary requirements is that we preserve as closely as
possible the original telemetry format of the HK data (which has a
packet format similar to SAX) with a minimum of reformating.  This a
partially driven by the desire to minimize the chances that data may be
corrupted during this reformating process.  Thus the ASCA format, which
involves a considerable amount of reformating of the telemetry stream,
is not suitable for XTE.  One of the characteristics of this data is
that there is a regular cycle of HK measurements in the telemetry which
gets repeated at a well defined set interval (which I'll call a 'major
cycle').  Some HK parameters get readout more frequently than others
during a major cycle, so that while some parameters only occur once per
major cycle, other parameters may be readout more frequently and appear
multiple times in each major cycle.

A fairly natural way to store this HK telemetry data in a binary FITS
file is to represent each major cycle as a single row in the table,
with a separate column for each HK parameter.  Those parameters that
occur more than once in each major cycle are stored in vector columns
in the binary table.  Very little processing needs to be done to
convert the HK telemetry into this FITS format; it mainly consists of
padding out the telemetry values (e.g., if the telemetry value has 1 to
8 bits, it is written as an 8-bit '1B' column in the FITS table).
Note that the ROSAT format is a special case of the more general
XTE format in which none of  the columns contain vectors.

Summary

>From the example of these three missions, it can be seen that
is fairly easy to preserve the HK telemetry data in FITS format files
thus making it available to all observers in a standard, relatively
easy to understand format.  The HEASARC is developing software which
can process these HK FITS files in various useful ways.  For instance,
we have software which will take a HK telemetry FITS file, along with a
set of criteria on the values of various HK parameters, and will then
output a list of time intervals during which the HK parameters
satisfied those criteria (i.e., produces a list of 'Good Time
Intervals').  This list of time intervals, which is also in a BINTABLE
FITS format, can then be feed into another program which will select
all the events from a Science FITS file which occur within one of those
time intervals.

In summary then, we would recommend that other missions consider
adopting a FITS format similar to these for their HK telemetry data
files.  I can supply examples of these files, (at least for ROSAT and
ASCA) if anyone wants to see more details.

-Bill Pence
 HEASARC