Subject: FITS plans for SOHO.

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Mon, 18 Nov 1991 20:07:00 GMT

The following LaTeX document (landscape orientation) shows some slides I used
recently to present some possibilities of using FITS for the data from the
upcoming Solar Heliospheric Observatory (SOHO).  We were discussing FITS from a
standpoint of not only using FITS as a data distribution standard, but also
possibly as a standard for sharing and using data between the different
instruments during operations.

Some concerns that people had about FITS were:

	* That FITS is wasteful of disk space.  Some of this was a
	  misconception of the FITS standard, which I managed to clear up.
	  Also, in making up my samples, I rejected any possibilities that
	  required padding arrays.

	* That FITS is better tuned for writing files to tape than for using
	  files from disk.  The driver for this seemed to be VMS.  I discussed
	  the possibility of storing files on disk in an "SDAS-like" format,
	  with the FITS header and binary data in separate files, with the
	  binary data in the native format of the machine rather than in IEEE
	  format.  Then the process of converting this to true FITS should be
	  trivial.

	* That FITS is out of date.  Somebody at the meeting compared writing
	  FITS files with eight character keywords to trying to program in
	  FORTRAN II.  People seemed to prefer the more flexible approach used
	  by PDS/ODL, or by PVL in the SFDU formalism.

	* That FITS is not flexible enough to store the complex data structures
	  that we will be generating.  A real concern--this was the crux of my
	  presentation.

What I emphasized in my presentation, and what people seemed to agree on, was
that the best way to store the data from our particular instrument--the CDS
Normal Incidence Spectrograph--was to use the "TDIMnnn" approach from Appendix
A of the Binary Tables document.  This was because it is the most
straightforward approach, and because it allows us to easily distinguish
between information that applies to an exposure, and information that applies
to a window array that is part of that exposure.  Information that applies to
an exposure as a whole--such as start and stop times--would appear as separate
columns in the table; and information that applies to a window--such as
position on the CCD--would appear in the table header as keywords.  With the
other possibilities considered, this would not have been possible without a lot
of needless repetition.  It was decided that when the observation mode
changed--i.e. when the number, positions, and/or sizes of the windows on the
CCD changed--then we would either start a new FITS file or a new binary table
extension within the same FITS file.

So the SOHO community is currently planning on using the binary tables
extensions together with the optional "TDIMnnn" keywords.  It also looks like
we will also be using other keywords with the "KEYWORD = '(a,b,c)'" structure
to associate other information with the dimensions of our data.

The following LaTeX document should be printed in landscape orientation.

Bill Thompson
Applied Research Corporation

NASA Goddard Space Flight Center.

---------------------------------CUT HERE-----------------------------------
\documentstyle[12pt]{article}

\special{landscape()}

\setlength{\oddsidemargin}{0in}
\setlength{\topmargin}{-0.5in}
\setlength{\textwidth}{9in}
\setlength{\textheight}{6.5in}

\addtolength{\parskip}{0.5\baselineskip}

%  The environment centerpage centers text vertically on the page.  This is
%  useful within a titlepage environment.

\newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{
	\protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}}

%  The environment quotetable combines the quote environment with the tabular
%  environment for unnumbered tables that appear within the text.  The
%  environment centertable does the same, except that tables are centered
%  instead of indented.

\newenvironment{quotetable}[1]{\begin{quote}\begin{tabular}
	{ at {}#1}}{\end{tabular}\end{quote}}
\newenvironment{centertable}[1]{\begin{center}\begin{tabular}
	{#1}}{\end{tabular}\end{center}}

\begin{document}
\pagestyle{empty}

\large

\begin{centerpage}

\centerline{\huge\bf Storing SOHO Data in FITS Files}

\begin{description}
\item[The problem:]
Traditional FITS files are unable to handle data sets with multiple arrays of
different sizes, data types, and dimensions.
\item[Possible solutions:]
The use of the EXTEND keyword permits additional capabilities to the FITS
format.  Possible extensions that could be used are:
\begin{itemize}
\item
the ``IMAGE'' extension,
\item
the ``BINTABLE'' extension with the ``Multidimensional Array'' convention,
\item
the ``BINTABLE'' extension with the ``Virtual FITS'' convention,
\item
the ``BINTABLE'' extension with the ``Variable Dimensions'' convention.
\end{itemize}
\end{description}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf The ``IMAGE'' Extension}

\begin{itemize}
\item
Proposed by the IUE community.
\item
Each array has its own separate FITS header.  Essentially a series of FITS
files concatenated together into one file.
\item
Example:
\begin{centertable}{l}
Main FITS header, contains ``EXTEND = T''.\\
Binary array\\
Extension FITS header, contains ``XTENSION = 'IMAGE' ''.\\
Binary array\\
\hspace{3em} \vdots
\end{centertable}
\item
Advantages:  Easily read.  Supported by IUE.
\item
Disadvantages:  Wasteful of disk space.
\end{itemize}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf The ``Binary Tables'' Extension}

\begin{itemize}
\item
Based on tables model used by relational database and spreadsheet software.
\item
Data organized into rows and columns.  The columns represent different data
objects (e.g. time, pitch, yaw, etc.) and the rows represent different entries
(e.g. successive observations).
\item
Richer selection of data types than in ordinary fits, including string
variables as columns in the binary tables.
\item
Columns can be arrays.  Columns can be defined as either having a fixed or
variable size.
\item
There is no official way to assign dimensional information to entries in binary
tables.  However, there are several proposed conventions as to how to do this.
\item
Example:
\begin{centertable}{l}
Main FITS header, contains ``EXTEND = T''.\\
Binary array (optional)\\
Extension FITS header, contains ``XTENSION = 'BINTABLE' ''.\\
Binary table arrays\\
\end{centertable}
\end{itemize}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf The ``Multidimensional Array'' Convention}

\begin{itemize}
\item
Described in the Binary Tables proposal document.
\item
Assigns a keyword to a column of the form ``TDIMnnn = '(l,m,n)' ''.
Dimensional information derived by parsing character string.
\item
Advantages:  Straightforward, referencable standard.  Uses disk space well.
Easy to concatenate similar observations together.
\item
Disadvantages:  May not be supported by all FITS readers.  However, the data
itself will be readable---it's just a question of interpretation.
\end{itemize}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf Example using ``TDIMnnn''}

\begin{centertable}{|c|l|l|l|}
\hline
	& \multicolumn{3}{|c|}{Columns}\\
\cline{2-4}
	& 1			& 2			& 3\\
\cline{2-4}
	& TFORM1='1000I'	& TFORM2='2000I'	& TFORM3='2500I'\\
Rows	& TTYPE1='Window\_1'	& TTYPE2='Window\_2'	& TTYPE3='Window\_3'\\
	& TDIM1='(10,100)'	& TDIM2='(20,100)'	& TDIM3='(50,50)'\\
	& XPOS1=10		& XPOS2=200		& XPOS3=500\\
	& YPOS1=10		& YPOS2=10		& YPOS3=100\\
\hline
1	& Image			& Image			& Image\\
2	& Image			& Image			& Image\\
\vdots	& \vdots		& \vdots		& \vdots\\
\hline
\end{centertable}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf The ``Virtual FITS'' Convention}

\begin{itemize}
\item
Also known as the ``Green Bank'' convention.  Definition document not available
yet.
\item
Defines a binary table where each column is associated with a keyword in an
ordinary FITS file, but with slightly different names.  Parameters which would
be the same for each row can be substituted with a keyword.
\item
The optional variable length array facility should be used if one wants to
minimize file size.  Otherwise, padding would be required.
\item
Advantages: Somewhat more FITS-like than the ``TDIMnnn'' convention.  Supported
by Green Bank.
\item
Disadvantages: Less straightforward than ``TDIMnnn'' convention.  Only allows
for one multidimensional array per record.  The variable length array facility
may not be supported by all FITS readers.  Difficult to associate auxiliary
data with different observations.
\end{itemize}

\end{centerpage}
\pagebreak

\begin{centerpage}

\normalsize

\centerline{\huge\bf Example using ``Virtual FITS''}

\mbox{}

\centerline{\large (Note that keyword ``MAXIS = 2'' is included in table
header)}

\begin{centertable}{|c|l|l|l|l|l|}
\hline
	& \multicolumn{5}{|c|}{Columns}\\
\cline{2-6}
	& 1			& 2			& 3
	& 4			& 5\\
\cline{2-6}
Rows	& TFORM1='1I'		& TFORM2='1I'		& TFORM3='1I'
	& TFORM4='1I'		& TFORM5='1PI(2500)'\\
	& TTYPE1='MAXIS1'	& TTYPE2='MAXIS2'	& TTYPE3='XPOS'
	& TTYPE4='YPOS'		& TTYPE5='ARRAY'\\
\hline
1	& 10			& 100			& 10
	& 10			& Pointer\\
2	& 20 			& 100			& 200
	& 10			& Pointer\\
3	& 50			& 50			& 500
	& 100			& Pointer\\
\vdots	& \vdots		& \vdots		& \vdots
	& \vdots		& \vdots\\
\hline
N+1	& 10			& 100			& 10
	& 10			& Pointer\\
N+2	& 20 			& 100			& 200
	& 10			& Pointer\\
N+3	& 50			& 50			& 500
	& 100			& Pointer\\
\vdots	& \vdots		& \vdots		& \vdots
	& \vdots		& \vdots\\
\hline
\vdots	& \vdots		& \vdots		& \vdots
	& \vdots		& \vdots\\
\hline
\multicolumn{6}{|c|}{Variable array 1 (2000 bytes)}\\
\multicolumn{6}{|c|}{Variable array 2 (4000 bytes)}\\
\multicolumn{6}{|c|}{Variable array 3 (5000 bytes)}\\
\multicolumn{6}{|c|}{\vdots}\\
\hline
\end{centertable}

\end{centerpage}
\pagebreak

\begin{centerpage}

\centerline{\huge\bf The ``Variable Dimensions'' Convention}

\begin{itemize}
\item
Most general case.  Not only allows any column to be multidimension, but also
allows the dimensions to be different for each row.
\item
Dimensions for one column stored in another column, with a keyword pointing
between them.  Keyword is of the form ``TNDIMnnn = mmm'' where nnn is the
column with the multidimensional array, and mmm is the column with the shape
information.  The first number in each row of column mmm is the number of
dimensions, and the following numbers are the dimension sizes.
\item
The optional variable length array facility should be used if one wants to
minimize file size.  Otherwise, padding would be required.  Again, this may not
be supported by all FITS readers.
\item
Similar advantages and disadvantages as the ``TDIMnnn'' convention, but more
general.  In fact, this convention is more general than the CDS example
requires.
\end{itemize}

\end{centerpage}
\pagebreak

\begin{centerpage}

\normalsize

\centerline{\huge\bf Example using ``TNDIMnnn''}

\begin{centertable}{|c|l|l|l|l|l|}
\hline
	& \multicolumn{5}{|c|}{Columns}\\
\cline{2-6}
	& 1			& 2
	& 3			& 4			& $\cdots$\\
\cline{2-6}
	& TFORM1='1000I'	& TFORM2='3I'
	& TFORM3='2000I'	& TFORM4='3I'		& \\
Rows	& TTYPE1='Window\_1'	& TTYPE2='Window\_1\_Dims'
	& TTYPE3='Window\_2'	& TTYPE4='Window\_2\_Dims' & \\
	& TNDIM1=2		&
	& TNDIM3=4		&			& $\cdots$\\
	& XPOS1=10		&
	& XPOS2=200		&			& \\
	& YPOS1=10		&
	& YPOS2=10		&			& \\
\hline
1	& Image			& 2, 10, 100
	& Image			& 2, 20, 100		& \\
2	& Image			& 2, 10, 100
	& Image			& 2, 20, 100		& $\cdots$\\
\vdots	& \vdots		& \vdots
	& \vdots		& \vdots		& \\
\hline
\end{centertable}

\end{centerpage}

\end{document}

---

From: baalke at kelvin.jpl.nasa.gov (Ron Baalke)
Date: 18 NOV 91 22:43:42

In article <18NOV199116070809 at stars.gsfc.nasa.gov>, thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes...
>upcoming Solar Heliospheric Observatory (SOHO).  

The SOHO data will be Reed Solomon encoded when it is transmitted by the
spacecraft, and then decoded at the tracking station.  However, the check
symbols will still be in place at the end of each frame of data.  Does FITS
provide some mechanism to ignore portions of the data, in this case the
check symbols?  If the check symbols are removed and the data pieced together
before being converted to FITS, then this wouldn't be a problem.
      ___    _____     ___
     /_ /|  /____/ \  /_ /|     Ron Baalke         | baalke at kelvin.jpl.nasa.gov
     | | | |  __ \ /| | | |     Jet Propulsion Lab |
  ___| | | | |__) |/  | | |__   M/S 301-355        | The two hardest things to
 /___| | | |  ___/    | |/__ /| Pasadena, CA 91109 | handle in life are success
 |_____|/  |_|/       |_____|/                     | and failure.

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Tue, 19 Nov 1991 21:06:00 GMT

In article <1991Nov18.225256.4063 at elroy.jpl.nasa.gov>,
baalke at kelvin.jpl.nasa.gov writes...

>In article <18NOV199116070809 at stars.gsfc.nasa.gov>,
thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes...
>before being converted to FITS, then this wouldn't be a problem.

I was only talking about level-1 data products generated by the P.I. teams in
the Experiment Operations Facility, e.g. the summary data, etc.  This has
nothing to do with the way the level-0 data is formatted for telemetry
purposes.

Bill Thompson