Subject: Any thoughts on nD arrays in binary tables?

From: thompson at stars.gsfc.nasa.gov (Bill Thompson)
Date: 26 Sep 91 16:24:09 GMT

I am connected with the Coronal Diagnostic Spectrometer for the Solar
Heliospheric Observatory (SOHO).  We plan to use FITS as the standard for data
storage.  My problem is that our data does not seem to exactly fit into the
FITS standard.

The detector we are building uses a CCD chip for readout.  We will be able to
operate this chip in "windowed-mode" in which we will be able to select out
up to 40 rectangular areas on the CCD chip that will actually be read out, with
the rest of the chip being ignored for speed.  One way to represent this in a
data file would be to reconstruct the entire chip, with areas outside the
windows being set to zero or NaN.  However, this would be extremely wasteful of
space, so I would prefer to store each window as a separate array within the
same file.  The problem is that each window could have a different size from
all the others.

I just downloaded the document "Binary Table Extension to FITS: A Proposal" by
W. D. Cotton and D. Tody.  The draft date is given as 20 September 1991.  I
found it very interesting.  In particular I was interested in Appendix A, the
"Multidimensional Array" Convention (I haven't digested Appendix B yet).  It
seems to address this vary issue.

Basically, they suggest using, as part of the binary tables extension, the
additional keywords TDIM1, TDIM2, etc., where the number refers to the column
associated with it.  I give an example below of my interpretation of how this
would be coded in the header.  This example shows only a single exposure, but I
can imagine a series of identically formatted exposures with each row in the
binary table representing a specific exposure.  This would be quite consistent
with the way we intend to operate our instrument.

I guess I'm still a little nervous about the fact that the TDIMnnn keyword may
not be generally supported by FITS readers.  I'm not worried that we couldn't
read our data, but that colleagues we send our data to could not use it.
According to the Cotton and Tody document, this keyword (Appendix A) is not
part of their proposed binary table definition.

I've also seen mention of something called the "Green Bank Convention" in an
earlier news article by Don Wells.  If I understand this correctly, the format
and dimensions of the array (the equivalents of NAXIS, NAXIS1, etc.) would each
appear as separate columns in the binary table, and each row would represent a
different window from the same exposure.  The data itself could either be null
filled to the maximum array size, or stored as variable length arrays using the
facility described in Appendix B of the Cotton and Tody document.

Either way would work at some level.  Personally, I prefer the "TDIMn"
approach--it seems less complicated to me.  I figure, though, that other people
must be struggling with similar problems, and I was wondering what the
community saw themselves as standardizing on.

Thank you,

Bill Thompson

			     Example (using TDIMn)

>From my understanding, the following should be a valid binary tables extension
header.  It describes an observation with three windows on the CCD.

			Window	Size	Position

			  1	10x100  10,10
			  2	20x100	200,10
			  3	50x50	500,100

The keywords TROWnnn and TCOLnnn were introduced by myself.  This can easily be
generalized to a series of observations made with the same window configuration
simply by modifying NAXIS2.

XTENSION= 'BINTABLE'            /  Binary table extension
BITPIX  =                    8  /  Binary data
NAXIS   =                    2  /  Table is a matrix
NAXIS1  =                11000  /  Width of table row in bytes
NAXIS2  =                    1  /  Number of rows in table
PCOUNT  =                    0  /  Random parameter count
GCOUNT  =                    1  /  Group count
TFIELDS =                    3  /  Number of windows
TFORM1  = '1000I'               /  16-bit integer
TTYPE1  = 'Window #1'           /  Label for window 1
TDIM1   = '(10,100)'            /  Dimensions of window 1
TROW1   =                   10  /  Starting row of window 1
TCOL1   =                   10  /  Starting column of window 1
TFORM2  = '2000I'               /  16-bit integer
TTYPE2  = 'Window #2'           /  Label for window 2
TDIM2   = '(20,100)'            /  Dimensions of window 2
TROW2   =                  200  /  Starting row of window 2
TCOL2   =                   10  /  Starting column of window 2
TFORM3  = '2500I'               /  16-bit integer
TTYPE3  = 'Window #3'           /  Label for window 3
TDIM3   = '(50,50)'             /  Dimensions of window 3
TROW3   =                  500  /  Starting row of window 3
TCOL3   =                  100  /  Starting column of window 3
END

---

From: tody at noao.edu (Doug Tody)
Date: 27 Sep 91 21:47:21 GMT

> From article <1991Sep26.152517.28870 at nsisrv.gsfc.nasa.gov>, by
> thompson at stars.gsfc.nasa.gov (Bill Thompson):
> 
> Heliospheric Observatory (SOHO).  We plan to use FITS as the standard for
> data storage.  My problem is that our data does not seem to exactly fit into
> the FITS standard.
> 
> The detector we are building uses a CCD chip for readout.  We will be able
> to operate this chip in "windowed-mode" in which we will be able to select
> out up to 40 rectangular areas on the CCD chip that will actually be read
> out, with the rest of the chip being ignored for speed.  One way to
> represent this in a data file would be to reconstruct the entire chip, with
> areas outside the windows being set to zero or NaN.  However, this would be
> extremely wasteful of space, so I would prefer to store each window as a
> separate array within the same file.  The problem is that each window could
> have a different size from all the others.

A third possibility which you didn't mention would be to write each
subregion out as a separate FITS image.  If ease of reading the data at
another site is important this might be the best approach.  Provided the
subregions are large (several hundred pixels square or larger) the overhead
would not be so bad.  I agree that if you have to record many small
subregions, the complexity of a table might well be justified.

If you decide to use a table there are two possible approaches, as outlined
in your article: one table row per subregion, or all the subregions for
an exposure on one row.  Either approach will work.  If you want another
opinion, I think I would probably go with a simple record structure and
output one subregion per table row (as in the Green Bank convention), using
two columns to specify the subregion size in X and Y, and one array type
column for the pixels.  A fixed size array should be used if the regions
do not vary greatly in size; only if this becomes seriously inefficient
should you consider using a variable length array.

> I just downloaded the document "Binary Table Extension to FITS: A Proposal"
> by W. D. Cotton and D. Tody.  The draft date is given as 20 September 1991.
> I found it very interesting.  In particular I was interested in Appendix A,
> the "Multidimensional Array" Convention (I haven't digested Appendix B
> yet).  It seems to address this vary issue.

If you want to use binary tables I should remind you that this is not yet
a FITS standard.  That doesn't mean that you shouldn't use binary tables,
just that there is no guarantee that things will not change.  Also there
is not a whole lot of software out there yet that can do anything very
useful with such data.

> additional keywords TDIM1, TDIM2, etc., where the number refers to the
> column associated with it.  I give an example below of my interpretation of
> how this would be coded in the header.  This example shows only a single
> exposure, but I can imagine a series of identically formatted exposures with
> each row in the binary table representing a specific exposure.  This would
> be quite consistent with the way we intend to operate our instrument.
> 
> I guess I'm still a little nervous about the fact that the TDIMnnn keyword
> may not be generally supported by FITS readers.  I'm not worried that we
> couldn't read our data, but that colleagues we send our data to could not
> use it.  According to the Cotton and Tody document, this keyword (Appendix
> A) is not part of their proposed binary table definition.

The binary tables proposal tries to avoid the issue of how to represent
multidimensional arrays because it is a harder problem than it might at
first appear.  This very discussion points out the kind of problem that you
can run into - TDIM doesn't work if the array size changes in each record.
We need more practical experience with embedding multidimensional arrays
in tables before we can extend FITS in this area.

> Either way would work at some level.

Right.  It is a database design problem.  Tables just provides the method,
it doesn't help you decide how to represent the data.

-- 
Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217
UUCP: {arizona,decvax,ncar}!noao!tody  or  uunet!noao.edu!tody 
Internet: tody at noao.edu             SPAN/HEPNET: NOAO::TODY (NOAO=5355)