Subject: Multidimensional arrays in binary tables

From: thompson at stars.gsfc.nasa.gov (Bill Thompson)
Date: 27 Sep 91 23:06:13 GMT

Earlier, I sent an example of what I thought a header would look like with
Cotton and Tody's suggested TDIMnnn convention.  I was in favor of that
convention, but wondered whether other people were planning to use and/or
support this convention.  This time I'm sending the same example case, but
using something Don Wells calls the "Green Bank Convention".

My understanding of the G.B. convention is as follows.  A column is assigned
for each keyword of the basic FITS header, with the column label designating
which keyword, except that NAXISn becomes MAXISn.  Keywords which would be the
same for each row of the table can be entered as a keyword in the binary table
header rather than as a column (it is described as a virtual column).  The
multi-dimensional data array is stored as another column with a specific column
label which I've assumed to be ARRAY.

I chose to put the keywords MAXIS1 and MAXIS2 in columns 1 and 2.  Since MAXIS
would always be 2, I put it in the header.  I didn't think that the equivalent
of BITPIX (MBITPIX?) would be necessary, as that same information is given by
the keyword TFORM5.  While using the variable length facility is not required
by this convention, I used it because it represents the most efficient way of
storing the data.  Otherwise, padding would be required.

A significant difference between this convention and the TDIMnnn convention, is
that this allows only one multidimensional array per row in the binary table.
It is therefore not obvious to me how one would most adequately describe a
situation in which there would be a series of identical exposures, with the
same window configuration in each exposure.  With the TDIMnnn convention, the
most straightforward way to do this was to make each row in the binary table a
different exposure.

Another difference is that TDIMnnn signaled its use from the very presence of
the TDIMnnn keyword.  It seems to me that it would be harder for a generalized
FITS reader to recognize when the "Green Bank Convention" was being used.

		    Example (using "Green Bank Convention")

>From my understanding, the following should be a valid binary tables extension
header.  This is as close an approximation as I can get to the "Green Bank
Convention" based on the information I currently have.  It describes an
observation with three windows on the CCD.

			Window	Size	Position

			  1	10x100  10,10
			  2	20x100	200,10
			  3	50x50	500,100

In this example, the array sizes and positions are stored inside the binary
table.

XTENSION= 'BINTABLE'            /  Binary table extension
BITPIX  =                    8  /  Binary data
NAXIS   =                    2  /  Table is a matrix
NAXIS1  =                   12  /  Width of table row in bytes
NAXIS2  =                    3  /  Number of rows in table
PCOUNT  =                11000  /  Variable length parameter count
GCOUNT  =                    1  /  Group count
TFIELDS =                    5  /  Number of columns
THEAP   =                   36  /  Start byte of variable length parameters
TFORM1  = '1I      '            /  16-bit integer
TTYPE1  = 'MAXIS1  '            /  Column 1 contains the 1st dimension size
TFORM2  = '1I      '            /  16-bit integer
TTYPE2  = 'MAXIS2  '            /  Column 2 contains the 2nd dimension size
TFORM3  = '1I      '            /  16-bit integer
TTYPE3  = 'COLUMN0 '            /  Column 3 contains window starting column
TFORM4  = '1I      '            /  16-bit integer
TTYPE4  = 'ROW0    '            /  Column 4 contains window starting row
TFORM5  = '1PI(2500)'           /  Variable length 16-bit integer array
TTYPE5  = 'ARRAY   '            /  Column 5 contains the window array
MAXIS   =                    2  /  Each window has 2 dimensions
END

The data portion of this FITS extension would be organized as is shown
symbolically below:

	10 ; 100 ;  10 ;  10 ; <Pointer> ;
	20 ; 100 ; 200 ;  10 ; <Pointer> ;
	50 ;  50 ; 500 ; 100 ; <Pointer> ;
	Variable array 1 (2000 bytes) ;
	Variable array 2 (4000 bytes) ;
	Variable array 3 (5000 bytes)

There are no gaps anywhere in the datastream.  The variable length arrays start
immediately after the last byte in the binary table.

---

From: tody at noao.edu (Doug Tody)
Date: 28 Sep 91 03:02:26 GMT

>From article <1991Sep27.220632.11388 at nsisrv.gsfc.nasa.gov>, by thompson at stars.gsfc.nasa.gov (Bill Thompson):

> A significant difference between this convention and the TDIMnnn convention,
> is that this allows only one multidimensional array per row in the binary
> table.  It is therefore not obvious to me how one would most adequately
> describe a situation in which there would be a series of identical
> exposures, with the same window configuration in each exposure.  With the
> TDIMnnn convention, the most straightforward way to do this was to make each
> row in the binary table a different exposure.

Bill, that's easy - you need to add another column giving the exposure
number to the table.  Select all rows having the same exposure number,
and you have all the regions output for that exposure.

The advantage of this over the all-on-a-row approach is that the record
structure remains the same regardless of the number of subregions you read
out.  This, along with the modest number of columns, could significantly
simplify the software required to produce and read the table.
-- 
Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217
UUCP: {arizona,decvax,ncar}!noao!tody  or  uunet!noao.edu!tody 
Internet: tody at noao.edu             SPAN/HEPNET: NOAO::TODY (NOAO=5355)

---

From: thompson at stars.gsfc.nasa.gov (Bill Thompson)
Date: 30 Sep 91 16:26:48 GMT

In article <1991Sep28.030226.5900 at noao.edu>, tody at noao.edu (Doug Tody) writes...

>and you have all the regions output for that exposure.

Yes, I also thought of that.  The difficulty with that approach is that it 
doesn't really express the underlying structure of my anticipated database.  I 
think that should also be the goal of designing a FITS file format.  However, I 
could also think of cases where that approach would make more sense than 
putting everything in a separate column.

I am in the process of preparing a proposal that would allow me to use the 
"Green Bank" convention, but still delineate the difference between windows in 
the same exposure, and different exposures.  I will send out this proposal in a 
later message.

Another one of my goals is to avoid padding out oversized arrays.  The TDIMnnn
approach allows me to do that in one way.  The "Green Bank" convention also
allows me to do that, but only if I use the more complicated variable array
size facility.