Subject: Possible additional keywords for binary tables.

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: 21 Nov 91 22:49:00 GMT

I've been looking into the question of using the binary tables extension for
SOHO data, and it seems to me that there are capabilities in basic FITS that
are not replicated in the binary tables extension, namely the keywords CTYPEn,
CRPIXn, CROTAn, CRVALn, CDELTn, DATAMAX, and DATAMIN.  However, I believe that
I can overcome this problem by defining additional keywords for the binary
tables extension, such that
				       
			     DATAMIN  -->  TDMINm
			     DATAMAX  -->  TDMAXm

			     CTYPEn   -->  TDESCm
			     CRPIXn   -->  TRPIXm
			     CROTAn   -->  TROTAm
			     CRVALn   -->  TRVALm
			     CDELTn   -->  TDELTm

I separate out the first two keywords because these are fairly simple ones.
However, the other keywords are more complicated.  The index n on the left
refers to a dimension in an array, while the index m on the right refers to
columns in a binary table.  There aren't enough characters in an
eight-character FITS keyword to allow including both n and m in the name of the
keyword.

If one sticks to the convention that arrays in a binary table are
one-dimensional vectors, then there's no real confusion.  If, on the other
hand, one wants to use binary tables to represent more complicated data
structures as we do, then a problem seems to arise.

However, we also intend to use the TDIMm keyword as described in Appendix A of
the binary tables proposal.  This keyword has the structure

			     TDIMm   = '(l,m,n) '

Hence, it manages to get all the shape information for a multidimensional array
in a binary table into one keyword.  One could also use this same structure for
the keywords TDESCm, TRPIXm, TROTAm, TRVALm, and TDELTm, and that is what we
are currently thinking about doing.

William Thompson

---

From: leb at gsfc.nasa.gov (Lee E. Brotzman)
Date: 22 Nov 91 15:34:22 GMT

>William Thompson

I have considered these two keywords for our ASCII tables as well.  It's a 
good thing to know the allowable range of values for a given field.  When 
posing a database query against a table, it can save a lot of time if the
software can first determine if the query is within the bounds for the field.
Also, there would be a built-in "example" of what the data in the field
look like, so users unfamiliar with a particular dataset can pose sensible
queries right away.

The EPOCH and/or EQUINOX keywords can also be put to use in field-specific
fashion:

			TEPOCn		TEQNXn

Astronomical catalogs sometimes have more than one set of coordinates, 
especially now that we are in the "cross-over" period from B1950 to J2000.
There is currently no way for software to inspect a table and determine the
epoch and equinox of the coordinate fields.

Another thing I worry about with binary tables is the loss of precision 
information that is found in their ASCII counterparts.  For instance, a 
magnitude field can be given as "10.  ", "10.0 ", or "10.00" depending on the 
precision of the measurement. The value is read in correctly and the concerned 
scientist can recover the precision of the number by inspecting the ASCII 
representation.  The binary version throws away this information.  There is no 
way to remedy the problem with a simple keyword, since the precision 
information can change from row to row.  I strongly recommend that scientists 
generating binary tables pay attention to the precision of the data going into 
them.  Fields where precision is important should be followed by a field that 
is an unsigned byte containing the number of significant figures in the primary 
data field.  For example,

	TTYPE1  = 'Mag     '            / Magnitude field
	TFORM1  = '1E      '		/ Single precision floating point
	TUNIT1  = 'mag     '		/ Units are magnitudes

	TTYPE2  = 'Mag_Prec'            / Number of sig figs in Mag
	TFORM2  = '1B      '            / Unsigned byte

This may not be the most convenient of things to do, but it's important to
keep the binary tables "honest".  Authors of catalog data generally take great
pains to make sure the machine-readable versions of their data truly reflect
the actual values that were measured.  We can't let the expedience of binary
representations outweigh the importance of accurate scientific interpretation 
of the numbers.

--
-- Lee E. Brotzman                    Internet:  brotzman at nssdca.gsfc.nasa.gov
-- Hughes STX                         SPAN:      NSSDCA::BROTZMAN
-- National Space Science Data Center BITNET:    ZMLEB at SCFVM
-- NASA Goddard Space Flight Center

---

From: hanisch at stsci.edu (Bob Hanisch,N406,4910)
Date: 22 Nov 91 23:15:52 GMT

>From article <21NOV199118492538 at stars.gsfc.nasa.gov>, by thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040):
  This approach seems overly complex and misses one of the more elegant
aspects of binary tables, i.e., using the columns and column labels themselves
to store information that was formerly in the main header.  That is, rather
than define a new keyword for CTYPEn, make CTYPEn a column LABEL, i.e.,

TFORM3 = '8A'
TTYPE3 = 'CTYPE1'
TFORM4 = '1D'
TFORM4 = 'CRVAL1'

etc.  This is essentially identical to the indirect naming scheme we use in
STSDAS data files for group parameters.  The real virtue of this method is 
that you don't have to make up new names for things that have standard usages
already defined.  As I recall, this idea is included in the 'Green Bank'
convention for binary tables.

Bob Hanisch
ST ScI

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: 25 Nov 91 17:31:00 GMT

In article <1991Nov22.231552.12194 at stsci.edu>, hanisch at stsci.edu (Bob Hanisch,N406,4910) writes...
>> ... I can overcome this problem by defining additional keywords for the binary
>  ...This approach seems overly complex and misses one of the more elegant

It seems that in the FITS world, one person's overly complex is another
person's simple.  My reaction is that the above approach is more complex than
putting everything into one keyword.  I guess it depends on what you're used
to.

We, in fact, will be using that exact approach for certain keywords.  Our
binary tables will be organized so that each row represents a different
exposure, and then the columns will be the individual parts of that exposure
(for instance separate readout windows on the CCD).  If a keyword parameter
changes from exposure to exposure, then the above approach will be used.  If,
however, the keyword expresses a property of an entire column, then reproducing
that information over and over again would be very wasteful, and would be more
confusing than putting the information into a simple keyword.

A lot of people seem to think the whole TDIMnnn approach is complex, and then
come up with what I think are much more complex schemes to get around using it.
What we've decided is that the TDIMnnn is the simplest, most direct way of
entering multidimensional data into a binary table.  Once we decided to use
TDIMnnn, then the extensions to TDESCnnn, etc., fall out naturally.

Certainly, the "Green Bank" approach is more traditionally FITS-like.  If you
want to think of your data as a series of basic FITS images, then the "Green
Bank" approach expresses that situation well, and I can think of cases where
that would be useful.  In fact, I prefer to call it the "Virtual FITS"
convention.

However, that is not the case with our data.  To use the binary tables
effectively, we have to be able to store multidimensional arrays into a number
of columns.  We've looked at the "Green Bank" (or "Virtual FITS") approach, and
there are real problems with fitting our data into that format.  The problem is
that it has no real way of separating out information that applies to an array
that is part of an observation, and information that applies to an observation
as a whole.  One can do it, but it's very sloppy and inefficient.

I don't believe we're alone in that situation either.  One thing that the FITS
community has to realize that there are a lot of people who've held off from
using FITS because it doesn't meet their needs.  I've talked to other people
who've also been frustrated by the restrictive data model used by FITS in the
past.

Bill Thompson

---

From: dwells at fits.cx.nrao.edu (Don Wells)
Date: Mon, 25 Nov 1991 22:26:12 GMT

In article <25NOV199113313863 at stars.gsfc.nasa.gov>
thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
writes:

 WT> ... It seems that in the FITS world, one person's overly complex
 WT> is another person's simple... 

Many things in life are like that.

 WT> ...Our binary tables will be organized so that each row
 WT> represents a different exposure, and then the columns will be the
 WT> individual parts of that exposure (for instance separate readout
 WT> windows on the CCD).  If a keyword parameter changes from
 WT> exposure to exposure, then the above approach will be used.  If,
 WT> however, the keyword expresses a property of an entire column,
 WT> then reproducing that information over and over again would be
 WT> very wasteful, and would be more confusing than putting the
 WT> information into a simple keyword. 

An alternative model is the relational database in 3rd-Normal-Form.
Use *two* tables, one for exposure parameters, and one for the parts
of the exposure. A unique exposure_id implicitly binds the two tables
together. The result is an elegant combination of efficiency and
homogeneity in the design, with additional advantages such as easy
support for variable number of parts per exposure. If your instrument
observes multiple sources you might want to have a third table for
source parameters and use a unique source_id to implicitly bind
sources to exposures.  If you object to two (or three) tables, then
"join" the tables and let the duplicate values of exposure (and
source) parameters appear in the rows for the parts; the part
matricies are probably big enough that the loss in efficiency will not
be too bad. 

 WT> ... If you want to think of your data as a series of basic FITS
 WT> images, then the "Green Bank" approach expresses that situation
 WT> well, and I can think of cases where that would be useful...

The "Green Bank Convention" has two enormous advantages: (1) it
inherits all of the accumulated design experience of the community,
and the programmer training associated with that experience, so it
stops the endless arguing and reduces documentation requirements, and
(2) general programs can be specified which will transform a set of
Basic FITS images into a table and then can invert that transformation,
thus making the data available to a wider range of datasytems.

 WT> ... To use the binary tables effectively, we have to be able to
 WT> store multidimensional arrays into a number of columns... The
 WT> problem is that [the Green Bank Convention] has no real way of
 WT> separating out information that applies to an array that is part
 WT> of an observation, and information that applies to an observation
 WT> as a whole.  One can do it, but it's very sloppy and inefficient.

I don't agree with several of the assertions in this paragraph. See
the discussion above.

	-=-=- Relational vs. Hierarchical -=-=-

This difference of opinion is a special case of a traditional argument
in the database community. Before 1970 almost all databases with
technical problems like yours were designed in essentially the same
manner that you are proposing. I say "before 1970" because that was
when Codd presented the relational model of database design.
Proponents of the older heirarchical model made arguments analogous to
yours (and there are other arguments that you haven't yet made), and
some of these people still make these arguments today. One by one many
of the older systems have been redesigned using the relational model,
essentially because practical experience during the 80s has shown that
the resulting designs are easier to maintain and easier to use for
applications development. The FITS BINTABLE extension is flexible
enough to permit designers to build in either the heirarchical or the
relational styles (the present discussion proves this), but because
the usual heirarchical machinery is not explicitly included in the
BINTABLE design it will usually be simpler to use the relational
model.

--

Donald C. Wells             Associate Scientist        dwells at nrao.edu
National Radio Astronomy Observatory                   +1-804-296-0277
520 Edgemont Road                                 Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N

---

From: markus at mso.anu.edu (Markus Buchhorn)
Date: 26 Nov 91 00:56:58 GMT

thompson at stars.gsfc.nasa.gov (William Thompson) wrote:

[he didn't write the >>> stuff, I think.. sorry]
>putting everything into one keyword.  I guess it depends on what you're used

[Discussion about putting single values into <keyword>n format, or putting
 multiple values into a single <keyword>...]
>Bill Thompson

It appears that either approach has its benefits, depending who wants to use
it for what type of data. Perhaps a case could be made for the following:

Where a FITS file contains more than a single data-block (multiple exposures,
whatever), the user can decide to use either approach for keywords, as long
as they follow this convention:

(i) A single keyword containing multiple values, or a single value applicable
to all the data blocks under that header should have the form

<keyword>0 (e.g. CDELT0)

(ii) Whereas separate keywords containing a single value for a particular
data block should have the form

<keyword>n (e.g. CDELT1, CDELT2, etc.)

As far as I can tell ( which isn't all that far :-) ) this should be quite
easy to implement in any FITS reader that has to handle either format anyway.
 
Possibly something can be done to split case (i) if needs be -
perhaps <keyword>m and <keyword>s for multiple or single values for
that keyword.

Cheers,
	Markus

markus at mso.anu.edu.au

 #include <std.disclaim> /* Maybe I misread the intent of what you wrote :-)*/

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: 26 Nov 91 23:17:00 GMT

There seem to be several people who think that the multidimensional array
proposal from Appendix A of Cotton and Tody's Binary Table Extension proposal
is a bad idea, and that the community should standardize on the "Green Bank"
convention instead.  To summarize,

The multidimensional array proposal says that there should be additional
keywords, TDIM1, TDIM2, etc. for the various binary table columns to describe
them as being multidimensional.  The structure of the keyword (and this seems
to be the sticking point) is that it defines a 20x30 array in column 1 (for
example) with the statement TDIM1='(20,30)'.  Some people don't like the idea
of having to parse this string to get the dimensions.

The "Green Bank" proposal turns a binary table into a series of ordinary basic
FITS files, with a column for NAXIS, a column for NAXIS1, etc., and a column
for the data itself.  One could also have columns for other items that would
ordinarily be keywords in a basic FITS file.

I say that the "Green Bank" proposal is fine if your data fits a model based on
a simple series of arrays.  I can see where that would be very useful.  But I
argue that our data more complex than that, and that we want to use the full
power of binary tables.  Therefore, I am throwing my support behind Appendix A
>from the Cotton and Tody proposal.  I don't see any problem in parsing
character strings.

Bill Thompson