Subject: Re: 3-D Binary Tables

From: [email-masked] (Don Wells)
Date: Tue, 14 Nov 89 15:50:26 EST

Dear Friends of FITS,

Bob Wilson of Bell Labs, who did not attend the recent Green Bank FITS
workshop (30 Oct - 01 Nov), wrote to me this morning asking what we had
agreed to, because he is ready to re-write their data acquisition program.
Here is my answer to him. Other people who did not attend may also find
this review of the meeting to be of some use, so
I am broadcasting it. Please note that this is my own very personal
view of what happened; it is surely biassed and/or incomplete.
Other participants are free to broadcast their versions of course.
Because this workshop, which was convened to agree on conventions for
single dish radio astronomy, in fact proposed a dramatic new set of
architectural conventions for all of FITS, conventions of great generality,
I have decided to broadcast this summary to the "ASTRO" and "BANANAS"
exploders, as well as to the "DISHFITS" exploder. Many people are on
more than one of these distribution lists and will get multiple copies.
Sorry about that...

		= = =

Dear Bob,

The big appendix of my draft paper of a month or so ago, 65 KB long,
contains the text from chapter 14 of "Going AIPS", the AIPS programmer's 
reference manual, which includes the description of Bill Cotton's "3-D 
Binary Tables" extension. Probably you have a copy of Going AIPS around,
and can copy it. Or else you probably got my big paper (if not I can
re-send it) and can print the appendix from it. This is currently the 
only description of 3-D binary tables. It is also completely sufficient
as a formal description. Because everyone has endorsed Bill's design we 
know that he will revise and extend that Going AIPS text into a formal paper
which will be the description that the FITS committees are now guaranteed
to endorse. Also, we know that nothing in that description will be
changed in any significant way. One or two new data types will
probably be added; the candidates discussed so far are 8-bit unsigned
integers and perhaps single precision complex. Its name will be changed
from 'A3DTABLE' to XTENSION='3DTABLE'.

The principal complaint against Bill's design was that it only allowed
1-D vectors in the fields; that hard fixed limit made many of the
designers uneasy. My draft paper said that the spectrum dimension should be
in a separate column of the table. When Bob Hanisch challenged me on this
point my answer on DISHFITS was that I wanted columns called NAXIS and NAXIS1
to define the dimension and "trusted that the generalization was obvious".
I did this because I wasn't sure how ready the spectroscopy community 
was to accept the idea that I really wanted to propose a design that was
far more general than just 1-D spectroscopy. By the time of the meeting
I judged that I could get away with it, and so in my 
verbal presentation I exposed the underlying concept, which was that
all of the architecture of Basic FITS N-dimensional matrix technology
could be mapped onto rows of binary tables. I.e., that each row could be 
a virtual Basic FITS file (required keywords plus optional keywords
plus N-dimensional binary matrix). This idea was accepted fully by
the workshop. Note that this is just a convention about how to use
Bill's primitive binary design and so his paper and the paper that 
will ultimately document the convention are independent. Also, the
optional keywords and value conventions peculiar to radio spectroscopy
must be agreed to by the discipline specific group and appear as yet
another independent paper. Eric's recent DISHFITS/BANANAS discussion
of real-time FITS via binary tables is also just a convention which
is independent of the three papers mentioned above.

Aside from any technical merit that my proposal may have 
(the 10-year success history of Basic FITS suggests plenty of merit) 
the real reason I proposed this architecture was that it
eliminates almost all options on conventions for keywords and values,
because it means that the proposed architecture automatically inherits
all of the prior FITS agreements and R-and-D work *intact*. Thus it vastly
simplified the negotiations, which were easily completed by the end
of the second day (except for a number of details which the editors
of the drafts will handle). It also means that implementors have almost
nothing new to learn, that they already have an understood 
language for describing any new implementations that use the convention.
I considered these advantages to be compelling and apparently others
agreed.

A major objection to my proposal was that real data acquisition
systems often need auxiliary arrays to accompany the main array, and
many designers wanted these to be in the same row. Sure, Bill's binary 
tables can handle this problem trivially, but that is just saying that
anarchy is always an implementable design. Negotiations were in
difficulty on this point for some hours until Eric Greisen suddenly proposed
a simple compromise which saved my proposal. Eric agreed that
we could only afford to have one set of columns to document the
matrix structure (Preben Grosbol's alternative proposal for multi-valued 
NAXISn was considered briefly by the workshop, even though Preben himself
labelled it "disgusting" and said he wouldn't vote for it; I myself thought 
it was an ingenious negotiating ploy that stimulated the compromise.) 
Eric proposed that multiple matricies be allowed with the understanding
that they have the same dimensionality as the primary matrix even if
not necessarily the same data type (the BITPIX keyword of Basic FITS
is redundant in my proposal because the 3-D binary table syntax does
the job). This proposal was accepted. It appears that only one new keyword
is needed to label the matricies. A recent DISHFITS message from
Jodrell proposed the obvious convention that the first matrix is
the primary one. An example of an auxiliary matrix would be IR detectors
that average multiple short integrations and have an RMS matrix.
Clearly any such row is equivalent to two or more Basic FITS files
with almost identical headers. Therefore it is still
consistent with the architectural concept that I proposed. It is clear that 
the architecture implies that a generalized filter could be built that 
would combine a set of Basic FITS files into a table extension or would
decompose the table extension back into the files.

The net effect of all this is that we have agreed on a set of conventions
built on top of the new 3-d binary table extension type which allows
us to write sets of astronomical observations *much* more efficiently 
than before because the repeated header data is now in binary form.
This is even more true becuase of my suggestion that keyword=value
pairs that are constant through a set of data (i.e., constant value
in a column of the table) should be pulled out into the header for
the table as keyword=value (i.e., a virtual column convention).
The total result is a nearly optimum design, better than I and others had 
anticipated when we began this design exercise. By combining the 
ideas of a whole team of people this new architecture merges
most of the prior history of Basic FITS, Random Groups and Tables
into one comprehensive design that can do spectroscopy plus almost
every other astronomy application that we currently know about.
Eric recently proposed that we consider using this design
instead of Random Groups for interferometer data. Before the GB meeting
any such suggestion would have been regarded as heresy, as anathema,
by all real FITS people. Now it seems reasonable.

The new design is particularly important for
spectroscopic applications, for which FITS has always had an efficiency
problem. Although sets of ordinary 2-D images can be encoded in the new
design it would be a *serious* mistake for everyone to rush to do so, 
at least until that packing/unpacking filter is written by someone! 
The reason, of course, is that for ten years we have had 
effective interchange of binary imagery in astronomy using Basic FITS,
and it would be irresponsible to undermine this solid success. Likewise,
although ASCII tables can be encoded several times more efficiently
in binary form, it would be a mistake to do this for catalogs of 
observations in which the efficiency is more than compensated by the
greater degree of interoperability given by XTENSION='TABLE'. (Obviously
another generalized filter could be built to convert ASCII tables to
binary and vice versa; some student should take this on as an exercise,
hint, hint...)

		= = = 

The Grosbol/Wells hierarchical name convention proposed in another appendix 
of my DISHFITS paper (and which was discussed at the 1989 FITS meetings)
is currently favored, but some opposition still exists.
The FITS committee people agreed to resolve this disagreement within the next
month or so. There was a consensus at the Green Bank Workshop that astronomy 
really should use hierarchical keywords even though they are not absolutely 
required. Note that the new data architecture does not depend in any 
critical way on the hierarchical convention, if any, that is ultimately
endorsed.

Drafts of papers will soon appear on DISHFITS etc. with more details,
especially radio spectroscopy details, and will surely circulate widely
within the astronomical community. 

The high energy astronomy community appear now to have a consensus
favoring the binary tables for interchange of lists of descriptions of
photon events. Thus the 3-D binary extension proposal will have the 
backing of the Very Long Baseline Array (for which it was originally
developed), the radio single dish (spectroscopy) community and the high 
energy community. The FITS committee chairmen intend to recommend 3-D binary 
tables to the spring 1990 European and North American FITS meetings for 
endorsement because of the consensus that now exists favoring this design.

		= = = 

You ask: "Is there any chance that NRAO will publish a 'suggested' single 
dish FITS reader in code, pseudo-code, or flow chart form to guide the
naive implementer?" This must be number one on the all-time list of 
frequently-asked-questions regarding FITS. The standard answer is "maybe,
but don't hold your breath while waiting for it." The reason is that the
problem is not well-posed. The purpose of a reader is to translate from
the language of FITS to the language of a *PARTICULAR* data analysis system. 
Your question is like asking "is there any chance of devising a translator
from English to other languages?" If you ask for an English-to-French
translator you at least have a well-posed problem. Likewise, if you
ask for a single-dish FITS reader for AIPS or MIDAS or IRAF or ... you
will have a well-posed problem -- in each specific case. For AIPS and 
MIDAS the existing FITS readers can already read 3-D binary tables into their
tables systems. Currently only AIPS supports vectors in the tables, but
MIDAS will add them during 1990. Because of the interest among the high 
energy community it seems plausible that IRAF will acquire basic binary table
capability in its reader and writer during 1990, but I don't 
know the details of those plans. Once the data are in a generalized 
table system on disk their interpretation according to the agreed
conventions can be coded into application programs that read the fields
of the tables. The virtual columns will be stored as auxiliary files
of keyword=value pairs in all of the large data analysis systems, and
those files can be searched by library routines called by the spectroscopic
application programs.

A new answer to your question is that NASA has issued a contract to
produce a FITS "User's Guide" (an Implementors Handbook of Recommended 
Practices). The work is being done by Barry Geldzahler.
The first draft of this document is well advanced and will probably be
circulated during 1990. I don't expect that it will contain the sort
of pseudo-program that you asked for, but it will contain datatype
translation routines for specific architectures, a variety of hints,
do-s and don't-s, etc. The new FITS Support Office at GSFC is working
on yet another relevant project, the codification of the FITS papers
into a single canonical "FITS Standard" document. Barry Schlesinger
is doing this work, which will also circulate in draft form during 1990.

As for writing FITS, it has always been true that it is easy to produce a 
program to write almost any dataset out into FITS. Generalized readers
are the hard problem.

		= = =

A Sign-Of-The-Times: Many of the discussions during the Workshop were
conducted in the language of relational database systems. Relational
theory arguments were invoked by several participants as reasons for or
against various proposed design features. The proposed architecture
of Basic-FITS-files-in-Binary-Table-rows, even the binary vectors,
maps directly onto many of the current state-of-the-art RDBMS systems.
It is not inconceivable that in a few years one could decide to 
produce a generalized tool to read the table extensions into relations
in a database and then code application programs to manipulate the
relations. This may be a good path to a new generation of data analysis
systems for astronomy. We shall see...

Best regards,
Don

Donald C. Wells, Associate Scientist | NSFnet: [email-masked] [192.33.115.2]
National Radio Astronomy Observatory | SPAN:   NRAO::DWELLS    [6654::]
Edgemont Road                        | BITnet: DWELLS@NRAO
Charlottesville, VA 22903-2475 USA   | UUCP:   ...!uunet!nrao.edu!dwells
+1-804-296-0277  (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278