Subject: Re: GBT use of FITS for single dish data

From: [email-masked] (Bob Garwood)
Date: Thu, 2 Mar 1995 13:43:52 +0500

Hi Ray,

   Thanks for joining the conversation.  I hope that others out there
also chime in from time to time.

Before I get to your specific questions, let me state my philosophy on
this single dish FITS convention.

The purpose of the convention is to convey as much telescope independent
information as possible so that the data can be correctly used in
various analysis environments that may not have knowledge of the
quirks of the telescope that generated the data. 

Ok, now on to your questions:

"How do we store a series of spectra obtained with a telescope?"

First, this convention deals with binary tables only.  It does NOT
preclude multiple binary tables in a single FITS file.  In general,
any FITS file may have multiple extensions, some of which may be
binary tables, some of which may follow this convention.  So, you
are always free to divide up your data into multiple tables if that
makes more sense (either by making it easier on the writter, or
because it logically makes sense to divide the data, or for whatever
reason).

Having said that, each row of the binary table will contain one
multi-dimensional matrix that holds the data (spectra for spectral-line
observations).  This may in fact be more than one spectra so long as
we can agree on what the additional dimensions are (for example, one
might put data taken on a regular grid in sky position [i.e. a spectral
line cube]).  Clearly we need to talk about what axes are allowed.

Yes, this probably does mean that if you are writing the data
in real time that you need to write to disk and not to tape.  I think
this is a problem for the folks doing the writing and not a concern of this
convention.

From your list of possible answers to your question, I clearly prefer
(1).  I would not write a new file for each observation (your number 2).
Since this convention is strictly concerning binary tables, you could
write each observation to its own table (your number 3) and append the 
tables to one file but I think that's just as much a data management 
problem as writing out several files.  Your (4) is therefor the same
as (3).

" A common observing mode at Parkes (and I assume other instruments)
is to observe two transitions simultaneously (e.g. megamaser
observations at HI 1420 MHz and OH 1665 MHz).  Unfortunately many of
the FITS header keywords suggested (e.g. TSYS, RESTFREQ, etc) would be
good for only one of these bands, and so it would not be appropriate to
put them in the FITS header."

A keyword in a FITS binary table should be thought of as a column name
where the value for that column is the same for each row of the column.
If the keyword in question may vary in the table, it should be
a column.  So, if TSYS varies from row to row (which it will), it
should be a column, etc, etc.

If it is import to know what band the spectra came from, you should
have some telescope dependent keyword that indicates that information.
Then the same TSYS, RESTFREQ, etc, keywords could work for each row,
independent of which spectral band the data came from.  Note that
above I said that you could use the multi-dimensional array to hold
several spectra.  If, however, you want to specify a different TSYS
for each spectra, you should break that multi-dimemensional array
up into individual spectra.  

We need to make the job of reading this data as simple as we can and
it should require NO telescope specific information to read the
understand the basics about the data.  So, there should be a
single TSYS column, a single RESTFREQ column, etc.

You might alternatively want to break the table up into a table for
each band.  But I don't think its necessary at all.

"A related problem is that of multi-beam feeds (e.g. the 13-beam HI
feed now being built for Parkes). Again, we need a different Tsys etc
for each beam. Again, maybe a separate table for each beam is needed."

Again, while you might choose separate tables, I don't think its
necessary.  You may choose to put some telescope specific column indicating
the feed for each row.  But as long as the fundamental information
such as the sky-position, frequency, etc, for each row is correct,
you should be able to put all of it one table.

"Another related problem is the use of arrays of different lengths,
which others (e.g. Bob) have already noted."

I've come to the conclusion that we simply can't ignore the proposed
variable length array convention and we should allow for it in 
this convention.

Now, I have a request I'ld like to make of everyone.  Last Jan. 31st
Bob Payne reported that the GBT has decided to use the preliminary
table coordinates proposal for describing coordinates in their
binary tables.  This proposal differs from the proposal adopted at
Green Bank in 1989 for single dish data.  The proposal is available
via ftp at :

Preliminary Table Coordinates proposal
ftp://legacy.gsfc.nasa.gov/fits_info/ofwg_recomm/coord_keywords.ps

I would like everyone to read that proposal.  I have had some conversations
with a couple of people here who do not like the proposal, especially
as it relates to our needs.  I personally think it makes life unnecessarily
complicated for readers of the proposed single dish FITS convention.
I think the decision on whether to follow the original Green Bank
coordinates convention or the above proposal should be the first
decision we should try and reach.

Cheers,
   Bob

---

From: [email-masked] (Ray Norris)
Date: Thu, 16 Mar 95 21:06:32 EST

Hi Bob

Thanks for your answer. there's nothing you say with which I
fundamentally disagree. Certainly I agree with your philosphy, although
perhaps I would add the proviso that we should be careful to remain
pragmatic - the data have to be handled by real people and real
machines. So when you say

> this is a problem for the folks doing the writing and not a concern of this

then I think I disagree in principle - this *should* be our concern.
The original FITS convention said that you had to be able to write to
tape in real time. I for one am happy to abandon that, as disks are now
cheap and fast, and transfer to tape (or CDROM, or whatever) can take
place off-line, but I think we should make such changes consciously and
not merely implicitly, or by default. In this case the technology has
changed so this old proviso is no longer appropriate. In this
particular case, maybe I'm being a little pedantic, but I think as a
matter of principle we should think through all the practical
implications of wehat we agree on.

> problem as writing out several files.  Your (4) is therefor the same

Incidentally, maybe I didn't make myself clear in (3) - I meant that
some groups write the whole thing (main FITS header and all) out again
and again - standard FITS readers would barf at that so I don't think
it's to be advocated, but some groups do it!

The rest of what you say I agree with. If we treat each keyword as a
column header, and allow variable-length arrays, then all my problems
are solved.

Cheers
ray

---

From: "Harvey Liszt,222,344,9733744" <[email-masked]>
Date: Thu, 16 Mar 1995 09:21:43 -0500 (EST)

Ray--

I've been lurking, reading the dialog between you and Bob Garwood recently.

With regard to the matter of formatting data, I second Bob's comments.
I think data shouldn't be hidden; writing one row per spectrum is the
best thing conceptually and practically.  The rows of one table may have
to have the same structure but within that constraint, there can be quite
big differences from row to row.  There's no barrier to mixing OH and CO
spectra in the same table, array length aside.

With regard to the real-time problem, I think it is appropriate to think
of a "fix" to FITS which accomodates this better, although I think that
solutions are possible even without a formal fix.  I'm thinking of other
instances where formats that looked off-line-ish and finished and writable
a priori got such fixes.  The one I deal with most commonly is Postscript.

This is supposed to have a bounding box in the header; because it isn't
known ahead of time, most writers just put the size of a whole page and then
go on to cover whatever space they need; this can make it difficult to merge 
postscript illustrations.  Another problem is speed, because resources are 
used and discarded when they might better be cached.  To solve the problem,
one can now put in an afterword at the end of the file which lists the
fonts and resources and the actual space.  Some readers can use this info
but it isn't absolutely necessary.  This works typically because one deals
with disk files, but it is a response to exactly the same phenomenon you
have addressed.

FITS is a slightly different case because it is so highly structured.  It
seems to me that in many (most?) cases, the actual dimensions along all n axes
don't need to be contained in the header; only n-1.  Given the fixed size of
each instance, the size of the file alone determines the dimension along the
last axis.  Or alternatively one can simply read until the final END statment
is encountered or an EOF.

Does this make sense?  It seems to me that either a reader could purposefully
ignore the supposed value in the header for the number of rows in a table or
in a map, or the naxisn[naxes] could be set = * in the header to tell a reader
to read exhaustively, not up to a fixed index.  This applies to tape more than
to a disk file; if the size of a file is known, the naxisn value along the last
axis can be figured out even if it is not in the header.   We've grown too
accustomed to having the FITS header spell everything out for us; remember the
old days of writing programs that read in a number of data points that _wasn't_
known ahead of time?

regards, 

Harvey Liszt

----- End Included Message -----