Subject: Re: latest proposal -- some comments and questions

From: [email-masked] (Don Wells)
Date: Tue, 17 Oct 89 10:09:27 EDT

Friends, Bob Hanisch sent me this last night:

BH>Date: Mon, 16 Oct 89 17:51:42 EST
BH>From: (Bob Hanisch) <[email-masked]>
BH>Subject: latest proposal -- some comments and questions
BH>
BH>Hi Don.  Have been reading through your proposal that came out on DISH-FITS
BH>today, and am puzzled by a few things.
BH>
BH>First, I guess this is not really a puzzle, but just a comment.  At various
BH>times you equivalence columns in a binary table to keywords, and vice versa.
BH>In Section 3.1, a keyword is taken as a virtual column.  And in Section 3.5
BH>columns in the table are consider as virtual keywords.  I don't really see
BH>any difficulty here, except perhaps that this equivalence, or maybe a better
BH>term would be `correspondence', be discussed more explicitly.  

You are right. These two sections should be merged in the next revision of
the document (I did mark it as "{\em DRAFT}" and gave the date as "\today").
Yes, "correspondence" is the right word.

BH>There is a
BH>good analogy in SDAS format data, and for that matter, in FITS group format.
BH>There are _global_keywords_, which appear in the main header, and _subset_
BH>specific_ keywords, which appear encoded in the group parameter block.  The
BH>STSDAS/IRAF system regards both as keywords, period, and values are accessed
BH>via the name of the keyword.  It's just that the latter keywords are multi-
BH>valued within the same data file, whereas the former are fixed.
BH>
BH>I don't understand Section 3.3, specifically, "the order be reversed so that
BH>the last substring, the keyword, will be in the first 8 characters."  I could
BH>not find any examples of what you meant, but if I take you at your word then
BH>the standard header for a FITS image might have something like this
BH>
BH>WHATEVER= value NRAO-12M GENERAL SINGLDSH / here is some comment info
BH>
BH>while the same thing, written in an extension header, would be
BH>
BH>SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info
BH>
BH>Is this what you mean?  I'll reserve comments until I understand your intent.

Sorry for the confusion -- I will have to find better words in the next
draft, and add examples. What I meant was that in any header, main or
extension, we would have (in the HIERARCH notation)

HIERARCH SINGLDSH GENERAL NRAO-12M WHATEVER= value / here is some comment info

but that if the same keyword is a *column heading* in a 3-D table, then it
should appear as

TTYPE13 = 'WHATEVER NRAO-12M GENERAL SINGLDSH HIERARCH' / some comment info

There are two reasons for this suggestion: (1) the classic rule that string
values be unique/useful in first 8 characters and (2) implementors of
routines to display tables will want to be able to truncate the label
strings.

BH>On the more general question of using binary tables for single-dish data,
BH>I don't really have any reservations that this is indeed the most efficient
BH>way to go.  It is logically straightforward, relatively simple to implement,
BH>and clearly the most efficient thing to do within the context of FITS.  As
BH>has come up in other discussions, however, most notably in Ann Arbor, I think
BH>we need to generalize binary tables to include multi-dimensional arrays
BH>within the structure.  This seems to me to be such an obvious extension,
BH>I'm a bit surprised that Bill did not encounter it in dealing with VLA
BH>and VLBA data.

OK, OK, because I wanted to sell this idea to spectroscopists (1-D) I didn't
tell them how I intended to argue in the task force session where this would
be settled. I really think that, consistent with 10 years of FITS
traditions, we should do an N-dimensional solution if we can. In my section
5 where I use TTYPEnn values of 'NCHAN' and 'SPECTRUM', please read 'NAXIS',
'NAXIS1' and 'MATRIX', except that they would really appear as:

HIERARCH SINGLDSH GENERAL NAXIS = 1		/ 1-D spectrum (virtual column)
TTYPE5  = 'NAXIS1  SINGLDSH GENERAL HIERARCH'	/ (NCHAN)
TTYPE6  = 'MATRIX  SINGLDSH GENERAL HIERARCH'	/ (SPECTRUM)

I trust that the generalizations are obvious. An interesting side-effect of
this concept is that because the maximum product of the dimensions is set by
TFORM6 = '1024F', if NAXIS is 2 or greater this design allows matricies of
variable aspect ratio in the table! Note that value strings should be padded
with blanks in the first eight columns so that the first classification code
substring appears in either column 9 (as shown above) or in column 10 (if
the keyword is 8 characters long).

BH>The thing I was confused about in your discussion of binary tables, however,
BH>was in Section 3.2 - Multiple Feeds.  Aren't you overlooking being able to
BH>use the prime advantage of binary tables, i.e., table columns that are arrays,
BH>by saying that multiple feed data or multi-band data should go into separate
BH>structures?  I mean, if I have some feed- or band- dependent keywords, which
BH>I wish to store in a binary table, can't I define keyword columns of depth
BH>equal to the number of feeds or bands?  In a simple example, suppose I have
BH>four feeds, and one feed-dependent keyword.  Then the binary table could look
BH>like
BH>
BH>keyword(4I)	feed1(100E)	feed2(100E)	feed3(100E)	feed4(100E)
BH>
BH>Element 1 of the keyword array corresponds to feed1, element 2 to feed 2, etc.
BH>I imagine this may violate some basic tenet of relational databases, right?
BH>This is not an area of my expertise.  Of course, another thing I'd like to do
BH>is write this as
BH>
BH>keyword(4I)	feed(4x100E)
BH>
BH>which seems a lot cleaner.  Actually, probably want (100Ex4).

Yes, what you say is the alternative concept. Because I am inexperienced in
relational database design I hesitate to use RDBMS arguments one way or the
other. I merely think that it will be easier to build the spectral analysis
programs and easier to negotiate in the task force if we instead put the
four feeds in four separate rows of the table. I.e., I think that both
design and negotiation will be easier if there is only one 'MATRIX' column
in a table.  Anyway, if you take the "correspondence" principle seriously,
you would be arguing for multi-valued keywords, which the reviewers forced
us to give up in May 1979.

BH>Anyway, I don't know if I'm the only one confused or not.  One thing we clearly
BH>agree about is that the hierarchical keyword situation needs to be resolved
BH>regardless of how the data is actually structured.  And by the way, don't
BH>misconstrue my comments in my memo about the stack approach to hierarchies,
BH>that is about whether we need hierarchies at all.  I think we do; all I wanted
BH>to do was get people to agree to this, and present the arguments I could think
BH>of against it.

I understood that and, as I said yesterday, I thought you did the community
a service by raising the issue. I too hope that they will agree with us that
we simply must control our namespace if we want to transmit meanings
reliably.

BH>Should be a fun meeting.
BH>
BH>Cheers, Bob

Regards, Don