Subject: Too late may be ?

From: "GUILLOTE@FRGAG51" <[email-masked]>
Date: Sat, 28 Oct 89 00:02:22 EDT

Date sent:  28-OCT-1989 04:54:46.17
To:   DISHFITS@NRAO

            Dear FITS designers,
 
As I won't be able to attend the meeting, I sum up here in a rather
detail way my thoughts about the various propositions which came to
me. I am sorry to be so late. Hope you can get it before leaving,
otherwise Thierry will distribute it.
 
            Have a nice meeting !

            Stephane Guilloteau
            Institut de Radio Astronomie Millimetrique
            300 Rue de la Piscine
                F-38406 Saint Martin D'Heres
 
Phone       76 82 49 43
E-Mail      [email-masked]

----------------------------------------------------------------------
 
1) Reply to D.Wells proposal for 3-D tables
 
1.1) Comment on the general philosophy
 
The 3-D tables idea is quite nice. It is a generalisation of the
"random group" format that Thierry and I were proposing, and is
essentially more flexible. This is both an advantage
and a potential cause of problems.
 
------------------------------------------------------------------------
| The main difficulty with FITS has always be Reading FITS, not        |
| Writing. This was clearly realized by the original designers         |
| of FITS. It resulted in several compromises such as the              |
| fixed format and rigid alignments of keyword values. Also            |
| in the original papers, it was recommended that                      |
|       "transmitting institutions should write the tapes, whenever    |
|        possible, in a way which will aid the receiving institution"  |
|               (AA Sup p.367, middle of Section 7.)                   |
| and                                                                  |
|       "the user should always try to keep the requirements           |
|       and restrictions of the reading institution in mind"           |
|               (AA Sup p.372, end of section 4.)                      |
------------------------------------------------------------------------
 
With the 3-D tables, and with the idea of correspondence between
Keywords and Columns ("virtual columns" and "virtual keywords"),
the flexibility is even enhanced and these recommendations should
be further enforced.
 
One thing we MUST forbid is the use of pointers within the 3-D table.
Pointers would break the correspondence mentionned above.
 
And of course, the 3-D table should not be exclusive of more
usual random groups.
 
1.2) Hierarchical keywords
 
The correspondence principle is somewhat in conflict with the "HKBEGIN - HKEND"
idea for hierarchical keywords. This is because the "virtual
keywords" appear only as argument to keyword TTYPEnn keywords.
Since the "HKBEGIN - HKEND" idea implies an ordering
and (to keep it simple) a continuity also of hierarchical keywords.
The simplest continuity (only) is broken for TTYPEnn arguments.
Only a more subtle form of continuity is still valid.

1.3) Some misunderstandings
 
In section 3.2, the meaning of "structure" is not clear. Does it mean
"line" (by opposition to "columns") ?
 
1.4) Some absent issues
 
As Thierry and I already discussed in one of our E-Mail, an important problem
is that of non-regularly sampled frequency data. I suggest that
the frequency (or wavelength) sampling of the various backends
(if more than one) be handled in a separate 3-D table extension,
placed BEFORE the data table(s).
 
I do not believe resampling is the general solution for
that (a related comment by Bob Wilson), although it should be used
if possible.
 
1.5) Implementation constraints
 
To keep it simple, I) the number of columns should remain relatively
small, and II) the columns should concern only rather "standard"
keywords. A possible way in this direction is to define two
types of 3-D Tables: one for raw data and one for reduced data.
 
For reduced data, it is possible to handle virtually any observing
project in a single 3-D table, since only few parameters are
actually required.
 
For raw data, a 3-D table should contain only data for which time
dependant parameters can be assumed constant (e.g. calibration,
pointing corrections, etc...). Otherwise, those parameters
would have to placed in columns, most likely violating
either I) or II) (telescope dependent parameters).
 
A third level might be considered : archive. In this case anything
could go into the binary table, but probably only the writing
program will be able to read  back such complex data.

1.6) Reading constraints
 
The FITS reading programs which will support 3-D Tables should
also support the more usual "Random Groups", since it is in fact
a (rigid) subset of the 3-D table capabilities. No extra code
should even be necessary.
 
2) Preben Grosbol comment on Hierarchical keywords
 
2.1) I do not fully understand the point
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
     It seems that the blocking factor extension violate this
 rule? No?
 
2.2) Array keywords
 
Continuation cards should be implemented using the last non-comment
character. This is the only way to know a priori that the data
is not complete, so that further reading is required. Simple
string contatenation can then be used, with the final
READ using a (now standard) list-directed format.
 
All other schemes imply either to read keywords in advance, or
to count the number of arguments.
 
The full dimension of the array should be specified in the keyword.
Sub-array notations (ARRAY(2..4) or ARRAY(2:4)) are ambiguous or
require a distinct keyword to specify the array size. They should probably
be prohibited.
 
2.3) Hanisch proposal revisited
 
As mentionned in section 1.2, it is somewhat conflicting with
the "correspondence" idea. A point to clear up...

3) Ray Norris proposal
 
3.1) Is the GCOUNT=-1 idea in conflict with the
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
requirement ?
 
3.2) Turning back to our previous discussion on FITS for
real-time logging, I still believe this is the true problem has
not been considered. The key is Random Access Buffer Space  (RABS)
(either disk or memory), not only data throughput. If sufficient RABS
is available, FITS file headers can be updated for the current
size before writing to a sequential device.

4) Bob Wilson comments
 
4.1) multiple spectra :
This is converging with the Ray Norris proposal. It also points out
a possible difficulty in the 3-D tables. If many spectra of
variable length are written in a single 3-D table, extracting
just one may prove costly, since there is no way to predict
the position of the spectrum in the table (other than reading
part of the table). Please NEVER use pointers to circumvent
that small problem: this would lead to lot of confusion (cf 1.1).
 
4.2) "Core" keywords for reduced data:
I fully agree with this idea. It was behind the proposal 1.5 for
two (or three) levels of 3-D tables.
 
4.3) "Scan" numbers :
With the advent of multi-beam and multi-backends, this concept
is obsolete. Single-dish data reduction should tend towards
a descriptive approach of the data. The GAG-IRAM program
CLASS is an (uncomplete) attempt in this direction.

5) Pat Wallace comments:
 
5.1) Pointing corrections
Yes, these are really telescope specific parameters, and nothing
else. They should not even be mentionned for reduced data (see 4.2
and 1.5).
 
5.2) Source coordinates:
I have in my FITS papers a draft by Bob Hanisch and Don Wells,
entitled "World Coordinate Systems Representations Within the FITS Format".
It describes in a rather complete way conventions to be used for standard
geometries, equinox, etc... What is the status of that draft ?
 
I only have one very specific request. Drifts in horizontal coordinates
are often used for pointing determination in radio astronomy. That
king of data hardly conveys any useful astronomical content, but
is quite important. Would it be possible to add a -HOR projection
to the list of standard ones, implying that the coordinates
are horizontal offsets from the tangent point at the time of
observation? Or is that too specific?
 
6) Revised Stobie and Morgan memo
 
Even though it seems that many people have diverged from this proposal,
some comments might still be useful
 
6.1) p 22 ASCII versus Binary : Binary only please...
 
6.2) p 22 TABLE extension versatility for multi-beam or multi-backend
The real issues are:
- when will the processing systems be ready for multi-beam multi-backend?
- how do they handle a multi-something: as a whole or not?
- if not, where should be the separation ?
            - in real-time ?
            - in FITS writing ?
            - in FITS reading ?
            - in processing ?
 
6.3) p 22 Hierarchical approach :
Lack of space may become a problem if
            HIERARCHY SINGLDSH ANALYSIS Program Structure Variable
is required... Redundant keywords must be suppressed (e.g.
SINGLDSH with ANALYSIS or ANALYSIS with Program).
 
6.4) p 22 Units
Only SI units please. By the way, why are Degrees used in FITS for angles ?
Radians are the only consistent choice with SI units...
 
6.5) p 22 Tape layout:
This is against the basic FITS rule. Blocking factors are possible without
specifying the layout.
 
6.6) P 22 Missing Keywords:
That is right. Missing keywords should not be set to zero. They should
be omitted when writing, and set to Not-A-Number when reading.
 
6.7) P 22 Keyword repetition:
Not repeating the keyword imply to read and decode all previous extensions
even when only one is wanted. It is a waste of time. The 3-D table
extension more or less suppress this problem if the grouping of data
is done properly.
 
6.8) p 23 Ill defined keywords:
Those ones were redundant. Suppress them. See also BADCHV,
redundant with the usual BLANK (p 27)
 
6.9) p 23 Pointing parameters:
Observatory (or even telescope) dependent set. See 5.1
 
6.10) p 23 Observing parameters and scan numbers:
See 4.3. Each observtory and each analysis program has its own definition
of this number (not even consistent, see IRAM 30-m and CLASS!).
It will be hard to adopt any convention.
 
6.11) p 23 DESORG and array keywords:
We should play a leading role in array keywords.
 
6.12) p 24 Map parameters:
Random groups or 3-D binary Tables do that even better, since
no assumption is made on the map sampling.
 
6.13) p 24 PHASE table:
See 6.11 on array keywords.

7) Three further comments
 
7.1) What will happen to the DD/MM/YY date format in January 2000 ?
 
7.2) How should we describe the various velocity reference systems
and related topics in the axis definitions ?
(Frequency, Wavelength, Velocity, Redshift, LSR, Heliocentric,
Geocentric, Radio versus Optical definitions)
 
7.3) Since implementations of 3-D tables and hierarchical keywords
will be a rather major issue in any case, should we break the
"It MUST be ensured that additions to FITS does not generate
 wrong results even for old readers"
rule by specifying a
 
SIMPLE =          F     /No "standard" FITS
FORMAT = 'HIERARCHY'    /But correspond the 'HIERARCHY' revision instead
 
and all further keywords and tape organisation using the full flexibility of
Bob Hanish hierarchical proposal, array notations,
full list-directed input format, 3-D tables, GCOUNT=-1 ?
 
Wouldn't it be simpler (in terms of programming resources) for most people?