Subject: Comments from the AT

From: "Ray Norris, Australia Telescope, CSIRO, Australia" <NSSDCA::PSI%RPEPPING::[email-masked]>
Date: Fri, 27 Oct 89 02:11:12 EDT

Comments on Single Dish FITS formats and requirements for observing

Ray P. Norris

Australia Telescope National Facility

1 INTRODUCTION

Since the Australia Telescope (AT) can't send anyone to the Greenbank meeting,
this document represents our submission to the meeting. There are three parts 
to this submission: 

(i) Comments on the structure of a future single-dish FITS standard
(ii) An overview of what we have and where we're going
(iii) An Appendix proposing changes to the FITS standard to make it 
more suitable for real-time use.

2 COMMENTS ON FUTURE SINGLE DISH FITS STANDARDS

We would like to change most of our internal data formats to FITS, for
all the usual reasons. To be able to do so, we require that

(i) The FITS standard is extended to make it more suitable for real-time
data logging (see my comments in the Appendix to this submission). In 
particular for spectral line use we need to be able to write many spectra
into a file without knowing how many there are to come (see comments by
Norris and by Wilson).

(ii) We need to put in a lot of site-specific information, and so we would
strongly support the concept of hierarchical keywords. The most promising
format for this is option (2) in Preben Grosbol's recent Email (the mix of
Grosbol, Wells, and Hanisch proposals).

(iii) The idea of binary 3-D tables seems excellent, and we would support the
use of IEEE standards for these.

3 THE AT: WHAT WE HAVE AND WHERE WE'RE GOING

The AT consists of (a) the 6km compact array at Culgoora, which is a 6 x 22m
East-West synthesis array with operation planned up to 115 GHz, (b) the
Parkes 64-m antenna which operates up to 43 GHz, (c) the 22-m Mopra (Siding
Spring) antenna which will operate up to 115 GHz, (d) the Long-Baseline
Array which consists of the antennas above plus others used collaboratively
with other Australian groups to produce baselines upto 3000 km. 

Here we are concerned only with single-dish observations, and so restrict
attention to the single-dish use of the Parkes and Mopra antennas.

Parkes is used for a broad range of single-dish experiments, including spectral
line studies, polarisation (largely using the Bonn polarimeter), continuum
surveys, and pulsar work. We also hope to install the NRAO multi-beam feed on
Parkes in early 1990. At present most of the software differs radically from
one observing mode to another, and there is little in common either in the
software or in the data format. Mopra is not yet operational, but will become
operational over the next few months, and will be equipped with an 8000-channel
correlator for spectral-line use. 

3.1 SINGLE-DISH SPECTRAL LINE OBSERVATIONS

At present spectral line work at Parkes makes use of the SPECTRA observing
program and the 1024-channel correlator. Data output is in a home-grown SPC
format, which many would like to see replaced by FITS. Spectral-line data
reduction is generally done using a 'temporary' program (S) written by Rick
Forster some years ago but which has evolved into the default spectral line
reduction package. 

We plan to replace the 1024-channel correlator shortly by a 8000-channel VLSI
correlator (identical to the one for Mopra) and this would be a good
opportunity to overhaul the SPECTRA observing program. It is expected that the
observing software at Parkes and Mopra will be largely identical (except, of
course, for the antenna control software) and so we plan to install a version
of SPECTRA at Mopra. 

Whilst many users have come to know and love the S data reduction program, it
is felt that a more sophisticated program would be useful, and so we
are currently flirting with other programs such as SLAP.

3.2 RESPONSE TO QUESTIONS IN ORIGINAL CIRCULAR

In the original circular about the Greenbank meeting, a number of questions
were asked of each observatory. Here are our answers to those questions.
Many of these answers reflect the fact that we are currently undergoing
a transition from an in-house observatory/astronomical institute to a
national facility, and as such we expect to provide a higher level of 
support than we have in the past.

(i) What calibration/reduction/analysis software do you have?
We have a variety of observing software, most of which is not suitable for 
export. Similarly, most of our data reduction software is site-specific and
not suitable for export. A notable exception is AIPS which is heavily used
for AT, VLA, and VLBI data. We would welcome, and probably participate in,
any initiative to draw on the experience gained with AIPS and other packages 
to produce a real user-friendly single-dish spectral-line reduction program.

(ii) How is your software effort organised in-house?
While the bulk of our software effort is our Epping headquarters site,
much of the on-line software has been written either by users or by staff
at the Parkes and Culgoora observatories. Whilst much of this
software has been immensely succesful, there exists no coherent plan for its
development and this is one area that we would like to put some planning into.

(iii)How do you support your users?
 At present we have real-time support for users, plus a limited
amount of data reduction support. This user support will be increased to
a cradle-to-grave philosophy in the future.

(iv) What are your upgrade paths?
Hardware: at present we have VAXes providing real-time computing, and that will
continue for the immediate future. Off-line, our main processor is a Convex 
C-220 together with a number of Sun workstations, SPARCstations, etc.
We expect the use of these workstations to increase, and we would like to see 
more use made of distributed computing. We are also likely to
acquire a high-end workstation (e.g. a Sun 4/370 with TAAC board) for
algorithm and visualisation development. On the other hand, use of X-windows
means that we may make more use of dumb terminals like X-terminals, Macs,
and PC's in the future.

Software: We have grave concerns over the future of our off-line support,
and believe that community-wide development is the only way in which 
sufficient manpower can be brought to bear on the problem of providing
the level of software that we want and expect. Organising and providing
this development is likely to be extremely difficult, however. Meetings
such as this Greenbank meeting are a good first step.

(v) How do you view shared code?
As above - this is the best way forward for data reduction. 

(vi) Could single-dish work make better use of AIPS/IRAF/etc.?
It could, but only if manpower is available to include within
these packages the functions that single-dish users need.

APPENDIX: USE OF FITS FOR REAL-TIME DATA LOGGING.

I've already distributed my comments about the need for modifications to the
FITS standard to make it suitable for real-time logging, and I won't
repeat the detailed arguments here. Briefly, however, the argument runs 
as follows:

FITS was not designed for on-line use, and isn't, at present, suitable for
on-line use. However, with some small extensions, it could be made suitable for
on-line use. The lack of a good on-line format has resulted in a proliferation
of internal ad hoc formats around the world, with a resulting waste of manpower
and resources in re-inventing various site-specific wheels. Each site then
develops software for transcribing data from its internal format to FITS (more
manpower!) and then observing assistants spend time transcribing tapes. This
will become more of a problem as we move towards using common data reduction
software in several sites. 

Perhaps more importantly, as multi-beam feeds and large correlators increase
the volume of data (which could result in peak data rates of 100 kbyte/s in the
near future), the resources needed to transcribe data approach those needed to
take the data in the first place. Transcribing data then becomes a full-time
task rather than a little job to be done before you back up the data, and could
become a significant drain on observatory resources. 

I am therefore advocating the extension of FITS to make it suitable for real
time use. To summarise my argument, I contend that (i) to use FITS for on-line
logging makes sense in terms of efficient use of manpower and other resources,
(ii) FITS doesn't let you do this efficiently at present, because FITS requires
you to know in advance how much data you're going to write before you start
writing the file, and in general you don't know this in real-time data logging,
(iii) FITS is fragile if all the tables are put at the end of the data, (iv)
the alternative of writing many small files is cumbersome and causes problems
of data management. 

Point (ii) CAN be overcome with present FITS by some form of buffering, but
that is somewhat inefficient and may get impractical as data rates increase. A
better solution is to allow the option of GCOUNT=-1, meaning that you don't
know how many groups you will be writing, and that any FITS reading program
must in this case check each block for the start of a table, or (if scans are
concatenated in a file, which I believe also to be desirable) the start of a
new header. This extension of the FITS standard has no effect whatsoever on the
speed, efficiency, and implementation of present FITS. Those who don't like it
can ignore it, and write out GCOUNT as at present, EXCEPT that the FITS reading
programs must be extended to accomodate the possibility that they may encounter
data with GCOUNT=-1. 

Point (iii), the problem of fragility, can be overcome by writing headers and
tables within the body of the file, as well as at the start and end. In the AT
data format, each scan of data is preceded by a FITS-like header and tables,
but all scans are contained within the same file. In this way, the damage
caused by a machine crash or parity error is confined to one scan. 

An alternative approach to overcoming fragility has been suggested
by Eric Greisen, who proposes splitting up an observation into
several files - writing a fixed number of records in each tape 
file and then starting a new tape file. This approach is similar to mine
except that I prefer to have all the data in one file.

As a general point, I would propose extending the FITS standard to allow many
scans to be recorded in one file - effectively allowing the concatenation of
many FITS files into one file. This approach (plus the others suggested above)
has been used with great effectiveness for the AT data format.