Subject: Re: HDF as an archive format

From: sla at umbra.UCSC.EDU (Steve Allen)
Date: 5 Aug 1993 19:38:15 GMT

In article <5AUG199310442680 at nssdca.gsfc.nasa.gov>
bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER) writes:
>First of all, a general comment.  All FITS formats endorsed by the IAU
>have been published in the refereed literature...

Which is a good thing.  Anything that goes in to a FITS file will always
be possible to get out.  I use FITS as a part of my work, and as a
data storage mechanism for my thesis data.  Some of my thesis data
are synthetic.  They are model matrices with appropriate amounts of
structured information to describe what each element of the model is,
how it was generated, etc.

>The basic goal of a FITS software reader is to generate the
>appropriate table or array....

This sentence exposes one of the worst problems with FITS
	"tabel or array"
FITS is great for encoding any kind of multidimensional rectangular
array.  It is not so great for other entities.

>       ...           So FITS readers should be able to create
>correctly structured binary tables from any BINTABLE FITS file.

I have no argument here.  Again, anything that goes *in* to a FITS
file will always be possible to get out.  The problem is, that some
kinds of data are very difficult to get *in* to a FITS file.

I mean, specifically, data structures, linked lists, and other sorts of
documentary information which would be associated with an image.  It is
possible now, with the BINTABLE extension, to encode these sorts of things
in a FITS file.  BINTABLE provides all the necessary pointers and
data HEAPs.

However, there really are no tools for converting a generic data structure
into this format.  Likewise, there are no tools for guaranteeing that the
data can be extracted into a reader which will reconstruct the original
structures.  Bill Pence's FITSIO toolkit is perfect for dealing with the
low-level operations on FITS files, and I use it.  Being written in standard
Fortran 77, however, means that it cannot provide the routines necessary
to comprehend, encode, and decode structures of data.

Using FITSIO, I have created a number of tools of this sort for my thesis
data, but they are in no way standardized.  I would not expect that
someone else's FITS reader would have a chance of actually using my
FITS files to do the work they were written to do.  Basically, then,
I have invented tools with some of the features of HDF, but which are
based upon an "archival" underlying format.

If HDF makes doing the work easier, then that is a significant advantage
for it which FITS does not have.  This is especially true for graduate
students or other sorts of projects which do not have the time to
invent some kind of wrapper code that successfuly stores and retrieves
their data structures.  Bill Pence's FITSIO routines are good, but
FITS would benefit even more by having a much higher-level interface
in a language which comprehends the kinds of non-rectangular data
structures which have always been around but which are awkward to
manipulate in FORTRAN.
_______________________________________________________________________________
Steve Allen          | That was the equation!         |   sla at lick.ucsc.edu
UCO/Lick Observatory | Existence!...Survival must     | If the UC were opining,
Santa Cruz, CA 95064 | cancel out programming! -- Ruk | it wouldn't use me.

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 6 Aug 1993 11:43 EDT

In article <23rnj7INNo7d at darkstar.UCSC.EDU>, sla at umbra.UCSC.EDU (Steve Allen) writes...

>documentary information which would be associated with an image. 

Tables and arrays are the only standard extensions and proposals under 
consideration at present.  But the field is open for more development.

				Barry Schlesinger
				NSSDC/NOST FITS Support Office