Subject: Thoughts from WGAS -- IMAGE extension and data structures

From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock)
Date: Fri, 19 Jun 1992 19:22:38 GMT

Here are a couple of related issues which I got to thinking about at the
WGAS meeting in Columbus last week.

I want to spur a discussion on namings, in connection with the proposed
IMAGE extension.  While I don't have any particular problems with the
proposal, as such, there is an issue I feel needs to be addressed by the
FITS community, so we can achieve some sense of direction for the
future.

The word "IMAGE", as commonly used by astronomers, and as used in the
IMAGE extension proposal actually has two meanings.  The first is the
interpretation of the word "IMAGE" as a high-level object, i.e.,
something which the scientist (or whoever) views, prints, processes,
whatever.  The second meaning is that of the underlying data structure,
in which an image is stored as a 2-D raster, implied by the data
representation described by the original FITS document and the IMAGE
extension proposal.

What I'm aiming for here is to point out that we place an undue burden
on the end-user by confounding the two meanings.  There is a natural
division between data structures (which machines have to know about and
end-users don't), and data interpretations (which end-users have to know
about and machines don't).

At some point, we (the astronomical community in general, and the FITS
community in particular) need to address this issue.  It would even be
acceptable, though probably not desirable, to say that we understand the
issue and will explicitly ignore it.  But I think as FITS becomes more
and more of a working format (via Pence's FITSIO and FTOOLS packages,
for example), it becomes a crucial issue.

Right now, it's simply the {\em name} IMAGE which bothers me.  To the
end-user with some computer science knowledge, there are any number of
ways images can be stored besides the conventional pixel-by-pixel,
row-by-row.  Block storage, for example, can be efficient and fast.
Yet, if we approve this extension with this particular name, no other
data structure can ever have the name IMAGE, though images they may be.

This imparts a burden on the end-user which can easily be off-loaded to
the machine if we can develop a scheme to name underlying data
structures independently from the high-level data interpretations.  The
result of such a scheme would be to allow end-users to deal with things
they are familiar with (images, spectra, tables,..), without having to
know about the specifics of data storage and structure.

Let me give a brief example, from outside FITS, to illustrate the point
a bit better.  In the course of ingesting the International Halley Watch
(IHW) data into the Planetary Data System (PDS), we had to develop a
translation program to generate PDS-style labels from the FITS headers
which IHW originally supplied.  We found that there were spectra in the
IHW archive, from different disciplines, which had been stored in a
variety of ways - as ordered pairs (wavelength, flux), as pairs of
vectors (all wavelengths, then all fluxes), as a starting wavelength, an
increment and a vector of fluxes, even a 2-D "image" (an echelle).  All
of these use different data structures (within FITS!) to represent a
single high-level object - a spectrum.

Our initial contacts with the PDS had them suggesting that each of these
things have a unique name, thus requiring the end-user to know all of
the possible names a spectrum might have.  Our response was that the PDS
label syntax allowed for a nesting of these "object" definitions, and
that a single high-level object called "SPECTRUM" would suffice,
providing that the underlying data structures could be appropriately
named.  In this way, the software developer would be able to supply the
I/O routines to read/write the data structures without having to know
that the user was calling them spectra (and saving code - an echelle
could be read with the same code that read an IMAGE).  In addition, the
end user doesn't have to know what data structures are being used -
he/she gets to read and write "SPECTRUM" objects.

Prior to this proposed extension, there really wasn't an instance where
such confusion could arise.  The "official" extensions were pretty much
self-descriptive.  Data which occurred in the primary data stream wasn't
identified as to interpretation at all.  The current IMAGE extension
proposal is the first one to really blur that distinction, and I feel
the issue should be addressed in one way or another.

Do I have a proposal to make which will encode this kind of distinction?
Nope - at least, not yet.

Second, I'd like to point out the related issue, also raised by Immanuel
Freedman, that we'd better start thinking about fitting advanced data
structures into FITS.  If we're going to use FITS as a working format,
rather than just an interchange format, issues of efficiency become
important.  We will {\em need} to be able to put things like quadtrees
and block-stored images into FITS.  We will need to be able to store
related structures - like keeping palettes with images - and to do so in
a way which makes explicit the relations.  Just putting a bunch of
extensions into the same file doesn't do that.

The floor is open for discussion...

----------------------------------------------------------------------------
-- Archie Warnock                   Internet:  warnock at hypatia.gsfc.nasa.gov
-- Hughes STX                       "Unix --- JCL For The 90s"
-- NASA/GSFC                        

-- 
----------------------------------------------------------------------------
-- Archie Warnock                   Internet:  warnock at hypatia.gsfc.nasa.gov
-- Hughes STX                       "Unix --- JCL For The 90s"
-- NASA/GSFC