Subject: the "archival" issue

From: dwells at fits.CV.NRAO.EDU (Don Wells)
Date: Mon, 2 Aug 1993 03:20:02 GMT

The appended posting by Archie Warnock appeared in newsgroup
sci.data.formats two days ago.  I am reposting it to sci.astro.fits
because I suspect that many s.a.f. subscribers do not read s.d.f. and
because I agree with what Archie says here about design considerations
for archival-quality data formats. Glossary: HDF = Hierarchical Data
Format (developed by NCSA at UIUC), API = Application Programming
Interface, CDF = Common Data Format (developed at NASA-GSFC), EOS =
Earth Observation Systems (the "Mission to Planet Earth" project at
GSFC, which has chosen HDF as its archival format).  -Don
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock)
Newsgroups: sci.data.formats
Subject: Re: HDF as an archive format
Date: 30 Jul 93 14:30:30 GMT
Organization: NASA Goddard Space Flight Center -- InterNetNews site

mike at jpl-devvax.jpl.nasa.gov (Mike Tankenson) writes:
>1) Is there something inherent about HDF that would prevent its use as an
>archive format?

Depends on how serious you are about archiving.  The fundamental
question to ask is how will a user, some 20 or 30 years in the future,
upon finding a file in HDF format, access the data.

In my opinion, formats which are defined _solely_ by software toolboxes
and APIs are extremely dangerous for archival storage.  We have no way
of assuring future generations of users that the software will be
usable.  The questions to ask about archival storage are:

1.  Where is the byte-by-byte description of the format given?  In an
internationally recognized standards document?  In the refereed
literature?  In a private publication by some individual institution?
In source code?  Which is most likely to be accessible 50 years from
now?

2.  Is the format self-documenting?  Is the metadata in (at least)
ASCII?  Can the user determine anything about the contents of the data
from a simple dump?

3.  Is the format well-defined?  Can a user assume that versions are
backward compatible?

There are (at least) three different applications for data formats -
interchange, access and archival storage.  It is not at all obvious that
any single format can meet the needs of all 3 simultaneously, even less
obvious that any single format currently _does_ meet the needs of all 3.

In general, working formats need to emphasize access (read/write)
efficiency.  Interchange formats need storage efficiency and
standardized numeric representations.  Archival formats need to be
self-documenting, externally documented and simple.

Things like HDF, netCDF and CDF are good working formats but are risky
as archival formats (at least, I've never seen references to format
descriptions in the refereed literature - I hope I'm wrong, cause
otherwise EOSDIS is gonna be in a lot of trouble years from now).  FITS
and PDS are pretty good interchange and archival formats, but weak
working formats.

It's an important area for discussion...

--
_______________________________________________________________________
-- Archie Warnock              Internet:  warnock at hypatia.gsfc.nasa.gov
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC                   Project STELAR: WAIS to do science
--

  Donald C. Wells           Associate Scientist       dwells at nrao.edu
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road                              Fax= +1-804-296-0278
  Charlottesville, Virginia 22903-2475 USA         78:31.1W, 38:02.2N

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Wed, 11 Aug 93 14:57:18 EDT

Don Wells (2 Aug 1993) quoted Archie Warnock as saying:

>as archival formats. (...) FITS

The last sentence seems questionable since we have been using
FITS as a working format within the HEASARC for the past 2 years with
great success.  I need to give a talk on this subject in a couple
weeks, so I would be very interested to hear other opinions or
explanations on why FITS is not the ideal working format.

-Bill Pence
pence at tetra.gsfc.nasa.gov

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Thu, 12 Aug 1993 10:18:57 +0200 (MET DST)

On Wed, 11 Aug 1993, William Pence wrote:

> >interchange, access and archival storage. [...]
> >

  Essentially the reasons why (IMHO) FITS is not an ideal WORKING format
  are listed below (Bill you alrady know my opinion on this, I'm just
  repeating for the benefit of the list). However this is a matter of
  choice of the particular software package writer/user, and in my
  opinion has a lesser weight than the fact that FITS is at present
  an excellent INTERCHANGE format. 
  I've recently got access to a newsreader and subscribed to sci.data.formats
  and was surprised to find that there are more data formats between heaven
  and earth than are used by us astronomers, but my reaction was "oh not 
  yet another format" .... and in fact I've never seen other than FITS
  used for interchange (besides person-to-person arrangements using native
  formats of IRAF or MIDAS).
  Concerning ARCHIVING I am not so competent to judge, but I gather from
  the previous messages in this thread that FITS is good there too.

                            LONG life to FITS !!!

-----------------------------------------------------------------------------

  Here are the reasons why I choose not to use FITS as working format
  (in this following an historical trend)

  a) FITS uses non-native binary representation, i.e. is more or less 
     slower on non-IEEE machines like VMS, Ultrix (nowadays this might
     be not so important, but one might like to gain that little extra
     speed)
     [My working format uses native representation, and optionally allows
     in place conversion between different operating system formats]

  b) FITS headers are bulkier (because they're ASCII card images, which
     is good for exhcange/archiving) than a possible pure-binary
     representation  [which I use in my working format]

  c) FITS headers are not extensible (this is pointed out also by Chris
     Flatters). In my opinion this is a major snag of a working format if
     one wants to add keywords taking into account a file in-place (rever-
     sible or not) modifications.
     [in my working format I use a file made of three parts : a fixed
     miniheader at start, a data section, and the "header" at the end.
     This because I do not like to keep things separate a' la .imh+.pix]

  d) the FITS 2880 block size is somewhat unnatural (this is totally
     transparent to an user of a package, but may be of some relevance to
     debugging while developing ... I'd like my file to have a "natural"
     record size, e.g. one image or table row  ... so that I can easily
     look at a wrong file with some record oriented tool and find out what
     I did wrong)
     [in my working format I use such "natural" record sizes ... even if
     incidentally the data section on a non-record oriented (Unix), IEEE
     machine will be identical to the data section of a FITS HDU]

  e) Chris Flatters mentioned (as an additional inefficiency to be added
     to point c) the fact that sometimes a FITS file has extra extensions.
     I personally do not like the usage to stick more HDUs in the same
     file for working purposes (bseides the usual SINGLE binary table
     case, of course), while this might be used for interchange (in fact
     most s/w I know unpacks the extensions in separate native files)
     [in my working format each file is standalone, however I conceive
     to stick together some associated files when converting to FITS for
     exchange ... e.g. my original proposal for exchange of a response
     matrix was a primary HDU with the matrix, and an IMAGE extension with
     the associated energy grid]
     Of course this is nothing intrinsic in FITS (just it is TOO flexible), is
     just a matter of taste of software writers.

Actually most of the above is matter of tastes. It was enough for me to
choose a different working format, but definitely not enough to depart
from FITS for interchange (in fact I enforced a number of FITS conventions
and **limitations** in the definition of my format, in order to preserve
easy mapping)

  f) one additional element may be that FITS needs conventions (in order
     to be long-lived), which may not always be suitable to the needs of
     a particular packages. If you choose FITS as working format you have
     either to stick to the conventions, or to propose new ones (the sort
     of things from OGIP seen in the last few months over fitsbits and
     HEAFITS). If these are mainly "insider" conventions, it could be
     easier to arrange things within a native format.
     [e.g. things like  character keywords longer than 68 char, or arrays
     or keywords] 

-----------------------------------------------------------------------------   
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E          
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager           
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE            
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |                                    
Bitnet:   <please do not use any more>     |                                    
-----------------------------------------------------------------------------

---

From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock)
Date: 16 Aug 93 20:55:25 GMT

>explanations on why FITS is not the ideal working format.

First of all, note that I claim only that it's not an ideal working
format, not an unusable working format.  Until the development of FITSIO
(a relatively recent development in the long and illustrious history of
FITS), it _was_ essentially unusable as a working format.

There are still some problems, though.  A number have already been
mentioned.  I'm still waiting for a C interface to FITSIO that doesn't
require a Fortran compiler.  There's no support in FITS for advanced
data structures (yeah - I know about BINTABLE, but I'm still waiting to
see some complex data structures encoded and utilized portably.  It just
_might_ work, but I'm not sure it's actually been done yet).

It works pretty well as a working format in some (reasonably simple)
cases, but I think you'd need to look at some of the annotation and
other capabilities of HDF before you make the case that FITS is anything
close to ideal.

FITS is unsurpassed (I firmly believe) in doing what it was designed
for.  My remarks are in no way to be interpreted as an overall
criticism.  Heck, I've even seen a text editor written in Fortran.  Just
because you can do it doesn't mean you _should_ do it... <more
platitudes about teaching pigs to sing and everything looking like a
nail, etc, etc>