Subject: Re: HDF as an archive format

From: [email-masked] (Fred Walsteijn)
Date: Fri, 30 Jul 1993 09:42:24 GMT

In <dilg.743980475@xongmao> [email-masked] (Doug Ilg) writes:

>[email-masked] (Mike Tankenson) writes:

>>2) Is HDF software so volatile that one cannot depend on it for something
>>as critical as data archive?

>No.  HDF continues to evolve, as does any good piece of software, but it
>maintains backward compatibility with all older versions.  The worst case
>is that, in the future, there may be a conversion program needed to update
>old files to new libraries, although that hasn't been necessary to date.

>>3) Are HDF versions backward compatible?  If not, will they be at some
>>point in the future (i.e., will HDF settle down)?

>See 2)

The files are certainly backward compatible, 
but the function calls (required in your C or Fortran program) are not.
Over the past few years NCSA has changed their function prototypes
significantly many times.  That might be a disadvantage for you
(it certainly is for me...).

Hope this helps,
Fred (PhD student)
_____________________________________________________________________________
Fred H. Walsteijn                             |
Institute for Marine and Atmospheric Research |
Utrecht University                            |
Princetonplein 5                              | Internet: [email-masked]
3584 CC Utrecht                               | FAX:      31-30-543163
The Netherlands                               | Phone:    31-30-533169
_____________________________________________________________________________

---

From: [email-masked] (Archie Warnock)
Date: 30 Jul 93 14:30:30 GMT

[email-masked] (Mike Tankenson) writes:
>1) Is there something inherent about HDF that would prevent its use as an
>archive format?

Depends on how serious you are about archiving.  The fundamental
question to ask is how will a user, some 20 or 30 years in the future,
upon finding a file in HDF format, access the data.

In my opinion, formats which are defined _solely_ by software toolboxes
and APIs are extremely dangerous for archival storage.  We have no way
of assuring future generations of users that the software will be
usable.  The questions to ask about archival storage are:

1.  Where is the byte-by-byte description of the format given?  In an
internationally recognized standards document?  In the refereed
literature?  In a private publication by some individual institution?
In source code?  Which is most likely to be accessible 50 years from
now?

2.  Is the format self-documenting?  Is the metadata in (at least)
ASCII?  Can the user determine anything about the contents of the data
>from a simple dump?

3.  Is the format well-defined?  Can a user assume that versions are
backward compatible?

There are (at least) three different applications for data formats -
interchange, access and archival storage.  It is not at all obvious that
any single format can meet the needs of all 3 simultaneously, even less
obvious that any single format currently _does_ meet the needs of all 3.

In general, working formats need to emphasize access (read/write)
efficiency.  Interchange formats need storage efficiency and
standardized numeric representations.  Archival formats need to be
self-documenting, externally documented and simple.

Things like HDF, netCDF and CDF are good working formats but are risky
as archival formats (at least, I've never seen references to format
descriptions in the refereed literature - I hope I'm wrong, cause
otherwise EOSDIS is gonna be in a lot of trouble years from now).  FITS
and PDS are pretty good interchange and archival formats, but weak
working formats.

It's an important area for discussion...

--
_______________________________________________________________________
-- Archie Warnock              Internet:  [email-masked]
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC                   Project STELAR: WAIS to do science