Subject: Re: HDF as an archive format

From: [email-masked] (Eric W. Sink)
Date: Wed, 4 Aug 1993 14:38:52 GMT

[email-masked] (Fred Walsteijn) writes:

>The files are certainly backward compatible, 
>but the function calls (required in your C or Fortran program) are not.
>Over the past few years NCSA has changed their function prototypes
>significantly many times.  That might be a disadvantage for you
>(it certainly is for me...).

Quite true.  We've found a number of instances where files written with
one revision of HDF don't read with another version.  I'm not talking about
3.1 vs 3.2, I mean 3.2r2 vs 3.2r4 and so on.  Overall, HDF is an outstanding
file format, but only a mediocre/good library implementation. (my opinion only)

-- 
Eric W. Sink, Spyglass        | "In all the earth, only humans have the ability
1800 Woodfield Drive          |    to be content in bad situations...
Savoy, IL  61826              |    and vice-versa."
---- [email-masked] ---------|---------- 217-355-6000 -----------------------

---

From: [email-masked] (Mike Tankenson)
Date: Wed, 4 Aug 1993 15:28:16 GMT

>file format, but only a mediocre/good library implementation. (my opinion only)

According to the HDF Frequently Asked Questions (FAQ), dated July 20, 1993,
NCSA claims that for the most part, the HDF file formats are backward
compatible:

   Q21: Can new versions of HDF read hdf files written using older version 
        of the HDF library?
   A21: Our goal is to make HDF backward compatible in the sense of
        that HDF files can always be read by new versions of HDF.  We have 
        succeeded in doing so up to HDF3.3 Beta, and will continue to follow
        the principle as much as possible. [ more stuff, including a
        detailed table showing backward compatibility ]

They then go on to state that the HDF software (i.e., libraries) are
backward compatible, but not forward compatible (i.e., file formats created
with new versions of HDF may not be readable/processable by older HDF
libraries).  This seems like common sense to me.

I take Fred's comment to mean that HDF is changing their libraries
frequently, and therefore your application software must change in response
to this.

I take Eric's comment to mean that a previous version of his application
software could not read *newer* HDF files, which is also consistent with my
understanding.  In other words, a 3.2r2 software version could not
read/process a 3.2r4 file format.  Eric, am I off base here?

-- 
Michael Tankenson ([email-masked])             Telos Systems Group 
Jet Propulsion Laboratory                   Phone: (818) 306-6445, Fax: 6966
4800 Oak Grove Drive, 525-3670, Pasadena, CA. 91109

---

From: [email-masked] (Don Jennings)
Date: 4 Aug 93 15:32:52

>[email-masked] (Mike Tankenson) writes:
>>1) Is there something inherent about HDF that would prevent its use as an
>>archive format?
>
>Depends on how serious you are about archiving.  The fundamental
>question to ask is how will a user, some 20 or 30 years in the future,
>upon finding a file in HDF format, access the data.

>In my opinion, formats which are defined _solely_ by software toolboxes
>and APIs are extremely dangerous for archival storage.  We have no way
>of assuring future generations of users that the software will be
>usable.

This is a good point. If HDF is used as an archival format then then the
archivers are making a bet that HDF catches on big time in the scientific
and commercial communties. A major strength (and weakness) of HDF is that
its a tightly controled system with a uniform software interface. If the
HDF group at NCSA drys up and goes away then so will the central organization
and control. However, I feel that the choice of HDF as the EOSDIS main
data format has assured that HDF will be around for quite a while.

Another point: there is no guarantee that one can come back to a FITS file
in 20 or 30 years and read it. The FITS format might have been poorly
documented by the person(s) who created it, or they might have built
their data file using a "funny" application of binary tables that 
confound FITS readers (human and machine). As far as that goes, the invent 
of the binary table extension and the freedom granted therein almost assures
that your FITS readers will not be able to understand my FITS files and
my FITS readers will not be able to understand your FITS files; by
"understand" I mean make use of the data, not just be able dump table columns
into the appropiate typed arrays.  

I believe that formats such as FITS are resonable archival formats as long 
as you can get a consensus from the various communities on EXACTLY what 
keywords, keyword values and binary table structures should be used. I also 
believe that HDF is a reasonable archival format as long as NCSA keeps on 
supporting it.

---

From: [email-masked] (Archie Warnock)
Date: 5 Aug 93 14:01:57 GMT

>data format has assured that HDF will be around for quite a while.

It undoubtably will be, but until HDF is adopted as a documented and
internationally recognized standard, it won't be save _as an archival_
format.  That's not to say it isn't useful, or efficient, or powerful,
or durable.  But it _is_ saying that data in HDF may not be recoverable
at some arbitrary time in the future.

>confound FITS readers (human and machine). As far as that goes, the invent 

This is patently false!

There indisputably _is_ such a guarantee - the byte-by-byte
specification of the format is published in the refereed literature
where it will always be accessible to future generations.  The
specification has been codified by the NASA/OSSA Office of Standards and
Technology, and has been recognized by the International Astronomical
Union as _the_ standard format for the interchange of astronomical data.
These, in fact, precisely _do_ "guarantee that one can come back to a
FITS file in 20 or 30 years and read it."

Does that mean that all software can read all FITS compliant data?  Of
course not.  Indeed (the FITSIO package notwithstanding), software
toolboxes have been largely lacking in the FITS community.  But that's
not the issue - we're talking about archiving data, i.e., making the
data safe for retrieval by future generations.

It will always be possible to write software to _extract the bytes_ if
they're stored in FITS format.  You can't do that with HDF.  Can I read
HDF files in Ada?  How about in Forth?  Visual Basic?  Perl?  Where are
the descriptions to tell me how to decode the bytes in an HDF file?  I
can tell you where they are for any FITS data - they're in virtually
every astronomical library in the world.  They're in hundreds of
university libraries around the world.  They're in the Library of
Congress!

>as you can get a consensus from the various communities on EXACTLY what 

And the astronomical community has taken the steps necessary to
formalize that consensus at the highest possible levels.  HDF has not
formalized that process - from an archival standpoint, it's a moving
target. 

Do I believe that FITS is, in some sense, the optimal and ultimately
correct format?  Not necessarily - FITS has some operational
deficiencies that remain tied to its original design as a tape-based
format - but I do believe that through FITS, the astronomical community
has demonstrated that it takes the job of archiving data seriously, and
has demonstrated how to go about the process of building _and
documenting_ the consensus required to do the job correctly.

>believe that HDF is a reasonable archival format as long as NCSA keeps on 

That's like saying that, as long as a book is still in print, it's
reasonable to print it on high acid paper - we can always get a new
copy, right?  What do we do when we can't get a new copy?

And as long as support for HDF is tied to a single institution, with no
force of international standards behind it, no formal documentation
ensuring that the data can be accessed from the format at any arbitrary
time in the future, it is unreasonable and unsafe to use it as an
archival format.  It's perfectly reasonable as an interchange format,
and as a working format, but _not_ for archival storage.
--
_______________________________________________________________________
-- Archie Warnock              Internet:  [email-masked]
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC                   Project STELAR: WAIS to do science

---

From: [email-masked] (Leslie Robert Lait)
Date: 5 Aug 1993 14:29:54 GMT

>believe that HDF is a reasonable archival format as long as NCSA keeps on 

Unfortunately, laying down the law on keywords, structures, etc., does 
not seem to work well in practice.  Once the metadata is separated from 
the data, the two inevitably begin to slip out of sync as revisions are 
made to the data sets, despite the best efforts of planning and oversight 
commitees.

In the past ten years or so, I have worked with a number of satellite data
sets.  Each was supplied with voluminous documentation specifying its
format, down to each bit where needed.  The documentation and the data
were official documents of the agencies involved and were presumably
signed off on by responsible officials.  Yet, I have never encountered
a case where the data matched the documentation.  Never.  Some have
come close, but I have not seen one which has been complete and accurate.  

Sometimes I get sample programs from my colleagues who claim to be
reading the "exact same" data sets.  But because they often are
using a format which is not _quite_ the format I have, even the 
code supplied by the instrument PIs is often wrong.

Fortunately, I have almost always been able to detect and understand 
the discrepancies, then read the data.  That's part of what they pay 
me for, I guess.  But I am very concerned about archiving data without
its metadata inextricably bound to it.  Using HDF is at least a step
in the right direction.

-- 
-------------------------------------------------------------------------
Leslie Robert Lait                           | 
[email-masked]                   |  "...Now hit any key.
My opinions are my own, not NASA's.          |   WAIT! NO! Not *that* key!"

---

From: [email-masked] (BARRY M. SCHLESINGER)
Date: 5 Aug 1993 10:44 EDT

In article <[email-masked]>, [email-masked] (Don Jennings) writes...
>into the appropiate typed arrays.  

First of all, a general comment.  All FITS formats endorsed by the IAU 
have been published in the refereed literature, in the Astronomy and 
Astrophysics supplement series.  Not many formats have been documented 
in this form that will be available for decades to come.

The basic goal of a FITS software reader is to generate the
appropriate table or array.  A FITS construction is based a set of
logical rules that can be embodied in software to read it, that is, to 
generate the data construct appropriate to the format.   In fact,
a demonstration of data transport using any proposed FITS format is
*required* before the IAU and regional FITS committees will consider
it for endorsement.  So FITS readers should be able to create 
correctly structured binary tables from any BINTABLE FITS file.

Now the interpretation is not done by the software but by people. 
So one cannot necessarily expect the FITS reading software to
scientifically interpret any file that it reads.  The file should
include enough descriptive material, and FITS provides ample
opportunity to do so, to enable the reader to scientifically interpret
the file.  It is possible to include inadequate descriptions, but that 
can happen with any format.  Generators of data have the 
responsibility to include enough information, using the facilities 
available in a particular format, to allow interpretation.   

				Barry Schlesinger
				NSSDC/NOST FITS Support Office

---

From: [email-masked] (Archie Warnock)
Date: 5 Aug 93 14:27:20 GMT

>made to the data sets, despite the best efforts of planning and oversight 

Well, no governing body is going to be able to out-flank the lone
programmer, bound and determined to do a crappy job of leaving an
adequate paper trail.

Still, a number of formats _do_ require the metadata to be an inherent
part of the data stream.  You're right - it's important, and necessary.
But standards alone aren't sufficient - they require the cooperation of
users, as well.  You're simply never going to find a format that makes
it _impossible_ to screw up.

>code supplied by the instrument PIs is often wrong.

Wait a sec - if the format is specified adequately, and adequately
tested (which is crucial for discovering oversights), then either data
complies with the spec or it doesn't.  If it doesn't, it's not the
format's problem...

---

From: [email-masked] (Chris Houck)
Date: Thu, 5 Aug 1993 16:58:12 GMT

>at some arbitrary time in the future.

This is a very valid concern.  Over the last few months, we have begun
to investigate the process of getting HDF documented as an official
standard.  However, initial estimates we have received, from outside
sources, indicate that it would take on the order of a full-time
person an entire year to walk HDF through the standardization process.

The HDF group currently does not have the resources to do that.  I
think we, more than anyone, want to get HDF accepted as a recognized
standard --- right now our funding sources have set different
priorities (now if anyone reading this has lots of money buring holes
in their pockets.... :-)

>university libraries around the world.  They're in the Library of

The byte by byte description of HDF can be found in the "HDF
Specification Manual".  While it does not meet the (reasonable)
requirement of independent publication (i.e. its an NCSA Manual) it
would be possible for other people to write a Forth-based HDF library.

I believe someone outside of NCSA started to build an alternate HDF
library a few years ago; I do not know what ever became of it.

>formalized that process - from an archival standpoint, it's a moving

There are really two issues here, I think.  On the one hand, there is
the concern that the library comes out of NCSA and the format has not
been published in a manner to make it a standard.  On the other hand,
HDF is "a moving target".  We have been, and will continue, expanding
and improving the library (HDF 3.3 Beta 2 was just released last week,
for example).  

My main concern is how standardization will impact library development. 

Right now, many of the top extensions on the HDF priority list are
EOSDIS-driven (ragged arrays, point / station data structures, among
others).  If we standardized HDF right now, those structures could not
be used in any "archive" files.  But if we wait to standardize until
we have implemented those structures there will be something else on
the top of the list.  So the logical conclusion is that we would need
to re-standardize HDF every few years.  I'm not saying it shouldn't be
done --- but we do not have the resources.   

-Chris Houck
HDF Development Team
NCSA Software Development Group

---

From: [email-masked] (Mike Jones)
Date: 5 Aug 93 20:02:04 GMT

Let me back up what Chris says (if not his intent :-)). The group I work
with produces about 1GB of HDF format satellite data per day. I think we can
safely say that HDF is far from ready to be standardised.

I believe that over the 20 year lifetime of EOS they will be able to
come up with a useful format that they will call HDF. I also believe that
they will have the resources and interest to go through the standards
process. We do have a promise that what they come up with will read the HDF
files we create now.

For long term posterity, FITS may be better. For short term use, FITS may
still be better. Still HDF isn't a terrible bet for the long term.

Mike

---

From: [email-masked] (Steve Allen)
Date: 5 Aug 1993 19:38:15 GMT

In article <[email-masked]>
[email-masked] (BARRY M. SCHLESINGER) writes:
>have been published in the refereed literature...

Which is a good thing.  Anything that goes in to a FITS file will always
be possible to get out.  I use FITS as a part of my work, and as a
data storage mechanism for my thesis data.  Some of my thesis data
are synthetic.  They are model matrices with appropriate amounts of
structured information to describe what each element of the model is,
how it was generated, etc.

>appropriate table or array....

This sentence exposes one of the worst problems with FITS
	"tabel or array"
FITS is great for encoding any kind of multidimensional rectangular
array.  It is not so great for other entities.

>       ...           So FITS readers should be able to create

I have no argument here.  Again, anything that goes *in* to a FITS
file will always be possible to get out.  The problem is, that some
kinds of data are very difficult to get *in* to a FITS file.

I mean, specifically, data structures, linked lists, and other sorts of
documentary information which would be associated with an image.  It is
possible now, with the BINTABLE extension, to encode these sorts of things
in a FITS file.  BINTABLE provides all the necessary pointers and
data HEAPs.

However, there really are no tools for converting a generic data structure
into this format.  Likewise, there are no tools for guaranteeing that the
data can be extracted into a reader which will reconstruct the original
structures.  Bill Pence's FITSIO toolkit is perfect for dealing with the
low-level operations on FITS files, and I use it.  Being written in standard
Fortran 77, however, means that it cannot provide the routines necessary
to comprehend, encode, and decode structures of data.

Using FITSIO, I have created a number of tools of this sort for my thesis
data, but they are in no way standardized.  I would not expect that
someone else's FITS reader would have a chance of actually using my
FITS files to do the work they were written to do.  Basically, then,
I have invented tools with some of the features of HDF, but which are
based upon an "archival" underlying format.

If HDF makes doing the work easier, then that is a significant advantage
for it which FITS does not have.  This is especially true for graduate
students or other sorts of projects which do not have the time to
invent some kind of wrapper code that successfuly stores and retrieves
their data structures.  Bill Pence's FITSIO routines are good, but
FITS would benefit even more by having a much higher-level interface
in a language which comprehends the kinds of non-rectangular data
structures which have always been around but which are awkward to
manipulate in FORTRAN.
_______________________________________________________________________________
Steve Allen          | That was the equation!         |   [email-masked]
UCO/Lick Observatory | Existence!...Survival must     | If the UC were opining,
Santa Cruz, CA 95064 | cancel out programming! -- Ruk | it wouldn't use me.

---

From: [email-masked] (Don Jennings)
Date: 5 Aug 93 15:28:50

>at some arbitrary time in the future.

This is not completely true. HDF data formats are documented in the HDF
low level and application interface code itself. This may not be the best
way to document it, but it is at least A way to document it.

As far as being an internationally reconized standard goes, such things
take time. HDF has only been around since 1988. 

>FITS file in 20 or 30 years and read it."

Just because the FITS format standard is well documented does NOT mean that
data written in the FITS format will be usable in 20 or 30 years. For
sake of argument I will agree that any valid FITS file can eventually
understood by having someone dump it and decode the headers, BUT who is
going to do that 20 or 30 years from now? That can be a lot of work,
especially if there are gigabytes of data to sift through. The HDF
approach of a standard software interface gets around this problem of
"convenience".

I am not knocking FITS. In fact, I am quite impressed with its success as
a data format and am part of a group building a FITS based data archive.
However, I do feel that it contains some built-in limitations that are
going to hamper its future use (I claim to be neither a FITS nor an HDF
expert, so don't beat on me too hard for this statement). HDF does indeed
have limitations when it comes to archiving (such as a lack of 
self-documentation), but I have yet to think of any built-in limitation
to the HDF format thats going to stop it from becomming a better archiving
format. 

>>formalized that process - from an archival standpoint, it's a moving

I cannot disagree here. But as I have said, give HDF a little time.

---

From: [email-masked] (BARRY M. SCHLESINGER)
Date: 6 Aug 1993 11:43 EDT

In article <[email-masked]>, [email-masked] (Steve Allen) writes...

>documentary information which would be associated with an image. 

Tables and arrays are the only standard extensions and proposals under 
consideration at present.  But the field is open for more development.

				Barry Schlesinger

---

From: [email-masked] (John Vanderpool)
Date: 11 Aug 93 18:11:15 GMT

[email-masked] (Archie Warnock) writes:

>1.  Where is the byte-by-byte description of the format given?  In an
>internationally recognized standards document?  In the refereed
>literature?  In a private publication by some individual institution?
>In source code?  Which is most likely to be accessible 50 years from
>now?

i don't think anyone has ever even considered this - a major drop of the ball.
if the types of discussion you mention above hadn't occurred for things like
TCP and zillions of other net things we use daily we wouldn't even be having
this discussion in all likelihood - RFCs etc take time but make sure things
are correct and fair (i, btw, don't know much about the process)

>There are (at least) three different applications for data formats -
>interchange, access and archival storage.  It is not at all obvious that
>any single format can meet the needs of all 3 simultaneously, even less
>obvious that any single format currently _does_ meet the needs of all 3.

very few people begin to even understand that these three things are
discrete

>Things like HDF, netCDF and CDF are good working formats but are risky
>as archival formats (at least, I've never seen references to format
>descriptions in the refereed literature - I hope I'm wrong, cause
>otherwise EOSDIS is gonna be in a lot of trouble years from now).  FITS
>and PDS are pretty good interchange and archival formats, but weak
>working formats.

fully agreed - but i won't go into it.

>Archie

hi arch! (pickin' party at WT was amazing)

			fish

--
John R. Vanderpool     INTERNET: [email-masked]
All I can say is my life is pretty plain, I like watchin' puddles fill with rain
All I can do is pour some tea for two, and speak my point of view, but its not
 sane.  -blind melon