Subject: Good archive format ???

From: [email-masked] (Ata Etemadi)
Date: Mon, 9 Aug 93 18:59:46 BST

G'Day

Could someone explain what exactly is meant by a "good archive format" ?
Before I continue I would suggest that the best option for us as users 
is for the HDF, FITS, netCDF, CDF etc.. folks to get together and form a 
consortium to share resources and make for better data interchange, if 
not indeed integration. All competition is doing is making it harder for 
the users and wasting money. As far as I can see no one has the "perfect" 
format for archiving or otherwise so just give us users the tools to work 
with all your formats easily without the need to learn hundreds of different 
subroutine calls and reading masses of documentation. How about an IEEE 
standard on data modelling (a data Kernel) ? This would sure make it easier 
to integrate all these formats. As a user I would like to define data and 
Metadata and just do SaveAsFits(...) or SaveAsGIF(...) or SaveAsNetCDF(...) 
or ReadData(...) ReadMetaData(...). OK so its a pipe-dream, I know.

Any objections you may have about HDF as an archive format can be said 
about any other format since they all have their strenghts and weaknesses. 
A data format designed specifically for a particular data type and which 
is going to be supported indefinitely would clearly be "best". You may also 
want to consider compression, at which point you need to ask what compression 
scheme is best suited to the data. IMHO, the most important question is 
whether the format is well documented. In 10 years or 1000 years (if your 
data or for that matter humanity :-) survive that long) the main concern 
of the user will be how good is the documentation, since any programmer 
worth a damn should be able to access your data given good documentation. 

	adios
		Ata <(|)>.
P.S.   I plan to use HDF as an archive format.
P.P.S. I plan to make my software for converting data from various instruments
       to HDF freely available. If you want me to include your data just
       drop me a line.
-- 
| Mail          Dr Ata Etemadi, Blackett Laboratory,                          |
|               Space and Atmospheric Physics Group,                          |
|               Imperial College of Science, Technology, and Medicine,        |
|               Prince Consort Road, London SW7 2BZ, ENGLAND                  |
| Internet/Arpanet/Earn/Bitnet [email-masked] or [email-masked]  |
| Span                              SPVA::atae       or     MSSLC:atae        |
| UUCP/Usenet                       [email-masked]        |

---

From: [email-masked] (Don Wells)
Date: Mon, 9 Aug 1993 22:22:53 GMT

In article <[email-masked]> [email-masked]
(Ata Etemadi) writes:
 AE> Could someone explain what exactly is meant by a "good archive
 AE> format"? .. Any objections you may have about HDF as an archive
 AE> format can be said about any other format since they all have
 AE> their strenghts and weaknesses.. 

Strengths and weaknesses of ABI and datamodel features are largely
irrelevant when evaluating formats for archival purposes (i.e., to
survive for decades), whereas the treatment of metadata is critically
important.  For *archival* purposes the self-describing formats like
FITS are decisively superior to the tagged binary formats like HDF.

 AE> .. IMHO, the most important question is whether the format is
 AE> well documented..

On a timescale of decades you cannot depend on the documentation
approaches traditionally used for formats. The detailed documentation
of particular instances of a format (i.e., the metadata) must be
embedded in the files themselves in human- and machine- readable form
(this is what is meant by self-describing), instead of being supplied
on paper. In addition, the generalized format documentation must be
widely distributed in an archival form -- this requires formal
publication in a journal which will be kept by a large number of
research libraries all over the world.

 AE> ..In 10 years or 1000 years.. any programmer worth a damn should
 AE> be able to access your data given good documentation..

Experience shows that paper documentation is often incorrect and often
gets separated from bitstreams.  For *archival* purposes there is no
reliable substitute for metadata (documentation) which is an integral
and essential part of datasets, included in such a way that it cannot
be separated from the bitstream and essentially *must* be correct.
--

  Donald C. Wells           Associate Scientist       [email-masked]
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road                              Fax= +1-804-296-0278
  Charlottesville, Virginia 22903-2475 USA         78:31.1W, 38:02.2N

---

From: [email-masked] (Ata Etemadi)
Date: Tue, 10 Aug 93 01:17:37 BST

In article <[email-masked]>, [email-masked] (Don Wells) writes:
|> In article <[email-masked]> [email-masked]
|> (Ata Etemadi) writes:
|>  AE> Could someone explain what exactly is meant by a "good archive
|>  AE> format"? .. Any objections you may have about HDF as an archive
|>  AE> format can be said about any other format since they all have
|>  AE> their strenghts and weaknesses.. 
|> 
|> Strengths and weaknesses of ABI and datamodel features are largely
|> irrelevant when evaluating formats for archival purposes (i.e., to
|> survive for decades), whereas the treatment of metadata is critically
|> important.  For *archival* purposes the self-describing formats like
|> FITS are decisively superior to the tagged binary formats like HDF.

Lets get down to business and start the first sci.data.formats flame war.
This is not my aim but on the other hand I don't pull punches so my apologese 
for any perceived harshness. 

The above is an interesting assertion. I was not aware that HDF was not
considered self-describing. I would appreciate it if you could enlighten 
me regarding your experience with HDF. Unless I have missed something,
I expect my use of Metadata to have made my HDF files self-describing.

|>  AE> .. IMHO, the most important question is whether the format is
|>  AE> well documented..
|> 
|> On a timescale of decades you cannot depend on the documentation
|> approaches traditionally used for formats. The detailed documentation
|> of particular instances of a format (i.e., the metadata) must be
|> embedded in the files themselves in human- and machine- readable form
|> (this is what is meant by self-describing), instead of being supplied
|> on paper. In addition, the generalized format documentation must be
|> widely distributed in an archival form -- this requires formal
|> publication in a journal which will be kept by a large number of
|> research libraries all over the world.

I would expect that there are many hundreds of copies of the HDF documentation 
online on the internet. Publication in a journal ensures very little. FITS
has gone through many revisions since the original paper. As in all cases there 
is no avoiding human error. Its only possible to try and minimize it. And the 
best way to do that is to make it easy for the user. Sadly you deleted my
section on how nice it would be if people stopped blowing their own horns 
and got together to make it easier for users :-) I put my trust in the thousands 
of users whose feedback will help to shape things rather than on some misplaced 
trust in that publication in a journal ensures some kind error-innoculation.

|>  AE> ..In 10 years or 1000 years.. any programmer worth a damn should
|>  AE> be able to access your data given good documentation..
|> 
|> Experience shows that paper documentation is often incorrect and often
|> gets separated from bitstreams.  For *archival* purposes there is no
|> reliable substitute for metadata (documentation) which is an integral
|> and essential part of datasets, included in such a way that it cannot
|> be separated from the bitstream and essentially *must* be correct.

You contradict yourself. In the last paragraph you say that the data has to 
be self-describing followed by a sentence saying that the paper copies of 
the published format are what make FITS a good archival format, now followed 
by an assertion that "paper documentation is often incorrect". In  almost one 
breath you argue against my assertion about documentation and then argue for it 
and then argue against it and your own assertion about the importance of
documentation. Presumably you are also saying that publication in a journal
somehow ensures FITS code is bug-free or has no "features". I assume you
have done some programming and therefore know this to be, at best, wishful
thinking.

There is no way you can guarantee that the Meta Data *must* be correct.
You can only bank on certain easily-verifiable items such as number of
data or rows and columns of the image. If column 1 is density and it is
labelled velocity there is no way FITS or any format whatsoever can check
its correctness.

	adios
		Ata <(|)>.

---

From: [email-masked] (Don Wells)
Date: Tue, 10 Aug 1993 05:36:07 GMT

In article <[email-masked]> [email-masked]
    (Ata Etemadi) writes:
 AE> Lets get down to business and start the first sci.data.formats
 AE> flame war. 

Flame war?!? I hope not. This is an intellectual debate, not flames,
as far as I am concerned. I hope that the end result will be to make
progress toward consensus on the essential properties of formats for
archival purposes.

 AE> .. I don't pull punches so my apologese for any perceived
 AE> harshness.. 

Likewise for me.

 AE> .. I was not aware that HDF was not considered self-describing. I
 AE> would appreciate it if you could enlighten me regarding your
 AE> experience with HDF..   

I defined the term "self-describing" in my posting (and I define it
again below). I read the HDF documentation several years ago and
concluded that HDF did not meet that criterion, at least at that time
(it is possible, of course, that I misunderstood something). I also
participated in the NASA data format comparison workshop in June 1992,
and what I heard there did not cause me to suspect any error in my
prior understanding of the HDF architecture.

I can equally well ask you about your experience with FITS (and/or PDS
[Planetary Data System]).

 AE> I expect my use of Metadata to have made my HDF files
 AE> self-describing. 

In a dump of a FITS (or PDS) header "what you see is what you get", in
plain ASCII. What do I see and what do I get in a dump of your HDF
files?  What can a dump of one of your files tell me about the
provenance of the data objects, the schema, the names, the units, the
coordinate systems, ancillary values, etc.? I.e., to what extent is
the file human readable? Human readability is a valuable (arguably
essential) feature in an archival format.

 AE> .. hundreds of copies of the HDF documentation online on the
 AE> internet..

Those files are useless on a timescale of decades; there is no way to
assure that any of them will be accessible 30 years from now.  Note: I
am a big proponent of the Internet, and I hope and expect that those
files will survive, and I intend to do my part to try to make it
happen, but we need a *guarantee* for an archive.

 AE> Publication in a journal ensures very little. 

Scholarly activity as it is now practiced has existed for several
centuries, and librarians have strong archival traditions and
technologies. Therefore, publication of a paper documenting the
architecture of a format in a major journal absolutely assures that
that description will exist decades from now, indeed it probably
assures that it will exist one or more centuries from now, and that it
will exist in hundreds of libraries all over the world, everywhere
that scholarly activity exists.  Furthermore, references to the
journal papers can be embedded in scientific publications which are
themselves archival documents. Finally, references to the papers can
be embedded in the bitstreams as retrieval keys for future
programmers.

 AE> FITS has gone through many revisions since the original paper..

Not so. The files in anonymous FTP directory fits.cv.nrao.edu
/fits/testfiles/incunabula are examples of what FITS looked like in
1979-80; those files are still valid and are still readable by FITS
readers all over the world.  There have been *no* revisions in the
specifications of either Basic FITS or the Random-Groups format, both
of which were designed and specified in that period, and both of which
are still in use. The architectural descriptions of those formats,
which were published in a journal in 1981, are still valid and
authoritative, and references to them are embedded in the
bibliographies of several thousand astronomical papers in the
published (archival) literature.  ASCII Table objects were defined in
1984 as an optional addition to Basic FITS, not as a revision.
Likewise, binary tables (which are a self-describing XDR) are an
addition (an "extension" in FITS parlance), not a revision. In fact,
the FITS community formally agreed in 1988 to never make a revision
that would invalidate previously-valid FITS files. FITS is quite
flexible, and so this agreement does not prevent the format from
evolving, it merely somewhat restricts the paths which the evolution
can take.

 AE> .. users.. feedback.. help to shape things  rather than.. misplaced 
 AE> trust [that] publication.. ensures.. error-innoculation..

The process used to formulate standards is critical. Extensive and
time-consuming review and testing is essential before publication. The
FITS publications are based on prototype testing plus elaborate
community review processes to assure correctness and usefulness of the
architectural designs.

 AE> You contradict yourself. In the last paragraph you say that the
 AE> data has to be self-describing followed by a sentence saying that
 AE> the paper copies of the published format are what make FITS a
 AE> good archival format, now followed by an assertion that
 AE> "paper documentation is often incorrect". 

There is no contradiction. FITS is an architecture, a set of general
rules with sets of options. It is the formal specifications of the
architecture that are published in the journal. Instances of FITS
files are created by a variety of datasystems all over the world, and
use the rules and options in a variety of ways. The rules are
flexible, so FITS files in a variety of styles exist, are
interchanged, and are archived. The metadata which is embedded in the
FITS files documents the peculiarities of the individual instances,
and that metadata is to be construed by the general rules which appear
in the published papers. It is good to also have paper documents which
can describe the peculiarities of instances in more detail, but the
paper must not regarded as a substitute for the self-describing,
self-consistent metadata which are an integral part of the files.

Formats which depend solely on paper are vulnerable to loss of
self-consistency and to loss of the paper.

 AE> There is no way you can guarantee that the Meta Data *must* be
 AE> correct. You can only bank on certain easily-verifiable items
 AE> such as number of data or rows and columns of the image. If
 AE> column 1 is density and it is labelled velocity there is no way
 AE> FITS or any format whatsoever can check its correctness. 

Obviously programming errors can create invalid files. However, most
files are generated by library code, and so such errors are soon fixed
and stay fixed. The real question is whether the definitions of the
data structures are carried within the file in a form which is readily
understandable to both humans and machines. If so, then we have a
self-describing format. If instead the critical definitions (e.g., the
XDR descriptions) are kept in a central repository, the format is
*not* self-describing. Furthermore, it is not sufficient to include
text forms of the XDR descriptions as extra comment text objects.
Instead, the descriptions must be an integral and critical part of the
structure of the file, so that the library code which generates them
assures that they are not obsolete, and so that they are checked by
the act of trying to read back a file into the datasystem which
generated it. It is in this sense that the metadata "*must* be
correct".
--

---

From: [email-masked] (Bob Hanisch, ST ScI)
Date: Tue, 10 Aug 1993 18:23:24 GMT

In article [email-masked], [email-masked] (Ata Etemadi) writes:
> In article <[email-masked]>, [email-masked] (Don Wells) writes:
> |> In article <[email-masked]> [email-masked]
> |>     (Ata Etemadi) writes:
> |>  AE> .. I was not aware that HDF was not considered self-describing. I
> |>  AE> would appreciate it if you could enlighten me regarding your
> |>  AE> experience with HDF..   
> |> 
> |> I defined the term "self-describing" in my posting (and I define it
> |> again below). I read the HDF documentation several years ago and
> |> concluded that HDF did not meet that criterion,.......
> 
> I have to say that your definition of "self-describing" has me confused 
> (or is confusing). Are you saying that the FITS format contains the definition 
> of primitives such as long, float, double etc.. in terms of number of bits 
> or are you referring to the defintion of data structures such as fillvalue,
> maxvalue, name, label etc.. ? 

FITS does, in fact, define these types of quantities explicitly (e.g., BITPIX
gives the number of bits per data pixel, DATAMIN gives the minimum data value,
etc.  Since FITS originated at a time when there was no clear standard for word
lengths, it only really depends on a mutual understanding of terms such as bit
and byte (although even these terms are explicitly defined in the NASA
codification of the standard).  FITS data is not defined as longs, floats, or
doubles.  It is left to the FITS reading program to assign host datatypes which
support the FITS data appropriately.

> The reason I find HDF a good format, specially
> since it is evolving and not stagnant, is that the NCSA folks are clearly
> willing to cooperate with other institutes in improving HDF and the contributed
> format. They recognise that their format is not "perfect" and are willing to 
> do something about it. 

Exactly the same is true for FITS.  FITS _is_evolving_, and there is a 
substantial contingent within the astronomical community actively participating
in the evolution of FITS to accommodate more sophisticated data structures.
FITS was designed from the outset to be extensible.

> I feel you keep evading the issue:
> 
> Are the FITS folks willing to collaborate with HDF to incorporate their format 
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> within it ? Or are you suggesting that this is impossible or impractical ?
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Please give me a reason.
> ^^^^^^^^^^^^^^^^^^^^^^^^

I will not claim to speak for the entire FITS community, but for myself I would
say that there is no particular benefit, at this point in time, in trying to
incorporate FITS into HDF.  All of the major software systems and many
home-brew packages in use in astronomy support the FITS format.  The astronomy
community has been successful in this regard.  FITS is used for a number of
astronomical archives.  It is used for network file transfers.  Given this high
level of acceptance and the mechanisms that have been established to oversee
the orderly evolution of FITS, it would be ill-advised to drop it or try to
merge it under HDF.  My opinion only.  And my opinion as of this time.  As
needs/requirements evolve, my opinion may change.

> I don't have any experience with PDS. My experience with FITS goes back to
> my days using MIDAS and AIPS within the ISO project. FITS = Flexible Image
> **Transportation** System. I don't remember any mention of archiving in the
> AA paper. I don't fancy digging it up again. Maybe you can refresh my memory.
> Not all astronomical data is gridded. In particular the support for time-series 
> is not good (I am being kind here) unless things have changed.

The random groups format was designed to deal with non-uniformly gridded
data.  It was not widely adopted.  The binary tables extension to FITS solves
this problem, simply and efficiently.  This extension to the FITS standard
is now coming before the IAU FITS Committee for review/approval.

> I did have
> one major problem with FITS. When I took my FITS files off tape written on
> Vaxes and put them on a Sun I could not read the darn things. I contacted 
> the MIDAS folks whose software wrote some of the images and they were about 
> as helpful as a bunch of carrots and abusive too "Do you really have this 
> problem" "Oh so the tape drive went round and round and nothing happened". 
> Maybe I just got unlucky and came across the mandatory Fascist.

This comment is hardly worth responding to.  The MIDAS software support group
has, to the best of my knowledge, always tried to provide good service to its
user community.  FITS is used extensively for transfer of data from VAXes to
Suns and to a variety of other systems, including PCs.  Maybe you had a bad
tape.  In any case, this experience has nothing to do with FITS.

> |>  AE> I expect my use of Metadata to have made my HDF files
> |>  AE> self-describing. 
> |> 
> |> In a dump of a FITS (or PDS) header "what you see is what you get", in
> |> plain ASCII. What do I see and what do I get in a dump of your HDF
> |> files?  What can a dump of one of your files tell me about the
> |> provenance of the data objects, the schema,...........
> 
> Not correct. Try ncdump which does precisely the above for HDF files. I 
> suggest you are out of date.

What is ncdump?  Software from NCSA?  If so, this proves Don Wells' point
precisely, as making the metadata human readable requires special software.
When Don talks about dumping the file, he means using standard tools in the
host OS, not custom programs designed for the particular data format.

> My main points were: a) there are no guarantees b) the format is evolving
> and I would suggest its best to get involved now c) I doubt very much
> that the original FITS files will last decades. Most likely the data will 
> be recast at some future date. Are you are saying that you have never
> recast your data and your archives are stable ? In which case my hat off 
> to you.

FITS files have already lasted 15 years and have not had to be modified to
be readable by current FITS-reading programs.  That's a decade and a half,
anyway.  Given that a number of major archives, such as the HST archive,
use FITS as the archive format, I think we will easily see FITS files lasting
for decades.  However, I certainly would not rule out recasting this data
into another format at some time in the future.  Having it in FITS should
make this quite simple, should the time come.
  
> |>  AE> Publication in a journal ensures very little. 
> [brief lecture on the obvious deleted]
> 
> Again my point was that in so far as the user is concerned 2000000000000000
> publications won't help you with YOUR particular data unless the creator
> had taken care to ensure good documentation (as MetaData or paper). Hence
> my original assertion that good documentation is important.

Of course good documentation is important.  But I have to agree with Don that
unless it is part of the published literature it is vulnerable.

> |>  AE> FITS has gone through many revisions since the original paper..
> |> 
> |> Not so. The files in anonymous FTP directory fits.cv.nrao.edu
> |> /fits/testfiles/incunabula are examples of what FITS looked like in
> |> 1979-80; those files are still valid and are still readable by FITS
> |> readers all over the world.  There have been *no* revisions in the
> |> specifications of either Basic FITS or the Random-Groups format ...
> 
> From the dictionary:
> 
> re.vise \ri-'vi_-z\ vb
>   1: to look over something written in order to correct or improve
> 
> Are you suggesting FITS was not revised ? Clearly untrue.

FITS has been improved, namely, extended.  This is a critical distinction.
If you wish to use the term "revised" to mean "extended", fine.  But the
usual connotation of revised is "changed" or "corrected".
 
> |>  AE> .. users.. feedback.. help to shape things  rather than.. misplaced 
> |>  AE> trust [that] publication.. ensures.. error-innoculation..
> |> 
> |> The process used to formulate standards is critical. Extensive and
> |> time-consuming review and testing is essential before publication......
> 
> The original paper was refereed by 2 persons. I don't think this makes
> up a community :) At which point we come back to the true community who
> are the users which as I pointed out are the people who will shape things
> no matter how many papers are published.

It is worth understanding the process by which the FITS standard evolves.  The
refereeing of the published paper is a negligible part of the process, and is
not intended to convey community support.  The original FITS design came about
via a process of cooperation in data exchange between several major
observatories.  There are currently three regional FITS committees -- one in
Europe, one in North America (under the authority of the American Astronomical
Society), and one in Japan.  Anyone can suggest/submit a proposal for an
extension to the FITS standard, or for conventions to be layered upon it.  Such
a proposal is reviewed by each of these committees, and each committee votes
independently on whether or not to accept the proposal.  If all three
committees approve, the proposal is then taken up by the FITS Committee under
Commission 5 of the International Astronomical Union, which is the world-wide
professional organization for astronomers.  The IAU FITS Committee then votes
on approval, and if accepted, the proposal is forwarded to the IAU General
Assembly for endorsement.  As Don Wells alluded, this is a long and drawn-out
process, usually requiring 2-3 years for a proposal to move through the
pipeline.  This long time period is GOOD.  It assures that proposals have
extensive and thorough review, and a period of real testing and data exchange.
Community input to the process also takes place through the sci.astro.fits
newsgroup, where there are frequently animated discussions on the pros and cons
of new proposals.

It is this _process_ which is as critical to the development and acceptance
of a data format standard as is the technical merits of the format itself.

> |>  AE> You contradict yourself. In the last paragraph you say that the
> |>  AE> data has to be self-describing followed by a sentence saying that
> |>  AE> the paper copies of the published format are what make FITS a
> |>  AE> good archival format, now followed by an assertion that
> |>  AE> "paper documentation is often incorrect". 
> |> 
> |> There is no contradiction....
> 
> con.tra.dict \.ka_:n-tr*-'dikt\ vb : to state the contrary of :
> 
> Please re-read your posting and my comments. It is either true that the paper 
> copy is unimportant or it is false. You can't have it both ways. Let me spell it
> out for you. Ie either the papers published on FITS [ie in paper format and not
> in electronic form or MetaData] are important or NOT.

This is silly wordplay.  As I stated above, the documentation is important.
We have both the standards documents and a user's guide (developed by the
NASA FITS Support Office) which describes conventions, good practices, etc.
If this were not important we would not have it.  Don Wells' point, if I may
paraphrase, is that it would be considerably easier to properly read and
interpret a FITS file, owing to the immediate readability of its metadata,
than to properly read and interpret a binary structure in which the metadata
is encoded, without the help of external documentation.

Finally, I hate to see this as an argument of whether FITS is better than
HDF, or vice versa.  That is not the point I am trying to make in responding
to these postings.  I am concerned primarily that incorrect or misleading
statements about FITS not be promulgated.  Similarly, I hope that if I have
made any incorrect or misleading statements about HDF that these will be
corrected.   There would not be a variety of data formats if there were not
a variety of needs, a variety of technical solutions, and a variety of
processes employed to endorse the resulting standards.  

---
Robert J. Hanisch                       Tel. (410)338-4910   Fax. (410)338-5090
Space Telescope Science Institute       NSI: [email-masked] [130.167.1.2]
3700 San Martin Drive                   DECNet: STSCIC::HANISCH [6549::] 
Baltimore, MD  21218  USA               Bitnet: hanisch@stsci

---

From: [email-masked] (Ata Etemadi)
Date: 10 Aug 93 22:46:07 GMT

In article <[email-masked]>, [email-masked] (Bob Hanisch, ST ScI) writes:
|> In article [email-masked], [email-masked] (Ata Etemadi) writes:
|> > In article <[email-masked]>, [email-masked] (Don Wells) writes:
|> > |> In article <[email-masked]> [email-masked]
|> > |>     (Ata Etemadi) writes:
|> > |>  AE> .. I was not aware that HDF was not considered self-describing. I
|> > |>  AE> would appreciate it if you could enlighten me regarding your
|> > |>  AE> experience with HDF..   
|> > |> 
|> > |> I defined the term "self-describing" in my posting (and I define it
|> > |> again below). I read the HDF documentation several years ago and
|> > |> concluded that HDF did not meet that criterion,.......
|> > 
|> > I have to say that your definition of "self-describing" has me confused 
|> > (or is confusing). Are you saying that the FITS format contains the definition 
|> > of primitives such as long, float, double etc.. in terms of number of bits 
|> > or are you referring to the defintion of data structures such as fillvalue,
|> > maxvalue, name, label etc.. ? 
|> 
|> FITS does, in fact, define these types of quantities explicitly (e.g., BITPIX
|> gives the number of bits per data pixel, DATAMIN gives the minimum data value,
|> etc.  Since FITS originated at a time when there was no clear standard for word
|> lengths, it only really depends on a mutual understanding of terms such as bit
|> and byte (although even these terms are explicitly defined in the NASA
|> codification of the standard).  FITS data is not defined as longs, floats, or
|> doubles.  It is left to the FITS reading program to assign host datatypes which
|> support the FITS data appropriately.

In that case HDF is also self-describing. Actually any format which contains
or can contain such a definition is self-describing under this definition.
Therefore I don't understand why Don Wells concluded "that HDF did not meet 
that criterion". Not having used FITS for ages I would be interested to
know how mixtures of chars, longs etc.. are now handled. I know BITPIX is
standard, so presumably there is another standard to define mixtures. Correct ?

Let me get this straight. What Don and you are saying is that because FITS is 
a) a published standard b) goes through a review process c) contains an ASCII 
header which is human readable implies that it is a good archive format for 
*any* kind of data. This in essence is what you are putting forward as the
definition of "a good archive format". Yes ? OK. I have an MPEG movie. Can you 
put it into FITS ? I have some sparse matrix data is this supported by FITS ? 
Bzzzzzzzzt :) Ah but I can see the retort already. These things may or may not 
be included in some future FITS format. Lets suppose not. What I should do
is dismantle the MPEG into thousands of images and put it in FITS for archiving
purposes. Get REAL as you Americans say.

Horses for courses. Maybe if you tried harder to understand the problem as 
opposed to concentrating on the solution we'd get further. For some applications
FITS is very good for others HDF is very good for some CDF is very good etc..
Just to re-iterate. What is required is to make it easier for the users to
use all of these formats. Is this not logical and reasonable ? Can you not
appreciate this point ? Do you really think that all these other formats
GIF, Tiff, Postscript, Jpeg etc. etc.. evolved out of the fact that people 
are silly and like making up formats while there is this nice FITS format
all ready to use ? I am being particularly silly here to make sure I see
a response.
               
|> > The reason I find HDF a good format, specially
|> > since it is evolving and not stagnant, is that the NCSA folks are clearly
|> > willing to cooperate with other institutes in improving HDF and the contributed
|> > format. They recognise that their format is not "perfect" and are willing to 
|> > do something about it. 
|> 
|> Exactly the same is true for FITS.  FITS _is_evolving_, and there is a 
|> substantial contingent within the astronomical community actively participating
|> in the evolution of FITS to accommodate more sophisticated data structures.
|> FITS was designed from the outset to be extensible.

Obviously you took my comments to mean that FITS is stangnant. I have to
say on reading it myself there is that implication if you start off on
a certain tack. I am very glad to know that FITS is evolving. I'd also
appreciate knowing what its evolving into ? What you imply is that it
is evolving into an all embracing format which can handle everything. Well
again my hats off to you since I obviously don't have your crystal ball.

|> > I feel you keep evading the issue:
|> > 
|> > Are the FITS folks willing to collaborate with HDF to incorporate their format 
||> > within it ? Or are you suggesting that this is impossible or impractical ?
||> > Please give me a reason.
||> 
|> I will not claim to speak for the entire FITS community, but for myself I would
|> say that there is no particular benefit, at this point in time, in trying to
|> incorporate FITS into HDF...... 

[some more yes, and no lines deleted]

In other words you are saying nothing useful and not providing any valid reasons.

|> > I don't have any experience with PDS. My experience with FITS goes back to
|> > my days using MIDAS and AIPS within the ISO project. FITS = Flexible Image
|> > **Transportation** System. I don't remember any mention of archiving in the
|> > AA paper. I don't fancy digging it up again. Maybe you can refresh my memory.
|> > Not all astronomical data is gridded. In particular the support for time-series 
|> > is not good (I am being kind here) unless things have changed.
|> 
|> The random groups format was designed to deal with non-uniformly gridded
|> data.  It was not widely adopted.  The binary tables extension to FITS solves
|> this problem, simply and efficiently.  This extension to the FITS standard
|> is now coming before the IAU FITS Committee for review/approval.

Is there now support for time-series ? How about sparse matrices ? How about
JPEG compression or MPEG ? I need these features for my data. Should I wait
to see if the IAU are going to do something about it ? I would be very glad
to know that the IAU has the good of the whole scientific community (very
very few of whom are astronomers) in mind.

|> > I did have
|> > one major problem with FITS. When I took my FITS files off tape written on
|> > Vaxes and put them on a Sun I could not read the darn things. I contacted 
|> > the MIDAS folks whose software wrote some of the images and they were about 
|> > as helpful as a bunch of carrots and abusive too "Do you really have this 
|> > problem" "Oh so the tape drive went round and round and nothing happened". 
|> > Maybe I just got unlucky and came across the mandatory Fascist.
|> 
|> This comment is hardly worth responding to.  The MIDAS software support group
|> has, to the best of my knowledge, always tried to provide good service to its
|> user community.  FITS is used extensively for transfer of data from VAXes to
|> Suns and to a variety of other systems, including PCs.  Maybe you had a bad
|> tape.  In any case, this experience has nothing to do with FITS.

Get off your high horse. I know FITS is used extensively for this purpose.
My point was that if the implementation is corrupt no amount of publication
in journals is going to help.

|> > |>  AE> I expect my use of Metadata to have made my HDF files
|> > |>  AE> self-describing. 
|> > |> 
|> > |> In a dump of a FITS (or PDS) header "what you see is what you get", in
|> > |> plain ASCII. What do I see and what do I get in a dump of your HDF
|> > |> files?  What can a dump of one of your files tell me about the
|> > |> provenance of the data objects, the schema,...........
|> > 
|> > Not correct. Try ncdump which does precisely the above for HDF files. I 
|> > suggest you are out of date.
|> 
|> What is ncdump?  Software from NCSA?  If so, this proves Don Wells' point
|> precisely, as making the metadata human readable requires special software.
|> When Don talks about dumping the file, he means using standard tools in the
|> host OS, not custom programs designed for the particular data format.

This is better. What he said was "What do I see and what do I get in a 
dump of your HDF files?" which I took to mean the process of dumping as
opposed to printing to screen in ASCII. This is a very good feature of FITS.
One may argue that the ncdumped HDF file is essentially better than the 
original HDF file. NOTE: ncdumped file contains all the data and Metadata. 
Well there are pros and cons. It does not however ensure that all the Metadata 
is correct. Maybe the HDF folks will put on a standard ASCII header since 
it would be nice for long-term archiving. For me it is not big problem since 
there will be a documented description of all my archived data and, short of
NCSA getting nuked, I think I can say my data will be readable.
 
|> > My main points were: a) there are no guarantees b) the format is evolving
|> > and I would suggest its best to get involved now c) I doubt very much
|> > that the original FITS files will last decades. Most likely the data will 
|> > be recast at some future date. Are you are saying that you have never
|> > recast your data and your archives are stable ? In which case my hat off 
|> > to you.
|> 
|> FITS files have already lasted 15 years and have not had to be modified to
|> be readable by current FITS-reading programs.  That's a decade and a half,
|> anyway.  Given that a number of major archives, such as the HST archive,
|> use FITS as the archive format, I think we will easily see FITS files lasting
|> for decades.  However, I certainly would not rule out recasting this data
|> into another format at some time in the future.  Having it in FITS should
|> make this quite simple, should the time come.

I agree for images it is so and I hasten to add for single images this is
what I used and would recommend. Maybe when the archives become huge enough 
the benefits of compression will become more apparent. However what Don and 
you are suggesting is that this format is good for all data archiving. Clearly 
this is rubbish not least by your own statements later which show that there 
is a need to accomodate new data types.

|> > |>  AE> Publication in a journal ensures very little. 
|> > [brief lecture on the obvious deleted]
|> > 
|> > Again my point was that in so far as the user is concerned 2000000000000000
|> > publications won't help you with YOUR particular data unless the creator
|> > had taken care to ensure good documentation (as MetaData or paper). Hence
|> > my original assertion that good documentation is important.
|> 
|> Of course good documentation is important.  But I have to agree with Don that
|> unless it is part of the published literature it is vulnerable.

Surely what you mean is refereed literature. I have no idea how it can become
vunerable. Give me an example.
 
|> > |>  AE> FITS has gone through many revisions since the original paper..
|> > |> 
|> > |> Not so. The files in anonymous FTP directory fits.cv.nrao.edu
|> > |> /fits/testfiles/incunabula are examples of what FITS looked like in
|> > |> 1979-80; those files are still valid and are still readable by FITS
|> > |> readers all over the world.  There have been *no* revisions in the
|> > |> specifications of either Basic FITS or the Random-Groups format ...
|> > 
|> > From the dictionary:
|> > 
|> > re.vise \ri-'vi_-z\ vb
|> >   1: to look over something written in order to correct or improve
|> > 
|> > Are you suggesting FITS was not revised ? Clearly untrue.
|> 
|> FITS has been improved, namely, extended.  This is a critical distinction.
|> If you wish to use the term "revised" to mean "extended", fine.  But the
|> usual connotation of revised is "changed" or "corrected".

I don't know how to respond to this. The dictionary definition is quite
clear and I am writing in English. I mean "revised" as per its definition.
 
|> > |>  AE> .. users.. feedback.. help to shape things  rather than.. misplaced 
|> > |>  AE> trust [that] publication.. ensures.. error-innoculation..
|> > |> 
|> > |> The process used to formulate standards is critical. Extensive and
|> > |> time-consuming review and testing is essential before publication......
|> > 
|> > The original paper was refereed by 2 persons. I don't think this makes
|> > up a community :) At which point we come back to the true community who
|> > are the users which as I pointed out are the people who will shape things
|> > no matter how many papers are published.

[lots of useful information on how FITS evolves deleted]

|> It is this _process_ which is as critical to the development and acceptance
|> of a data format standard as is the technical merits of the format itself.

In other words its the user community that are important and not the publication
per say. Am I writing in English ?
 
[some lines deleted since they were "silly wordplay" according to an expert]

|> Don Wells' point, if I may
|> paraphrase, is that it would be considerably easier to properly read and
|> interpret a FITS file, owing to the immediate readability of its metadata,
|> than to properly read and interpret a binary structure in which the metadata
|> is encoded, without the help of external documentation.

In other words human-readable metadata included as part of the data is better 
than supplying software to read the metadata. Personally I prefer the s/w
approach since it saves me time in accessing the data. In so far as archiving 
is concerned I have already agreed that this is a very nice feature of FITS.

|> to these postings.  I am concerned primarily that incorrect or misleading
|> statements about FITS not be promulgated ...... 

I can't remember making any such statements. Please point them out since I 
find this assetion rather offensive.
 
|> ---
|> Robert J. Hanisch                       Tel. (410)338-4910   Fax. (410)338-5090
|> Space Telescope Science Institute       NSI: [email-masked] [130.167.1.2]
|> 3700 San Martin Drive                   DECNet: STSCIC::HANISCH [6549::] 
|> Baltimore, MD  21218  USA               Bitnet: hanisch@stsci
|> 

	adios
		Ata <(|)>.

---

From: [email-masked] (Edward Hartnett)
Date: 11 Aug 93 08:27:01

Great flame war guys! Who knew that data formats could be so exciting!
--

   Edward Hartnett	             [email-masked]
                                        (301) 286-2396

---

From: [email-masked] (Andy Jones)
Date: Wed, 11 Aug 1993 15:42:37 GMT

Excuse me a sec, but I would like to add my own two cents, since I think
some interesting points have been raised in this discussion. First, before
you all try to figure out what side of the issue I am on, I should add
that I have no FITS experience and only some HDF, netCDF experience, but gobs
of frustrating experiences with other data formats related to meterological
satellite data sets, radar data sets, and other surface observations.

HUMAN-READABLE ISSUE:

Does data have to be in human-readable form to make it a good archive
data format? I would say yes, BUT not necessarily in its native form
(i.e. I am very willing to run the data through a dump program to see
the data contents, whether this is on the OS or a special program I
could care less, just make sure the dump program is in some standard
conforming language and you've eliminated that problem). To expect that
all headers should be in ASCII form is very limiting, expand your
horizons, tagged formats are wonderful things.

side note: NCDUMP and NCGEN are nice handy utilities used originally with
NetCDF data files. There is a short article in IEEE Computers Graphics
and Applications, July 1990, p. 76-82 by Rew and Davis which describes
their function without having to go through a user's guide. This seems
like an easy way to have the human-readable information available,
but it is NOT a prefered way to scientificly to use the data set.
It is a convienence. The binary representations are much more efficient
for storing and accessing the data sets.

SIMPLIFICATION:

I think some of this discussion missed one of the earlier points. Aren't
these data formats supposed to SIMPLIFYING our access to the data? While
it may be in our human nature to expect that "our" data format will be able
to do everything for everybody, I think this is hopelessly wrong. To incorporate
other data formats into an umbrella data format would go against this
principle of simplification of the data formats. I can see instances of
where with another widely accepted data format it would be wise to
include other data formats if the other data format added capabilties
which did not already exist. But I think we should also be willing to live
with a handful of distict data formats (much better than the dozens I have
had to work with). What is pragmatic here? Total instant data format utopia,
or an orderly progression to just a SMALLER number of data formats?

COMPRESSION:

Someone mentioned that when data sets get large enough data compression
would become more important. I like to add that I would expect more data
formats are generated BECAUSE someone is trying to save space and write
their data in a more efficent manner. My own experience with the satellite
data sets has taught me that it is better to use a difficult unfriendly
data format (NON-FITS,HDF,NetCDF,CDF, etc. of course) than a more modern
flexible format mearly because it is then POSSIBLE to get my work done
on the machines we have. When you have more than 100 GB of data to process
you don't have the resources to wait for the expense of having flexible
data access. Compression is important NOW. You can have a wonderful
format, (personally I think HDF has recently progressed to a point of
becoming a viable satellite data archival format), but without compression
techinques and efficent access it will still be bypassed in favor of
the old bag of tricks that makes our work that much harder.

Andy.

P.S. I thought I read in the past that work is being undertaken to combine
the HDF and NetCDF interfaces. Is there a reference somewhere that somebody
could point me to?

************************************************************************
Andrew S. Jones                 phone: (303) 491-8448
CSU/CIRA                        fax:   (303) 491-8241
Foothills Campus                internet:
Fort Collins, CO 80523          [email-masked]
************************************************************************

---

From: [email-masked] (Andy Jones)
Date: Wed, 11 Aug 1993 15:42:37 GMT

HUMAN-READABLE ISSUE:

SIMPLIFICATION:

COMPRESSION:

Andy.

---

From: [email-masked] (Ata Etemadi)
Date: Wed, 11 Aug 93 19:44:41 BST

In article <[email-masked]>, [email-masked] (Andy Jones) writes:
|> Excuse me a sec, but I would like to add my own two cents, since I think
|> some interesting points have been raised in this discussion. First, before
|> you all try to figure out what side of the issue I am on, I should add
|> that I have no FITS experience and only some HDF, netCDF experience, but gobs
|> of frustrating experiences with other data formats related to meterological
|> satellite data sets, radar data sets, and other surface observations.

Amen. I am sick and tired of trying to decipher new formats and writing 
programs just to read and extract data with all sorts of darn special cases
and flags and headers and structures. What I am doing is writing filters
to convert as many data formats as I can find in my field (space physics)
to HDF. The reason being I'd only have to write analysis/vis programs for
a single data type. The important point is that all the formats in my field
are amenable to being put into HDF easily. Furthermore many of these formats
are "home-grown" so its not a proposal to replace some global standard with 
some other standard. Others are clearly going to make different choices 
for what they want to use. Please please lets have more tools to make using 
data simpler and to convert between formats.

|> 
|> HUMAN-READABLE ISSUE:
|> 
|> Does data have to be in human-readable form to make it a good archive
|> data format? I would say yes, BUT not necessarily in its native form
|> (i.e. I am very willing to run the data through a dump program to see
|> the data contents, whether this is on the OS or a special program I
|> could care less, just make sure the dump program is in some standard
|> conforming language and you've eliminated that problem). To expect that
|> all headers should be in ASCII form is very limiting, expand your
|> horizons, tagged formats are wonderful things.
|> 
|> side note: NCDUMP and NCGEN are nice handy utilities used originally with
|> NetCDF data files. There is a short article in IEEE Computers Graphics
|> and Applications, July 1990, p. 76-82 by Rew and Davis which describes
|> their function without having to go through a user's guide. This seems
|> like an easy way to have the human-readable information available,
|> but it is NOT a prefered way to scientificly to use the data set.
|> It is a convienence. The binary representations are much more efficient
|> for storing and accessing the data sets.

Again I agree. It would be nice if it was human readable but that does not 
mean a given format is not a good archive format because its not human-readable. 
Depends what your goals are. An archive format is whatever you decide will 
fit your particular needs as opposed to x or y is the best (hence the ??? in 
the subject). 

|> SIMPLIFICATION:
|> 
|> I think some of this discussion missed one of the earlier points. Aren't
|> these data formats supposed to SIMPLIFYING our access to the data? While
|> it may be in our human nature to expect that "our" data format will be able
|> to do everything for everybody, I think this is hopelessly wrong. To incorporate
|> other data formats into an umbrella data format would go against this
|> principle of simplification of the data formats. I can see instances of
|> where with another widely accepted data format it would be wise to
|> include other data formats if the other data format added capabilties
|> which did not already exist. But I think we should also be willing to live
|> with a handful of distict data formats (much better than the dozens I have
|> had to work with). What is pragmatic here? Total instant data format utopia,
|> or an orderly progression to just a SMALLER number of data formats?

Just lets not produce a "home-grown" format if one of the dozen or so standards 
will do. OK, its easier for the producer, but think of us users sometime.
 
|> COMPRESSION:
|> 
|> Someone mentioned that when data sets get large enough data compression
|> would become more important. I like to add that I would expect more data
|> formats are generated BECAUSE someone is trying to save space and write
|> their data in a more efficent manner. My own experience with the satellite
|> data sets has taught me that it is better to use a difficult unfriendly
|> data format (NON-FITS,HDF,NetCDF,CDF, etc. of course) than a more modern
|> flexible format mearly because it is then POSSIBLE to get my work done
|> on the machines we have. When you have more than 100 GB of data to process
|> you don't have the resources to wait for the expense of having flexible
|> data access. Compression is important NOW. You can have a wonderful
|> format, (personally I think HDF has recently progressed to a point of
|> becoming a viable satellite data archival format), but without compression
|> techinques and efficent access it will still be bypassed in favor of
|> the old bag of tricks that makes our work that much harder.

Since you can't pipe most of these formats its does make it a pain if the
compression is not part of the structure as it were (ie the software does
it for you without you needing to do anything about it). Note that given
the speed of processing these days compared with the speed of disc access
compression will most likely improve performance. I use gnu-zip compression 
since I know that people are going to download the data as opposed to access 
it directly. I would much prefer allowing people to access data and extract 
whatever bits interest them via gopher and WWW. Seems like a lot of folks 
are pointing to compression as an important feature. Hope people are taking 
notice. As a quick fix, I know there is disc compression software around
which means you are not aware that the data is kept compressed. However
this costs money and as far as I know its only available on PCs.

|> Andy.
|> 
|> P.S. I thought I read in the past that work is being undertaken to combine
|> the HDF and NetCDF interfaces. Is there a reference somewhere that somebody
|> could point me to?

HDF-netCDF (ie HDF version > 3.3) is what you want. Anonftp to ftp.ncsa.uiuc.edu 
and look in the HDF directory.

|> Andrew S. Jones                 phone: (303) 491-8448
|> CSU/CIRA                        fax:   (303) 491-8241
|> Foothills Campus                internet:
|> Fort Collins, CO 80523          [email-masked]
|> ************************************************************************

adios
	Ata <(|)>

---

From: [email-masked] (Marc Andreessen)
Date: Wed, 11 Aug 1993 21:08:01 GMT

In article <[email-masked]>
[email-masked] (Andy Jones) writes:

   P.S. I thought I read in the past that work is being undertaken to
   combine the HDF and NetCDF interfaces. Is there a reference
   somewhere that somebody could point me to?

HDF 3.3; see ftp.ncsa.uiuc.edu in /HDF.

Marc

--
Marc Andreessen
Software Development Group
National Center for Supercomputing Applications
[email-masked]

---

From: [email-masked] (BARRY M. SCHLESINGER)
Date: 11 Aug 1993 16:31 EDT

To save bandwidth, I'm clearing out some of the history. 

In article <[email-masked]>, [email-masked] writes...
>|> > (or is confusing).... 

The definition made in the posts by Don Wells for a self-describing
format was that information about the actual data objects, including,
but not restricted to, the organization, the names, the units, the
coordinate system, and the ancillary values, in short, the metadata,
is embedded in the files themselves in human and machine-readable
form.  Does HDF do this?  How? 

>... Not having used FITS for ages...

And yet claiming to be able to describe FITS and its merits as an 
archival format vis-a-vis HDF, basing the entire discussion on FITS as 
it was years ago.  Further discussion should be based on the FITS of 
today.  In particular, familiarity with the contents of the following 
two papers is necessary.

Grosbol, P., Harten, R. H., Greisen, E. W., and Wells, D. C., 
"Generalized extensions and blocking factors for FITS," Astronomy and 
Astrophysics Supplement Series, 73, 359-364, 1988. 

Harten, R. H., Grosbol. P., Greisen, E. W., and Wells, D. C., "The 
FITS tables extension, Astronomy and Astrophysics Supplement Series, 
73, 365-372, 1988. 

Also necessary to understanding the FITS of today is a reading of the 
binary tables proposal, available in PostScript and LaTeX forms by 
anonymous ftp from fits.cv.nrao.edu in the directory 
fits/documents/proposals.  Note that the proposal is an a form 
appropriate for publication; thus, following approval by the IAU, it 
can be submitted to a refereed journal. 

The NASA/Science Office of Standards and Technology has two documents
available that provide a comprehensive introduction to the FITS of 
today, the Definition of FITS, the NOST Standard that is a formal
codification of the rules of FITS, and the User's Guide, which
includes not only the rules, but a discussion of FITS history,
recommended practices and directions in which FITS is evolving.  More
information can be obtained by anonymous ftp to nssdca.gsfc.nasa.gov, 
starting with the AAREADME.DOC and BASICS_INFO.TXT files.

> I have some sparse matrix data is this supported by FITS ? 

Binary tables.  High energy astrophysics groups are routinely using 
binary tables for data of this kind; in fact, it was the need for such 
a structure that has spurred the development of this extension.

> ... Get REAL as you Americans say.

I fail to understand what this comment contributes to the discourse.

>... For some applications

As I recall, the question was about which are most suitable for 
archival.
			...
>||> > within it ? Or are you suggesting that this is impossible or impractical ?

In June, 1992,  NOST sponsored a workshop to which were invited
leading developers and representatives of a number of methods for
organizing and labeling scientific data: FITS, HDF, CDF, netCDF, PDS,
BUFR/GRIB, SDTS, the CEOS superstructure, and SFDU.  While it would
not be appropriate to discuss any conclusions of the attendees until
the meeting report is published, I would note that there were a number
of discussions about ways in which developers of the different formats
could work together.  These discussions were in the sense that
different formats serve different needs, but that there were gains to
be made from cooperation and collaboration.  One cannot say in an
absolute sense that one is better than another; it depends upon the
intended purpose. 

> 
>|> > In particular the support for time-series 

Binary tables again, in particular the "3D" feature, allowing the 
storing of arrays in table fields.

>Should I wait

FITS was designed originally as a data format for astronomy, although 
other applications have never been ruled out.  That the governing body 
is a group of scientists is an important contributor the acceptance
FITS has gained among astronomers.  NASA has decided that data it
archives should be in standard formats appropriate to the fields. 
Astronomical data must be in FITS format.  That a format had already
been adopted by people in the field made this choice easier and more
likely of acceptance. 

> 
>... Get off your high horse. 

I missed the substantive point here.

>... For me it is not big problem since 

The problem of loss of access to the documentation -- the orphaned 
tape for which the paper description has long since disappeared -- is 
internationally recognized.  That is why CCSDS (Consultative Committee
for Space Data Systems; the French name has the same acronym) group
including representatives of space agencies of many nations, is
developing the concept of Standard Formatted Data Units.  This concept
is a system whereby metadata are embedded with the data, in a
prescribed form.  All data descriptions referred to in the metadata 
are at locations which can be determined from the information in the 
metadata.  A Control Authority system controls where the documents are 
and ensures that these locations are consistent with the data.

>clear and I am writing in English. I mean "revised" as per its definition.
Precisely because "revised" has this connotation of "changed" the 
NOST, at the insistence of FITS leadership, has not used the word in 
connection with the evolution of FITS.  One can't just go by the 
formal dictionary definition.  One has to understand the connotation 
in practical use.  That's why there's a User's Guide, which discusses 
FITS practices, in addition to the formal standard.  That's why 
people who try to start new newsgroups based solely on reading the 
guidelines, without six months exposure to news.groups, get into 
trouble.   

>find this assetion rather offensive.

"FITS has undergone many revisions"

That's discussed above.

"Are you saying that the FITS format contains the definition of 
primitives..."

Implying that it doesn't describe the included datatypes.

"Not all astronomical data is gridded.  In particular the support for 
time series is poor.

Implying that FITS is still where it was in 1982, in particular, 
completely ignorant of binary tables.

"I doubt very much that the original FITS files will last decades."

Contradicted by the rule that no change may be made that will cause a 
valid FITS file to go out of conformance.

"The original paper was refereed by 2 persons.  I don't think this 
makes up a community."

Completely misrepresenting the process by which FITS actually evolves, 
later explained in the posting by Bob Hanisch.

				Barry M. Schlesinger
				NSSDC/NOST FITS Support Office

				NCF::FITS
				[email-masked]

				+1-301-513-1634

---

From: [email-masked] (BARRY M. SCHLESINGER)
Date: 11 Aug 1993 16:31 EDT

That's discussed above.

				Barry M. Schlesinger

				NCF::FITS

				+1-301-513-1634

---

From: [email-masked] (Ata Etemadi)
Date: Thu, 12 Aug 93 16:58:38 BST

In article <[email-masked]>, [email-masked] (BARRY M. SCHLESINGER) writes:
|> To save bandwidth, I'm clearing out some of the history. 

I am sorry to see the history you "cleared" up was so selective.

|> The definition made in the posts by Don Wells for a self-describing
|> format was that information about the actual data objects, including,
|> but not restricted to, the organization, ....... is embedded in the 
|> files themselves in human and machine-readable form.  Does HDF do this?  How? 

HDF-netCDF has these precise mechanisms. However to make it human readable
one has to use a program. I fail to see what human-readability has to do
with self-describing ala what you say here. I also note that the last two
statements are questions and not assertions and I am dealing with them in 
this context. You however frequently, and on purpose, confuse these issues 
in the subsequent lines. Where I asked a question you translate said question 
to an assertion which I find somewhat of a low blow.

|> >... Not having used FITS for ages...
|> 
|> And yet claiming to be able to describe FITS and its merits as an 
|> archival format vis-a-vis HDF, basing the entire discussion on FITS as 
|> it was years ago.  

Your reading seems to be somewhat selective and you also seem to be somewhat
selective with the truth. I have not been making any such claims. My claim 
was that a single format does not exists that is all things to all men. I
pointed out that an archival format is in the eye of the beholder and HDF
has some very nice features which also make it suitable for this purpose. It
was messrs Wells et al. who assumed this was some sort of attack on FITS.
READ MY ORIGINAL POSTING and you see I do not single out FITS for anything. 
Just as a matter of interest let me also point out to you, as you and
Wells et al. keep pointing out to me, that FITS of ages ago is not supposed 
to have undergone any revisions (however you may wish to define it -- P.S. 
see, no reference to dictionary John ;-) so I do believe you are somewhat 
confused as to what you are trying to say here.

|> > I have some sparse matrix data is this supported by FITS ? 
|> 
|> Binary tables.............

I note that you dropped the reference to MPEG. Just in case you were going to
ask what I meant by selective, this is one of many examples.

|> >... For some applications
|> >FITS is very good for others HDF is very good for some CDF is very good etc..
|> 
|> As I recall, the question was about which are most suitable for archival.

And it was in this context I was using the word "applications". Just to make it 
clearer for you I believe in some cases HDF-netCDF is a very good archive format.
As a quick aside I currently don't think its as good as FITS for image archiving 
and I don't believe FITS is as good as HDF-netCDF, or CDF for archiving 
space-science data (seems like NASA agree with with me). However I am going to 
check out the Binary tables feature.

|> >|> > Are the FITS folks willing to collaborate with HDF to incorporate their format 
|> >||> > within it ? Or are you suggesting that this is impossible or impractical ?
|> 
|> In June, 1992,  NOST sponsored a workshop to which were invited
|> leading developers and representatives of a number of methods for
|> organizing and labeling scientific data: FITS, HDF, CDF, netCDF, PDS,
|> BUFR/GRIB, SDTS, the CEOS superstructure, and SFDU................

Great!! 

|> One cannot say in an absolute sense that one is better than another; it 
|> depends upon the intended purpose.

I do hate it when people agree with me and pass it off as an argument.

|> >|> > In particular the support for time-series 
|> >|> > is not good (I am being kind here) unless things have changed.
|> 
|> Binary tables again, in particular the "3D" feature, allowing the 
|> storing of arrays in table fields.

Thanks for answering my question. I will look into this feature. It may
be that I can now use FITS for things other than images. I also hope
to find Binary tables are easy to use and I can convert from this format
to other formats easily.
 
|> FITS was designed originally as a data format for astronomy, .....
|> NASA has decided that data it
|> archives should be in standard formats appropriate to the fields. 
|> Astronomical data must be in FITS format.............

Seems like you are arguing by agreement again. Clearly the ISTP chose 
CDF as their archival format since they believed it suited their data
and archival purposes better than FITS or HDF etc.. I sure hope that
we see more collaboration between folks in the netCDF, HDF, CDF, communities 
to make it easier for us to do science and worry less about the data format.

[references to the ISTP Brown Book deleted]

>find this assetion rather offensive.
|>
|> "FITS has undergone many revisions"
|> 
|> That's discussed above.

It is discussed above, and I don't see that I made any incorrect or misleading
statements given I used the dictionary definition (sorry John) and you used the
"practical" definition of revision.
 
|> "Are you saying that the FITS format contains the definition of 
|> primitives..."
|> 
|> Implying that it doesn't describe the included datatypes.

This was a ***question****. Also I was referring to long, integer etc..
This is another example of being selective. Where is the darn question 
mark ? <--- like this one
 
|> "Not all astronomical data is gridded.  In particular the support for 
|> time series is poor.
|> 
|> Implying that FITS is still where it was in 1982, in particular, 
|> completely ignorant of binary tables.

Again you are being selective. I am not sure now I should have even lowered 
myself to answer such blatant rubbish. The part you deleted from my statement 
was "unless things have changed".

|> "I doubt very much that the original FITS files will last decades."
|> 
|> Contradicted by the rule that no change may be made that will cause a 
|> valid FITS file to go out of conformance.

Who said anything about changing the format. I was referring to the ***files***
not the format. I sure hope that in 100 years time all these FITS images
are still around. But then I don't have your crystal ball. Does this mean
compression is completely ruled out as far as FITS is concerned ? This is a 
question.

|> "The original paper was refereed by 2 persons.  I don't think this 
|> makes up a community."
|> 
|> Completely misrepresenting the process by which FITS actually evolves, 
|> later explained in the posting by Bob Hanisch.

Again selective. You deleted all the references I made to the fact that
I believed it was the user community that was important and not the journal
paper. I have to say I am disgusted and rather angry about your misquotations
and slander. Fortunately the previous articles are available for all to see.
Below I have appended a copy of my original posting. Take time to read it
Barry, you may learn something.

Ata <(|)>.

BEGIN APPEND MODE:

G'Day

        adios
                Ata <(|)>.