Subject: more FITS fits

From: [email-masked] (John Bates)
Date: Wed, 18 Aug 1993 18:28:24 GMT

Archie Warnock's comments on the FITS vs HDF have me even more confused.
As I science user of the data, I completely and utterly disagree that the
archive format should be discussed totally seperate from a usable format.
WRONG!!!  This is the source of most of my problems in working with 
operational environmental satellite data over the past 15 years.  Scientists
are the CUSTOMERS, not the archivists!!!  

I would also urge those commenting on this discussion to cut through the
BS.  What is the purpose of this discussion?  Is it to have FITS replace HDF
as the EOSDIS standard?  And those that comment, do you have a vested
interest in a particular format (i.e., is your job directly related
to one of these formats?).  My job is not.  In fact, I prefer netCDF, but
since NCSA has done such a good job incorporating netCDF into HDF, HDF
is fine too.

>  I was there.  The process was rigged from the start. 

>I'll try to put together a set of specific responses to some of the
>issues raised in individual messages in a day or so - this is long
>enough. 

>OK, let's try and recap what this discussion started out to accomplish
>and see if we can get back in that direction.  I'll try to use small
>words so no one will have to refer to a dictionary :-)

Very patronizing and insulting...there is no need for this crap...

I vote no confidence in a summary of the FITS fits by a biased party.
Mr Warnock, please do not waste your time.

John Bates, data user...

---

From: [email-masked] (Greg Hennessy)
Date: Wed, 18 Aug 1993 19:06:23 GMT

In article <[email-masked]>,
John Bates <[email-masked]> wrote:
#As I science user of the data, I completely and utterly disagree that the
#archive format should be discussed totally seperate from a usable format.
#WRONG!!!  This is the source of most of my problems in working with 
#operational environmental satellite data over the past 15 years.  Scientists
#are the CUSTOMERS, not the archivists!!!  

I have often worked with archival data, and I am a scientist. 

The format of the data which is most convient to work with today is
almost certianly *NOT* the format of the data that someone who is
trying read a "dusty tape" 40 years from now is going to need to know
to understand the data. 

I have read in data from instruments that no longer exist, but were in
FITS format, and I was easily able to reconstruct the data when the
scientist who TOOK the data could not give me any references to what
was on the tape. 

#Very patronizing and insulting...there is no need for this crap...
#
#I vote no confidence in a summary of the FITS fits by a biased party.
#Mr Warnock, please do not waste your time.

Mr. Bates, the only patronizing and insulting I have seen in this
"discussion" has been from you. 

--
-Greg Hennessy, University of Virginia
 USPS Mail:     Astronomy Department, Charlottesville, VA 22903-2475 USA
 Internet:      [email-masked]  
 UUCP:		...!uunet!virginia!gsh7w

---

From: [email-masked] (Louis Giglio)
Date: Wed, 18 Aug 1993 22:13:42 GMT

In article <[email-masked]>,
John Bates <[email-masked]> wrote:
|> #As I science user of the data, I completely and utterly disagree that the
|> #archive format should be discussed totally seperate from a usable format.
|> #WRONG!!!  This is the source of most of my problems in working with 
|> #operational environmental satellite data over the past 15 years. 
Scientists
|> #are the CUSTOMERS, not the archivists!!!  

I'm not really sure what this means.  Are we supposed to pay some
mystery person to put all of our satellite data in a universal archive
format so that we don't have to worry about it?  We (scientists) archive
our own satellite data (SSM/I and GMS in my particular branch) as we
receive it, so we're not exactly customers, but in fact "archivists." 
Maybe I've misunderstood you, but I don't see how scientists
shouldn't/can't be deeply involved in the archive issue.

|> #I vote no confidence in a summary of the FITS fits by a biased
party.
|> #Mr Warnock, please do not waste your time.

Having used FITS along with too many other formats, I don't think that
Mr./Dr. Warnock is out of line.  Everything he's said so far is pretty
much on the mark.  Perhaps I missed something?

Louis Giglio
NASA/GSFC

---

From: [email-masked] (Ata Etemadi)
Date: Thu, 19 Aug 93 16:46:16 BST

In reply to: [email-masked] (Greg Hennessy)
        and: [email-masked] (Louis Giglio)
>trying read a "dusty tape" 40 years from now is going to need to know

I think there are some MAJOR points you have missed here, which is also 
illustrated by your storey (yawn) about the tape from some scientist who 
was nice enough to put it in FITS format. There is a big difference between 
home-grown formats (which cause all the head-aches with old tapes) and 
formats such as ->MPEG<-, HDF, CDF, netCDF etc.. which have a very wide 
user-base. Another MAJOR point is that an archive is supposed to be maintained 
and used by lots of people. A lone scientist sticking data on tape does
not constitute an archive unless you have a very silly definition of archive.
Its the responsibility of an archive maintainer to make sure the data is
usable, stable and to recast if and when necessary. Also some formats
are not going to fit into FITS easily and it would be considered idiotic
to do so for many reasons eg MPEG. Finally I am in full agreement with the 
EOSDIS folks for the choice of HDF and also the choices of CDF and netCDF 
by other institutes in my field since these formats allow you to do science 
much much more quickly, efficiently and scientific work is what we use archived 
data for. When the archivists are planning to all commit seppuku or when
the data is no longer getting used reasonably often I suggest they should
then consider using FITS or even better putting together good documentation 
in ASCII with the data and on paper. If you ARE a lone scientists with
the need to archive data then by all means use FITS. Finally I think Mr. 
Bates remarks regarding "Very patronizing and insulting..." were quite correct 
unless ofcourse you were waiting for Mr./Dr. Warnock to come back from 
holiday and point out the obvious. I would guess also that some folks DID 
miss something namely the points I have made above.

	adios
		Ata <(|)>.

---

From: [email-masked] (Pat Murphy)
Date: Fri, 20 Aug 1993 16:06:22 GMT

In article <[email-masked]>, [email-masked]
   (Ata Etemadi) writes (among other things): 

AE> There is a big difference between home-grown formats (which cause
AE> all the head-aches with old tapes) and formats such as ->MPEG<-,

I've noticed you emphasizing MPEG here before.  Am I correct in assuming
that it uses a lossy compression algorithm?  If so and if there is no
way to get around that without compromising or eliminating the
compression advantage, I cannot see MPEG as being suitable for
astronomical data.  To us, the noise is important and not to be messed
with.  If my assumptions about MPEG are not correct, please let me know
and elaborate with details.

AE> A lone scientist sticking data on tape does not constitute an
AE> archive unless you have a very silly definition of archive.

I shouldn't.  I _really_ should let this slip, but...

I beg your pardon???  Do you mean a scientist who is stuck out in the
booneys in a small liberal arts college performing valuable research and
attempting to preserve the results for posterity and perhaps future
researchers?  

AE> Its the responsibility of an archive maintainer to make sure the
AE> data is usable, stable and to recast if and when necessary.

 ...if one has the luxury of being able to afford an archive maintainer.
A lot of science is done on a shoestring budget.

AE> Finally I am in full agreement with the EOSDIS folks for the choice
AE> of HDF and also the choices of CDF and netCDF by other institutes in
AE> my field since these formats allow you to do science much much more
AE> quickly, efficiently and scientific work is what we use archived
AE> data for.

Likewise for any astronomical researcher; all the major reduction
packages make it easy to read data in FITS format (AIPS for radio
astronomy, IRAF for optical, STSDAS for Space Telescope data, SAOimage
for immediate X11 display, etc.)  The key phrase here is "IN MY FIELD".
                                                          ~~~~~~~~~~~
Getting astronomical data in HDF format would certainly make life
difficult for most astronomical researchers that I know.

Can we stop the emotional flames and move on to some useful discussion
about METADATA?  I challenge you to respond constructively and
rationally to this post, e.g. by summarizing for us who don't know about
HDF _how_ it might be suited to astronomical or other data.

				- Pat
--
==========================================================================
| Patrick P. Murphy, Ph.D.                Scientific Programming Analyst |
| National Radio Astronomy Observatory    Net:       [email-masked]    |
| 520 Edgemont Road                       Phone:     (804) 296-0372      |
| Charlottesville, VA 22903-2475          VoiceMail: (804) 980-5889      |
|      "I don't believe in the no-win scenario"  --- James T. Kirk       |
==========================================================================

---

From: [email-masked] (Archie Warnock)
Date: 23 Aug 93 10:00:15 GMT

>and elaborate with details.

MPEG is a lossy encoding scheme for animations.  It's a picture format,
not a data format.  As far as I'm aware, it has no provisions for
carrying along metadata.

>HDF _how_ it might be suited to astronomical or other data.

Actually, Pat, HDF _could_ be used for astronomical data.  It's an
eminently useful format for data interchange and manipulation.  Like
FITS, PDS, CDF and a host of others, it has strengths and weaknesses but
the NCSA developers have done a good job.

The discussion of how it might be suitable starts a new thread, however,
since it addresses issues other than it's suitability for long-term
archival storage.
--
_______________________________________________________________________
-- Archie Warnock              Internet:  [email-masked]
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC                   Project STELAR: WAIS to do science