Subject: Re: Why this many formats.

From: [email-masked] (Keith Grant)
Date: 3 Jun 1993 14:20:25 -0700

In article <[email-masked]>, [email-masked] (Venkata Chaganti) writes:
|> I would like to know why there are a lot of data formats like
|> cdf,hdf,netcdf, fits,etc.
|> 
|> it seems all are common in some sense, but some are standardized for some 
|> applications?
|> 
|> Why should not we use the same for different areas.?
|> 
|> If you use any of two please compare them, if comparable?
|> 
|> Is any of these works like relational databases?
|> 

To greatly simplify, because the formats were originally developed to
serve different needs.

Each format has different strengths and limitations.  The following is a list
of questions you might ask in evaluating whether a particular format meets
your data-handling needs.
---------

Does a format store image data as well as strictly numerical data?

What abilities does the format and its interface have for including
data about the data (i.e. meta data).  Is this meta data global to a file
can it also include parts that vary between data items.

Is the format portable between a large number of different platforms?

Does the format support different types of data: 32-bit floating, 64-bit
floating, integer, character, a standard time representation, ...  Can these
types be intermixed in a single file?

Does a format allow arrays of different dimensionality and sizes to be
written in the same file?  Can an array of data be written in increments with
other data being added in between?  Are there restrictions are the amount
or variations in the amount of data written in between increments of an 
array?

How difficult is it to add new data to an existing file or to modify the
structure of an existing file?

Does the format have an interface for more than one programming language
(e.g. FORTRAN and C)?

Is the format used by third party analysis tools (MatLab, IDL, 
Mathematica, ...)?

-- 
  +-------------------------+-----------------------------------------------+
  I                         I Common sense and a sense of humor are the     I
  I Keith Eric Grant        I same thing, moving at different speeds.  A    I
  I [email-masked] I sense of humor is just common sense, dancing. I
  I                         I (or perhaps dancing is just common sense)     I
  +-------------------------+-----------------------------------------------+
  I For certain it is, that any ideas expressed above are of my own humble  I
  I opinion and bear nary a relation to the policies or positions of LLNL   I
  I or of any agency or contractor of the U.S. Government.                  I
  +-------------------------------------------------------------------------+

---

From: [email-masked] (Keith Grant)
Date: 4 Jun 1993 08:48:51 -0700

In article <[email-masked]>, [email-masked] (Roland Karlsson) writes:
|> 
|> > > I would like to know why there are a lot of data formats like
|> > > cdf,hdf,netcdf, fits,etc.
|> 
|> > To greatly simplify, because the formats were originally developed to
|> > serve different needs.
|> 
|> I do not think that this is the main reason why.  Sure, there exist
|> some reasons for using different formats.  But I think that the main
|> reason for all those formats is that there have been lots of picture
|> hacking going around in the world.  It is quite easy to construct your
|> own picture format.  It can only be made in so many ways.  So, when no
|> standard existed, everyone invented his own.  Most formats are totally
|> unnecessary and can be replaced by some other.  A kind of "proof" is
|> that there exist the 'portable pixmap format'.  Most other formats can
|> be converted to this format and then converted back without any loss
|> of information.
|> 
The diversity of formats for storing pictures is certainly true, as is the
past lack of standards for either picture storage or other needs.  In
addition to picture storage, formats have been devised to fill other 
data needs.  These include archiving data derived from satellite and
measurement campaign observations and output of the results of large
model calculations.  The requirements of self-discription, machine 
portability, ease of creation, and usability.

As an example, the ease of creation could vary significantly from one format
to another depending on whether you have a complete data set to achive when
the file is created or whether you have a model which is creating and 
interspersing parts of different data sets as it runs.
-- 
  +-------------------------+-----------------------------------------------+
  I                         I Common sense and a sense of humor are the     I
  I Keith Eric Grant        I same thing, moving at different speeds.  A    I
  I [email-masked] I sense of humor is just common sense, dancing. I
  I                         I (or perhaps dancing is just common sense)     I
  +-------------------------+-----------------------------------------------+
  I For certain it is, that any ideas expressed above are of my own humble  I
  I opinion and bear nary a relation to the policies or positions of LLNL   I
  I or of any agency or contractor of the U.S. Government.                  I
  +-------------------------------------------------------------------------+

---

From: [email-masked] (Bear Giles)
Date: Fri, 4 Jun 1993 22:45:55 GMT

>|> Most formats are totally

Reread the previous comment.

"Portable pixmap format" is great for portability, since it is an ASCII
representation of the color components of an image, but it requires an
_immense_ amount of space.

The binary equivalence (e.g., Sun raster files) are easy to manipulate
but still take a lot of space.  But they aren't quite a portable -- 
being binary data instead of ASCII text.

Lossless compressed images (e.g., .GIF) don't take as much room, but
are nearly impossible to manipulate _in situ_ except for color-table
games.

Lossy compressed images (e.g., JPEG) take the least amount of space,
but may introduce artifacts which can prevent their use in scientific
analyses.  They also take longer to compress and decompress.

Still not convinced?  Large systems (e.g, a Sun workstation) can handle
a large image in core at once.  So 'monoblock' formats like Sun raster
format can be used.  However, you may want to process a high-resolution
SVGA image (1024x768) on a PC with a segmented memory model and less
than 500k free, so you'll need a "stripped" image format like TIFF format.

However "stripped" images introduce a lot of unneccesary work if you have
a large linear memory space, so most people want to avoid them on those
system....

-- 
Bear Giles
[email-masked]
-<My views rarely reflect those of my employer or the Federal government>-

---

From: [email-masked] (Moss A D)
Date: 11 Jun 93 11:35:35 GMT

In article <[email-masked]> [email-masked] (Joel Plutchak) writes:
   In article <[email-masked]> [email-masked] (Bear Giles) writes:

[some stuff deleted]

   >_immense_ amount of space.

      And that's only a part of the picture-- at the risk of being dense,
   just the "picture" part.  Many images, e.g. the planetary images I
   work with, have a whole bunch of auxiliary data associated with them.
   Most of the image formats which come out of various hardware-specific
   worlds (e.g. sunraster, TIFF, GIF) have little or no allowance for
   such data.  Without the auxiliary stuff, all we have is a pretty
   picture.

I have to differ with you concerning your labelling of the GIF format...
The GIF format didn't come out of a "hardware-specific world" - GIF
stands for Graphics Interchange Format (tm) :) and, as the name
implies, it's not tied to a particular hardware world.
What's more, version 89a of the GIF format does support various
miscellaneous Extension Blocks, which might be suitable
for holding all your favourite "auxiliary stuff" :)

Later,
	Adam.

   --
   Joel Plutchak, Research Programmer/Analyst
   "Dies ist meine 12305te Nacht/Die erste paar tausend,
    die kannst du vergessen/ich hab' es auch getan" - EN
--

--------------------------------------
LOOK AFTER MY SHED

---

From: [email-masked] (Joel Plutchak)
Date: 11 Jun 1993 13:43:12 GMT

>implies, it's not tied to a particular hardware world.

   Yes, I was a bit brief in my wording.  It came out of a specific
commercial application-- Compuserve-- and still is mostly used by
the PC/BBS community.  (Yes, yes, I know about things like 'xv',
and my own Unix/VMS software reads GIF images.  We're talking
generalities here.)

>for holding all your favourite "auxiliary stuff" :)

   Not really.  I can put whatever I want in those extension
blocks, but almost everybody ignores them, and there's no control
over the formatting of specific information.  Even for closely-
controlled specs like that for JPEG files we end up with vendor-
introduced incompatibilities.  There's no way loosely-defined
and -controlled things like the GIF89a extension blocks can ever
be useful to a wide range of people doing real-world things.
--
Joel Plutchak, Research Programmer/Analyst for hire
"Dies ist meine 12305te Nacht/Die erste paar tausend,
 die kannst du vergessen/ich hab' es auch getan" - EN