Subject: Re: HDF, netCDF

From: Ethan Alpert <[email-masked]>
Date: 14 Feb 1996 17:37:34 GMT

[email-masked] (Edward Hartnett) wrote:
>>>>>> "J" == Joel Parriott <[email-masked]> writes:
>
>    J> I'm currently running Fortran simulations which produce multi-dimensional arrays
[deleted: wants to move data from IBM RS6000 to Sun]

>    J> of HDF or netCDF.

>I hate to be a data format heritic, but why can't you use binary?
>Can't both machines write IEEE? I would have thought that this would
>be the default output on both machines, but even if it is not, does
>the IBM compiler not even offer such an option?

Not all machines are IEEE binary. I believe the IBM machines DON'T use a
hidden bit, but I might be wrong wrt AIX. This would make writing data on the
IBM and moving it to the Sun difficult. Binary, regardless of the IEEE
standard,
is still very unportable. There are the DEC byte swapped machines and now you 
have 64 bit longs on some machines. CRAY's don't write IEEE by default, its a 
mess.

>
>ASCII obviously stinks. No way!

I agree.
>
>(Again, playing the data format heritic) you could take this
>oppurtunity to pack your data into 16 byte integers. It's a fairly
>simple process. (I can send you a description of two possible methods
>if you're interested).

This would be nice and portable but think about the unpacking code given, the
lack of standards, on the systems I mentioned before, would make this code
highly machine dependent. GRIB packs data into integers, sort of, and my code
to
unpack the data has to be diferent on SUN's, CRAYS, and IBM's. So you don't
gain 
anything here. Furthermore, the original poster's code is FORTRAN. I'd HATE
to have to do all this bit-twidling in FORTRAN, yech!

Anyways I'm really responding here to make a point that you are missing the
point of "standard" data formats like HDF and netCDF. My point is the
interface means everything and the format means nothing. For example, let's
say you write your 16 bit integer packed format into a file. Two years later
you find the binary file sitting in your directory. How do you know what the
data is, what the units are, when you created it, what its dimensionality is
and 
most importantly how do you know how its packed and where the code you wrote to 
unpack it is? 

"Standard" data formats alleviate these problems and provide you a way to
store with your data information describing your data. Assign your data
varibles
and dimensions names, etc. But most importantly they give you a uniform
interfacefor writing, reading and inquiring about data, that also is portable.

These formats open the doors to develope usefull tools to browse and manipulate
data. For example, here at NCAR I've worked on developing an interpreted
language that is an example of what can be built using these kinds of standard
interfaces.(http://ngwww.ucar.edu/ngdoc/ng4.0/ref/ncl/NclIntr.html). I'm
currently adding the GRIB format to NCL. GRIB does not have an interface and it 
has been quite a chore to create one because of the forementioned binary 
difference between architectures. 

-ethan

>
>
>--
>Edward Hartnett                 [email-masked]
>* I don't speak for NASA or ARC, I just work for them! *
>     Goddard Distributed Active Archive Center (DAAC)
>Check out cool Earth Science data at: http://daac.gsfc.nasa.gov/
>

-- 
-------------------------------------------------------------------------------
Ethan Alpert						[email-masked]
-------------------------------------------------------------------------------