Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought

From: "Donald W. Denbo" <[email-masked]>
Date: Tue, 01 Oct 1996 08:07:14 -0700

Dave Senciall wrote:
> 
> 
> My questions are :
> 
> 1) Are there any other good formats/softwares that I have missed that may be
> of use.
> 
> 2) If you have used any of these (netcdf HDF other) can you comment on their
> suitability  (or more importantly UNsuitability) for my purposes.
> 
> Any suggestions comments or criticisms are welcome and appreciated..
> 
>                                         Dave Senciall

Dave,
	You might take a look at http://www.pmel.noaa.gov/epic/home.html
EPIC is a collection of analysis tools that use the netCDF format.  EPIC is
designed for use with oceanographic data sets and new tools are being brought
on line all the time.  The software is freely available from anonymous ftp
at ftp://ftp.pmel.noaa.gov/epic.

Don Denbo
NOAA/PMEL/OCRD
206 526-4487
[email-masked]
or
[email-masked]

---

From: [email-masked] (Robert Grumbine)
Date: 3 Oct 1996 12:40:42 -0400

In article <[email-masked]>,
Don Eliason <[email-masked]> wrote:
>
>In my opinion, netCDF is fine for oceanography data, but more so for 2D and
>3D data (e.g., the Levitus data sets; model output) than for station data.
>The JGOFS data format may be better for station data.  The ECMWF uses Grib
>and/or Grads (difference = ?).  

  Grib is the World Meteorological Organization's standard for gridded
fields.  It isn't a very good thing for station data.  For that, the
WMO standard is BUFR.  Grads is a freely available software package
for graphic display.  Among other things, it can read grib files.
Bufr is probably a good design for the ocean station data since 
atmospheric soundings were one of the drivers in bufr's development.

  Having spoken of grib, one of the things about grib is that one can
have a different mapping of parameter number to physical variable.
I maintain such a table and would appreciate suggestions on additions.
The table (and some other ocean related material) is at 
http://polar.wwb.noaa.gov/.  The page, of course, is under development
with significant changes and additions expected in the next copule of
weeks.

-- 
Bob Grumbine [email-masked]
Sagredo (Galileo Galilei) "You present these recondite matters with too much 
evidence and ease; this great facility makes them less appreciated than they 
would be had they been presented in a more abstruse manner." Two New Sciences

---

From: Chris _topher_ Lakey <[email-masked]>
Date: Thu, 03 Oct 1996 17:51:34 -0500

Another option would be WMO GRIB.  The navy seems to be pretty heavy
into GRIB right now.  It's worth checking out.

I've been researching this lately and I think the netCDF data model
is much better than the HDF SDS data model, but I don't know much about
the netCDF binary data format.  HDF is fairly standard (adopted by EOS)
so you might want to consider using the netCDF features of HDF.

The NCAR Graphics package uses the netCDF data model as the basis for
their data model, so if you have access to NCAR Graphics 
(version 4+ I think) you might want to consider netCDF.  Also, netCDF
has an ASCII format (netCDL) that's very readable.  If you're doing
any type of distributed stuff, check out 

http://dods.gso.uri.edu/DODS/
The Distributed Oceanographic Data System 

They provide some distributed versions of the netCDF API.  Ok, while
I'm rambling, netCDF has APIs for several languages including C++.

Good luck!

-topher

-- 
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 Chris "topher" Lakey          "Only two things that money can't buy
 mailto:[email-masked]       That's true love & homegrown tomatoes"
 http://www.pobox.com/~topher/ http://www.pobox.com/~topher/guyclark/
 (205) 534-4844 FAX: 534-4846  ***      THIS SPACE FOR RENT       ***
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

---

From: Dave Senciall <[email-masked]>
Date: 2 Oct 1996 19:02:39 GMT

rich pawlowicz <[email-masked]> wrote:
>Dave Senciall wrote:
>> 
>> I'm in the process of evaluating options for reworking our oceanographic
>> data archives, (CTD, current meter, T-chain, ADCP, timeseries thermographs
>> etc).
> 
> 
>>   3) compact binary storage.
>>   4) some degree of platform independence for the files/software (we use Sun -
>>       unix for are archives but do a lot of work on the PC, also DEC ALPHA
>>      /openVMS may creep into the picture)
>                                 
>Just curious - but why binary storage? The amount of ascii data 
>you have in even a very comprehensive
>CTD/current meter dataset is not all that large, comparatively. The storage
>breakpoint between binary CTD + (e.g.) netCDF libraries vs. ascii CTD is fairly
>high.
>
>For example, I have in one directory 250 CTD files in IOS header-file format
>(more header info than data in this format!); they take up 2M. The HDF
>distribution (which I happen to have around for some reason or other)
>appears to take up something like 13M...and as a user, I much
>prefer being able to look at ascii files.
>
>
>
>
>--
>Rich Pawlowicz,             ph: (604) 363-6339  fax: (604) 363-6798 
>Acoustical Oceanography Research Group, Institute of Ocean Sciences,  
>PO Box 6000, 9860 West Saanich Road,  Sidney, B.C. CANADA   V8L 4B2  
>email: [email-masked]      WWW: http://pinger.ios.bc.ca/people/rich

Rich,
  I have approximately 450,000 Ctd/xbt/bottle files (not counting other 
instrument types(30 ADCP cruises,c-meter,t-graphs)) ranging in size from a
a single level to 10,000 levels, at ,say, 6 bytes per value in ascii
(eg 32.456) for pressure temp sal plus derived channels that adds up to a
lot of disk space, the same value (32.456) could be stored as 32456 in a 2
byte unsigned int, giving a saving of 66% not counting the white spaces between
data columns.

  One of my concerns is that netcdf or hdf would be gready since they are
general. And I might be better served using my own formats if space and
speed were my biggest concerns.
  One of my goals however is to bring a 'standard theme' to the data sets.
I hope that netcdf or similar might provide that. Additionally since there
are various public and commercial software that can read netcdf format this
might prove an added advantage.

 I agree that ascii is nice to view, but if you provide a utility to let you
view the binary file then the effect is the same.

						Dave Senciall
						Physical Oceanography Section
						NWAFC St.John's NF
					[email-masked]

---

From: Chris _topher_ Lakey <[email-masked]>
Date: Thu, 03 Oct 1996 17:51:34 -0500

Good luck!

-topher

---

From: [email-masked] (Edward Hartnett)
Date: 04 Oct 1996 15:04:58 -0400

I don't think you should worry about the overhead too much. A properly
written HDF or netCDF file ill not have significant amounts of
overhead. THat is, it will be almost as small as you could get it with
any custom definition.

Certainly I think you should stay far far away from your own
formats. It's just a dead end. If you're really smart and spend a lot
of time on it you'll end up with a format almost as useful as HDF,
except without all the free tools etc.

I prefer HDF myself, and it does have a good way to store station
data. I believe netCDF is more often used in the oceanagraphic field
and it does have one nifty feature. With a recompile of your code you
can get it to write HDF files instead wiothout changing any code. This
is because the HDF people stole the netCDF library and incorporated
it. 

Note that the two file formats are very different. But with a
recompile the exact same code that output netCDF will output HDF
files.

You will be tempted to do your own custom formats when you start
because it will seem like too much of a pain to use any standard
format. But beliieve me this is a mistake that you and future
programmers who work with you will regret. Pick a standard, don't make
up a new one. Heaven knows we have enough "universal" data formats
already!

-- 
Edward Hartnett                 [email-masked]
* I don't speak for NASA or ARC, I just work for them! *
     Goddard Distributed Active Archive Center (DAAC)
Check out cool Earth Science data at: http://daac.gsfc.nasa.gov/

---

From: Dave Senciall <[email-masked]>
Date: 10 Oct 1996 12:55:57 GMT

"Donald W. Denbo" <[email-masked]> wrote:

>I've been following this thread for while now and one aspect of data
>management that has not been addressed it how do you keep track of
>what processing has been done to the data?  We use netCDF to store our
>CTD data (over 50,000 stations) and the processing programs
>automatically 
>include comment lines documenting what has been done.  We had a
>in house developed format (ASCII headers, binary data) and have
>developed a library (epslib
>http://www.pmel.noaa.gov/epic/eps-overview.html ) to provide a file
>independent way of accessing the data.  The eps library allows us to
>continue to use the old data format with the new netCDF format!
>
>One big problem of ASCII data ... People can pull it into an editor and
>change values as well as header information. Those changes are very
>difficult to find. If you have a lot of data and are doing a lot of
>interactive processing large ASCII data files are a lot slower to read.
>If you store them compressed, not only do you have to uncompress them,
>but you have to convert the ASCII the ascii during a formated read.
>
>The final advantage that netCDF provides for us... we have a fairly
>large user group, and a self-describing data format with the appropriate
>well designed software, means a user doesn't have to know which of
>dozens of formats the data is written in.
>

  I agree, actually issues of logging of processing activities,quality control
etc. are factors that I must consider, Often if an internation study program,or
nation archive reqests you provide data to them they want to know what you have
done to it, the processing and manipualtion history is often as important as
the actual data .
 Another issue is that of 'indexing' or cataloguing the data for searching,
It's fine to stick 500,000 profiles into a archive but if you can't readily
search and rapidly extract data you can't accomplish much.

When I first posted my question I had no idea of that number of responces that
I would recieve ,nor the variety of opions aired.

I think the fundamental thing that distingushes the respondents is the 'scope'
of the projects that they work on. If your data set is small and/or personnel
then 'internation standard' quality data formats are likely more of an
inconvience than an asset. However if your group is large, your datasets large
and varied, if you have to exchange data with other institutes and you have to
manage data that is not part of 'your person project' but is more of an
institutional asset then formats like netCDF and HDF are seriously worth
considering. 

My currant archives are ascii based using unix compress for storage efficieny
for low usage data. Space and performance are not excessively pressing issues,
though in 5 years when the data sets have doubled in size they will be. My
concern centered more on access, as it is now a researcher looking at ,say,
water temperature in a region can access ctd data. If he wants to get
temperature data from a current meter sensor in the same region he has to
access a different data set that is in a totally unrelated format for meta data
and sensor data that is controlled by total different software, similarly if
they
wanted to get the thermister data from the ADCP head in a ship, the software
and
data formats are totally un-related. My goal is to, at the least, get a common
software interface to the data. ie point your program at desired dataset and
read out temperature data, with little effort having to be put into 'knowing'
the format of that data. Beyond that it would be nice that the 'style' of each
data set have a common 'theme' so that when a new instrument comes along you
have a 'formula' for designing the data storage rather that the add-hock method
that people tend to fall into where the archieve ends up begining a version of
the acquistion software format, or someones pet format that they dreamed up
over a beer.

Dave Senciall              Physical Oceanography Section
[email-masked] NorthWest Atlantic Fisheries Centre
ph. 709-772-4883           Dept. Fisheries & Oceans, Gov of Canada
fax 709-772-3207           Box 5667 St.John's,NF Canada A1C-5X1

---

From: [email-masked] (Marinna Martini)
Date: Thu, 10 Oct 1996 15:08:24 GMT

[email-masked] (Edward Hartnett) wrote:

>Yes, but I will point out that the number of cases where a data set
>starts as a personal thing and ends up being shared is not trivial. So
>I think you should amend the above to say that if the data are goiing
>to be shared, now, or ever, than a universal format is probably
>better.
>
This exact thing has happened to me, and it is only because I was
using netCDF that I was able to provide data with any sort of
reasonable turn around time to someone who wanted it.  I have also
found that when using a computer with limited power (laptop in the
field) I can still look at parts of large data sets stored in netCDF
format while ASCII would be very unweildy if not impossible for my
little underpowered machine to deal with.

>Actually if you just learn HDF or netCDF and start using it for all
>your data, personal or otherwise, the learning curve will quickly be
>paid back and you can stick to that one format for all data.
I can attest to that.

>Sinc the essence of science is colaberation and sharing, and since the
>net maakes the sharing of data easy and fun, I don't think it makes
>sense to plan on personal data. You should probably figure that almost
>all of your data will need to be shared at some point.
Never assume you are the only one... sooner or later you will be
collaborating!

Marinna

------------ "Be seeing you..."  - The Prisoner --------------
Marinna Martini            | U.S. Geological Survey
Ocean Engineer             | Marine & Coastal Geology Program
[email-masked]          | Woods Hole, MA 02543
(508) 548-8700 x326        | http://marine.usgs.gov/~mmartini
------------- Earth Science is a Public Trust ----------------

---

From: "Donald W. Denbo" <[email-masked]>
Date: Thu, 10 Oct 1996 08:46:54 -0700

Jeroen Schaap wrote:
> 
> Donald, if you want to tell me, how large is your group?
> 
> Jeroen

The EPIC format was originally started about 10 years ago by Nancy
Soreide and at that time stood for "Equatorial Pacific Information
Collection". The number of users probably was about a dozen. Since then
data from all over the world has been brought into the format (we
no longer pretend that the initials E.P.I.C. stand for anything!) and
the number of user sites is probably about 10.  Included several in
foreign countries and at several Oceanographic institutions.

Dave,  We too have the difficulty of finding the data we want to
process. EPIC has connections to both miniSQL and Ingress databases.
Several command line and X/Motif tools are available for selecting data
>from a large number of sites. When I first read your message I suggested
EPIC because there already exists a large number of programs in
existance that were designed to be robust.  We are always looking for
new groups with which we can collaborate!

-- 
         Donald W. Denbo   |    Pacific Marine Environmental Laboratory
     [email-masked]     |    7600 Sand Point Way NE
     Fax: (206) 526-6744   |    Seattle, Washington 98115
   Phone: (206) 526-4487   |

---

From: Russ Rew <[email-masked]>
Date: 11 Oct 1996 09:14:50 -0600

"Mark Hadfield" <[email-masked]> writes:

> NetCDF version 3 introduces a new file format (which will eventually
> allow a more general data model) and a completely reworked API. A
> pre-release of version 3 (C only) is available at the netCDF site
> (http://www.unidata.ucar.edu/packages/netcdf/index.html).

Well, you're right about the reworked API, but netCDF version 3 will use
the same file format used by all previous versions of netCDF, so files
written with the new version can be read with previous versions and vice
versa.

Rewriting the library offered an opportunity to implement a new
interface that provided significant benefits to programs that use
netCDF: type-safety, automatic type conversions, improved readability,
and simpler error behavior.  The new interface also removes some
obstacles to adding future enhancements, such as packed data and
enhanced concurrency.

The netCDF-3 library will include support for all netCDF-2 function
interfaces and behavior. The benefits of the new interface will be an
incentive to use it in future applications, but current applications
that use the netCDF-2 interface will continue to work. Programs may be
converted to the new interface incrementally, since a mixture of
netCDF-2 and netCDF-3 calls is permitted.

Version 4 will require a new format in order to support packed external
data (e.g. an array of 10-bit values), but the netCDF library will still
support reading and writing data in the current format.

More details about the alpha (prerelease) version of the C-language-only
part of netCDF-3 are available at

    <URL:http://www.unidata.ucar.edu/packages/netcdf/3.1a.html>

--Russ
_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
[email-masked]                     http://www.unidata.ucar.edu

---

From: Russ Rew <[email-masked]>
Date: 11 Oct 1996 09:14:50 -0600

--Russ

---

From: Ray Osborn <[email-masked]>
Date: Wed, 09 Oct 1996 09:24:00 -0500

Alistair Murray wrote:
> Why not use ASCII in a standard layout and then use a freely available compression
> program such as GNU gzip or zip?  This gives easy readability of data when
> uncompressed and someone else writes and maintains the compression software.
> It is also cross-platform portable at least from DOS to UNIX (I just checked!).
> Binary formats tend to run into portability problems with the dreaded byte-swapping
> fiasco (some operating systems / languages expect least significant byte first -
> others vice-versa!!)
> 
> We have large data sets of the kind you have (plus associated biological and
> acoustic data) and we find that gzip gives extremely high levels of compression.
> Often better than 66%.
> 

I have always had a problem with the idea that ASCII is somehow easier
to read.  Of course, any operating system allows you to print out ASCII
files, but viewing lists of numbers are not usually the best way of
visualizing the data, and reading such data in programs can be fraught
with difficulty if we do not have access to the precise format used to
write the file.  Are the columns tab-delimited, (multiple-)white space
delimited, free format etc?  We also normally want the data to be
plotted with suitable axes and labelling, titles and other annotations
so we need unambiguous ways of associating column headers or header
block information with the remaining data.  These are not serious
problems if the format is properly documented, but become a major
headache with more complex data sets if it is not.

It seems to me that portable self-describing binary formats such as HDF
or netCDF should make it easier to view the data since there is no
ambiguity about what is being stored.  HDF are in the process of writing
Java code to access HDF files, including some visualization tools, which
I hope means that it may soon be possible to browse through these HDF
files with our favourite web browsers.  Then binary data would become as
easy to view as ASCII for most of us. 

------------------------------------------------------
Dr Ray Osborn                  Tel: +1 (630) 252-9011
Materials Science Division     Fax: +1 (630) 252-7777
Argonne National Laboratory    E-mail: [email-masked]
Argonne, IL 60439-4845
[Please note change in telephone area code]

---

From: "Donald W. Denbo" <[email-masked]>
Date: Wed, 09 Oct 1996 09:08:13 -0700

Jeroen Schaap wrote:
> 
> 
> Interchanging data between various programs and platforms during the
> last year of a research project, I've got some experience with ASCII.
> I've tried some binary formats, but they were all laborious to program
> and hard to interchange. Although my dataset is considerably large
> (Estimation of 1 Gb uncompressed) with compression of data I am not
> accessing frequently I don't use too much space. Moreover, I can
> transport and read my formats (about 10 ASCII formats) really easily
> between programs, universities and platforms. To me it's very
> convenient, but most of all, time-effective. I don't want to spend weeks
> on programming, since I have to do all the programming myself. Programs,
> macro's and routines must be completed within a few hours.
> 
> Success
> 
> Jeroen

I agree that ASCII is faster to work with than netCDF or our eps
library, but your only talking about the initial development time, time
required to add new features, time spent debugging, robustness, all of
these are improved using a self-describing data format (and for us) a
library that allows multiple file formats to be read with the same
library.

-- 
         Donald W. Denbo   |    NOAA/PMEL/OCRD
     [email-masked]     |    7600 Sand Point Way NE
     Fax: (206) 526-6744   |    Seattle, Washington 98115
   Phone: (206) 526-4487   |

---

From: "Mark Hadfield" <[email-masked]>
Date: 9 Oct 1996 22:37:01 GMT

andreas schumm <[email-masked]> wrote in article
<[email-masked]>...
> 
> I prefer the netCDF data model to the HDF SDS model, but i like the
> HDF API. With HDF 3.3 and higher, however, you can have both, if you
> don't mind losing the physical netCDF file format.
> 
> Andreas Schumm

NetCDF version 3 introduces a new file format (which will eventually allow
a 
more general data model) and a completely reworked API. A pre-release of
version 3 (C only) is available at the netCDF site
(http://www.unidata.ucar.edu/packages/netcdf/index.html).

-- 
==============================================================
 Mark Hadfield                        NIWA (Taihoro Nukurangi)
                                      PO Box 14-901
 [email-masked]               Wellington, New Zealand

---

From: [email-masked] (Edward Hartnett)
Date: 10 Oct 1996 11:34:28 -0400

>>>>> "D" == Dave Senciall <[email-masked]> writes:

    D>   I agree, actually issues of logging of processing activities,quality control
    D> etc. are factors that I must consider, Often if an internation study program,or
    D> nation archive reqests you provide data to them they want to know what you have
    D> done to it, the processing and manipualtion history is often as important as
     D> the actual data .
    D>  Another issue is that of 'indexing' or cataloguing the data for searching,
    D> It's fine to stick 500,000 profiles into a archive but if you can't readily
    D> search and rapidly extract data you can't accomplish much.

    D> When I first posted my question I had no idea of that number of responces that
    D> I would recieve ,nor the variety of opions aired.

Hahahahaha!

Welcome to the Format Wars!

As a Format Warrior for the last ten years I know all the arguments
and aat one time or another have supported all different sides. Years
ago ASCII was my format of choice for small data files, and I thought
HDF was a giant pain in the butt. Now I've swung 180 degrees on the
issue. 

    D> I think the fundamental thing that distingushes the respondents is the 'scope'
    D> of the projects that they work on. If your data set is small and/or personnel
    D> then 'internation standard' quality data formats are likely more of an
    D> inconvience than an asset. However if your group is large, your datasets large
    D> and varied, if you have to exchange data with other institutes and you have to
    D> manage data that is not part of 'your person project' but is more of an
    D> institutional asset then formats like netCDF and HDF are seriously worth
    D> considering. 

The amount of effort scientists spend on data formats is tremendous! I
think we would all like to see this problem go away so that you-all
could spend that time doing science. HDF and netCDF are not the
perfect answer, but are as close as we've come so far.

For example the group I work with aat NASA has some data that, 5 years
ago, they would have sworn would never be of interest to anyone. Of
course I wouldn't be raising this example if the data were not of
tremendous interest to several other groups at the moment. For the
next iteration of their processing this group is doing almost
everything in HDF. 

    D> My currant archives are ascii based using unix compress for storage efficieny
    D> for low usage data. Space and performance are not excessively pressing issues,
    D> though in 5 years when the data sets have doubled in size they will be. My

Probably not really. 5 years ago the largest disk size you could get
was what, 1.2 GB? Now 9 GB disks are common. This trend will only
continue. Data doubling every 5 yeas is not worth worrying about. In 5
years you'll just buy a disk that's ten times the size of your current
one and you'll be good to go.

    D> concern centered more on access, as it is now a researcher looking at ,say,
    D> water temperature in a region can access ctd data. If he wants to get
    D> temperature data from a current meter sensor in the same region he has to
    D> access a different data set that is in a totally unrelated format for meta data
    D> and sensor data that is controlled by total different software, similarly if
    D> they
    D> wanted to get the thermister data from the ADCP head in a ship, the software
    D> and
    D> data formats are totally un-related. My goal is to, at the least, get a common
    D> software interface to the data. ie point your program at desired dataset and
    D> read out temperature data, with little effort having to be put into 'knowing'
    D> the format of that data. Beyond that it would be nice that the 'style' of each

HDF and netCDF come with a program called ncdump (or something of that
kind) that can give you an ascii dump of any variable (or all of them)
>from the data file. This gives you the best of both
worlds. Visualisation packages like grads can now read HDF files and
netCDF files. This trend will continue. HDF and netCDF are here to
stay and more and more tools will deal with them well.

    D> data set have a common 'theme' so that when a new instrument comes along you
    D> have a 'formula' for designing the data storage rather that the add-hock method
    D> that people tend to fall into where the archieve ends up begining a version of
    D> the acquistion software format, or someones pet format that they dreamed up
    D> over a beer.

This of course is the big savings when dealing with HDF or
netCDF. Smart format people have already thought of all the different
types of data and provided for them in the formats, so you don't have
to invent a way to store your data, just read the manuals and find the
way already provided.

I urge you to let those smart format people work for you!

Good luck, and let us know what your decision is.