Subject: netCDF problems and a proposed solution

From: [email-masked] (Paul J Turner)
Date: 22 Jul 93 21:56:50 GMT

For several months, I've been trying to sell management on the use
of netCDF as a transportable file format for our applications (I've 
looked at HDF also). While my initial enthusiasm was strong, I'm now 
having problems selling the notion to myself. The problems I'm 
having are:

1. Can't read/write from/to stdin/stdout. A real problem, as most of our
   visualization codes rely on opening the model generating the data as
   a pipe. The amount of data is often too large and too time consuming
   to generate to go through the hassle of writing the file completely
   before viewing it. We want our data in semi-realtime. This is a 
   killer, it forces me to write io for both native binary and netCDF.

2. Little support for other than multidimensional arrays. We use finite 
   elements, 1d, 2d, 3d with a variety of element types. We also have
   polygonal data, scalar fields, flows, pathlines, etc.. While these data 
   may be described using netCDF, there are no standards (that I know of). 

3. Ncgen is a nice tool, and was one of my selling points to management,
   but it doesn't go far enough. To be really useful, it should not only
   be able to generate source to define the netCDF file, but also to read
   and write the file, in a generic fashion suitable to plug into any
   one's code. Some of the data structures we work with are really big
   and contain lots of other data structures - writing io for these things
   would be a major task, and I see no reason why ncgen shouldn't be able
   to generate the code to do it.

4. CDL does not allow derived types to be easily defined. What I'd like to
   have is something like a C typedef (or a C++ class), that allows
   previously defined CDL descriptions to be reused.

5. netCDF files seem bloated and slow to read/write. A test I performed 
   yesterday on a small file:

        The ASCII, uncompressed version   ....  18,397,806 bytes
        Compressed ASCII                  ....   4,421,421 bytes
        Uncompressed binary               ....  11,933,712 bytes
        Compressed binary                 ....   4,311,315 bytes
        netCDF                            ....  20+ MB (got tired of
                                                waiting so I killed it)

   The binary file above can be read in 26 secs (one 20+ byte record at
   a time), I didn't have the time to wait for the netCDF test to
   conclude.

The basic problem is that netCDF is a flat file data format and we work
with objects. I'm considering the following solution and I solicit your
comments - if this has already been done, please let me know:

 * Fundamental types char, int, float, double, etc. maybe typedef'ed
   to things like Int16, Int32, Float32, Float64, etc.

 * In addition to code generation to read/write binary, generate code 
   to read/write ASCII if necessary (in case none of this flies).

 * Add a flag for byte order. In general, allow some method of attaching
   meta-data (which would also be an object). All data to be written in the
   machine's native byte order, IEEE floating point. 

 * If random access is needed, generate an object index.

 * For each object definition, allow for a version number. If the 
   object definition changes, increment the version number so support
   codes are aware of the difference.

 * Use a C language typedef as the object description (toss it in 
   with the data). Write a parser that interprets the typedef and 
   generates the read/write routines (did this, took about an hour).
   As with a typedef, the parser understands derived types, so with
  
  typedef struct _Element Element;

  typedef struct _Grid {
	int nelements;      /* number of elements */
	int nnodes;         /* number of nodes */
	FloatArray x, y, z; /* table of nodes */
	Element *elements;  /* table of elements */
	etc...
  } Grid;

I can derive a new class, GridT (adaptive FE grids with a structure that 
changes in time):

  typedef struct _GridT {
	int nsteps;      /* the number of grids */
	FloatArray time; /* at these times */
	Grid *grids;     /* the grid at each time step */
	etc...
  } GridT;

Now I throw this at the parser and it generates C (or Fortran) to 
read and write these objects. The data description is simply 
#include'ed in my code, where needed.  Previously generated code 
to write Grids can be reused to write GridTs.

Another example:

A 3d hybrid grid - a 2d grid on the surface with
1d grids dangling from the nodes. Here is a definition:

    typedef struct _Grid3D {
	Grid grid2d;  /* the 2d grid */
	Grid *grid1d; /* a dynamic array of 1d grids, each attached to
                      /* a node in the 2d grid - already described by
		      /* the Grid structure, the number of nodes is
                      /* specified in the 2d grid. */
    } Grid3D;

etc. A further enhancement would be to toss in a metafile-like
description of how to display the object (or perform other 
operations) if not already known.
 
 * An object registry, if someone has a new object, either derived or 
   new, it is submitted for approval, then whatever support is needed 
   added to the necessary codes. Each object is assigned a magic number,
   with a set reserved for officially approved objects, and a set for user 
   defined objects.

 * Use of standard Fortran/C io (Fortran open/read/write/close and 
   C fopen/fread/fwrite/fclose).

 * What am I leaving out?

I just received a trial version of IBM's Data Explorer, and while having
better support for objects than netCDF, it is limited in the data types
it can handle (e.g. no 6 node subparametric triangles). There seems to
be no code generation capability (i.e. the ability to generate io 
routines using my names) for the data types it does support. No 
support for other than 4 byte reals. An idiosyncratic data description 
language. I've no doubt it does great graphics.

Is there anything out there that does what I've outlined above? I need
bindings for C and Fortran (C++ would be neat also). Automatic code
generation is essential.

--Paul

Paul J Turner
Center for Coastal and Land-Margin Research
Oregon Graduate Institute
20000 NW Walker Road
Portland, OR 97291-1000

---

From: [email-masked] (Ethan Alpert)
Date: Fri, 23 Jul 1993 21:31:30 GMT

>   killer, it forces me to write io for both native binary and netCDF.

This is not a netCDF problem it is a UNIX stream IO problem. If you can
figure out how to support random access to UNIX streams you'd be able to 
read and write from stdin/stdout with the netCDF API.

>   may be described using netCDF, there are no standards (that I know of). 

The netCDF mailing list touched on this several months ago with respect
to standardizing data representations but ran into problems agreeing on
what needs to be standardized. 

The application programmer types, like myself, were leaning towards the
type of solutions you mention. Basicly this structural information is the type 
of information a netcdf file needs to comunicate to a general visualization 
tool or application for the application to be able to manipulate the data.
Take a look at the AVS field file format. It communcates the structure and
organization of the data to AVS so AVS can do something meaningful with it.

However, this is not the kind of information needed to communicate the contents
of the file to a scientist. In fact several scientist suggested that these
types of conventions would make writing data so esoteric that it would not
be convenient to write data in netCDF.  The scientist in most cases just wanted
to standardize the attribute and variable names within each discipline. 
The advantage of this with respect to interpreting data from a human 
perspective is obvious, however it does nothing to support interpreting data 
>from an analysis and visualization tool. 

In the end the thread fizzled and nothing seems to have become of the 
suggestions. 

Basicly this topic could be a thread all to itself. What should the
term "self-describing format" mean?  To whom should a self-describing file 
format describe itself and what types of conventions are needed.

>   one's code. 

Could you elaborate on exactly what you mean here? You can generate source
to read the entire file and write the entire file, but if this is all you
want out of the netCDF interface your missing the point. I think the main
selling point beyond portability is the random access that the netCDF interface
provides to the individual variables and metadata. You don't have to read
the entire file into memory to get access to a small portion of it.

>   Some of the data structures we work with are really big

Once again could you elaborate. Automatically generating the source to read
files that contain "lots of other data structures" doesn't on the surface
sound trivial. There is also the issue of what you are reading them into.
If you are reading into C data structures how are these defined and comunicated
to the ncgen type tool you allude to.

>   previously defined CDL descriptions to be reused.

This is intimately connected with number 2.

>        netCDF                            ....  20+ MB (got tired of

What are you doing here writing a netCDF file full of doubles?

If your ascii representation is seven digits or less then you only
need floats. I've done the same types of tests you mention and have never
found the netCDF file to be more than 5% larger that the **equivalent 
binary representation**.  

If your binary represenation uses floats and you write it to a double 
variable you should expect a 20+ Mb file regardless of what format you choose.

I'm sorry I don't have any solutions for you but I would like to see some
of the issues in the thread discussed more in sci.data.formats.

-ethan

--
Ethan Alpert  internet: [email-masked] | Standard Disclaimer:
Scientific Visualization Group,             | I represent myself only.
Scientific Computing Division               |-------------------------------
National Center for Atmospheric Research, PO BOX 3000, Boulder Co, 80307-3000

---

From: [email-masked] (Russ Rew)
Date: Fri, 23 Jul 1993 22:42:12 GMT

>    killer, it forces me to write io for both native binary and netCDF.

There are tradeoffs between sequential I/O and direct-access I/O.  If you
want to be able to access small amounts of data efficiently from large files
without reading through all the preceding data first, or to access data in a
different order from the order in which it was written, you must use
direct-access I/O implemented with something like lseek(2).  You can't seek
on a pipe, so this choice precludes reading/writing from/to stdin/stdout.

On the other hand, if you usually want to read through all of the data in a
dataset in the order in which it was written and only rarely extract or
update small subsets of the data in larger collections, you may not need
direct-access I/O.  In that case, you may use sequential I/O and get the
advantages of pipes.  This also permits you to access data directly from
sequential devices like tapes without staging to or from disk first, which
may be an advantage for some kinds of data processing.

NetCDF uses the direct-access I/O model because it was designed for
visualization applications and data analysis applications that typically
require access to data in an unanticipated sequence or to small subsets of
large datasets.

>    may be described using netCDF, there are no standards (that I know of). 

Some conventions are evolving for netCDF representations, but the
conventions differ from discipline to discipline, and it would be premature
to standardize on the conventions developed in any one area.  It takes time
to discover the best ways to represent data for particular patterns of
access, and the patterns of access often change in unanticipated ways as a
result of making the data available.

>    to generate the code to do it.

These are good ideas for enhancements to ncgen, some of which have been on
our list for some time.  A lack of resources for netCDF development has led
to this list growing rather than shrinking over time.

>    previously defined CDL descriptions to be reused.

There is a proposal for an inheritance hierarchy of netCDF file classes that
would use a single global attribute "Class" to do what you want.  The idea
is that a global attribute, say "_Class", would be used to specify a class
hierarchy for netCDF files (and hance the CDL descriptions).  For example,

   :_Class = "Foo.Bar.Baz";

could be used to say that this netCDF file follows all the conventions of
Foo files (so Foo-specific applications may be used on it), and also follows
the conventions of the Bar subclass of Foo files, and the Baz subclass of
Foo.Bar files.  This would be enough to specify a colection of required
dimensions, variables, and attributes in the file, but additional components
could also be added by specifying them.  A place to register or look up
class hierarchy conventions would be required, and different disciplines or
organizations would need to maintain their own data class specifications.

Something like this is already happening in the oceanographic community.

>    a time), I didn't have the time to wait for the netCDF test to

Performance varies greatly depending on platform, data types, and how you
structure and access the data.  In some applications the storage overhead
over binary files is insignificant, and the access overhead is small.  The
benefits of machine-independent data will always require some overhead.
Machine-independent and application-specific data formats will always
outperform platform-independent and general-purpose data access interfaces.

> comments - if this has already been done, please let me know:

 ... [long description of requirements omitted for brevity]

> bindings for C and Fortran (C++ would be neat also). Automatic code

You should look into PDBlib (Portable Database Library) from Stewart Brown
of LLNL.  It sounds more closely matched to your requirements than netCDF.
It can be anonomous-FTP'd from phoenix.ocf.llnl.gov, and is part of the
"PACT" distribution (Stewart Brown's environment for writing simulation
codes), which is contained in the file pub/pact7_23_93.tar.Z.
--
________________________________________________________________________
Russ Rew					Unidata Program Center
[email-masked]				UCAR, PO Box 3000
						Boulder, CO 80307-3000