Subject: Re: two questions concerning NetCDF

From: [email-masked] (Russ Rew)
Date: 05 Jan 1995 17:38:51 GMT

In article <[email-masked]> [email-masked] (Gerhard Eckel) writes:

> We are in the process of evaluating netCDF. Our institute is specialized
> in musical applications and our main usage of netCDF would be storage and
> retreival of sound signal analysis data and sound synthesis control
> data. We are currently using home-brew formats. Compared to our own
> formats we found two problems with netCDF:
> 
> 1) We used to pass data over UNIX pipes from one program to another to
>    avoid writing intermediate data files. As far as we can see up to now
>    netCDF doesn't support this type of data exchange (stream oriented
>    rather than file oriented).  What are the reasons for this apparent
>    limitation? Will there be extensions of netCDF which will allow for
>    this feature? Could we add this extension ourselves and make it
>    available to UNIDATA such that it will be supported in the future?

The inability to use a "pipes and filters" approach directly with netCDF was
intentional to get the corresponding benefits of an engineering trade-off
between direct access and sequential access.  Direct access permits
efficient access to a small subset of a large dataset without reading
through all the preceding data.  Sequential access permits simple connection
via pipes, but is inefficient for accessing small amounts of data from a
large file.

A feature of the first release of the netCDF operators we developed is that
they can read an input netCDF data set from standard input and can also
write an output netCDF data set to standard output.  Thus, these operators
can be used in UNIX pipelines (although this is only recommended for small
data sets).  This was implemented on input by copying standard input to a
temporary file and opening it as a netCDF file, and similarly on output by
copying a temporary output file to standard output.  A similar approach can
be used with any program that deals with netCDF data, providing the
advantages of direct access when file names are provided for input or
output, but using standard input or output via transparent copying
otherwise.

> 2) We use variable record sizes for many of our analyses data. A maximum
>    record size is hard to estimate before the analysis is started and the
>    record size cannot be changed efficiently during the analysis
>    process. A maximum record size can also waste much disk space. Do other
>    people have this problem with netCDF? Are there work-arounds?

NetCDF only permits one unlimited dimension, so if this is used as the
"record" dimension, you can't have variable record sizes.  The reason for
this limitation is again an engineering trade-off, to provide efficient
access to cross-sections of data.  We know of no implementation that permits
multiple unlimited dimensions, efficient access to orthogonal cross-sections
of data, and the ability to later append data along any of the unlimited
dimensions.

We have investigated ways to remove the restriction on a single unlimited
dimension, but these appear to require adding other new restrictions to
netCDF data access.  For example, by restricting the writing of
multidimensional arrays to occur in the same order as they are stored,
rather than in arbitrary order as is now allowed, multiple unlimited
dimensions might be (weakly) supported.  Another approach requires garbage
collection, to recover unused locations so that a file doesn't grow
quadratically in size as a single vector is extended linearly along an
unlimited dimension.

A workaround when you need multiple unlimited dimensions in a single file is
to use multiple files with separate unlimited dimensions in each, but this
only works if no single variable uses more than one of the desired unlimited
dimensions.

> Gerhard Eckel
> IRCAM, Centre Georges Pompidou
> 1, place Igor Stravinksy
> F-75004 Paris / France
> [email-masked]