Subject: Re: Difficulties with netCDF and Sparse Data

From: [email-masked] (Carlie Coats)
Date: 3 Jun 1997 09:21:04 -0400

In article <[email-masked]>,
Jeremy Beal <[email-masked]> wrote:
>
>Forgive me for bringing up a difficulty which has been encountered
>before, but I'm interested in seeing if anybody has made any headway.
>
>We have an interest in storing a variety of numerical data as output
>from a software simulation of a physical system. The data which must
>be stored are physical quantities which depend on spatial coordinates
>and time. The output data must be read by another program, possibly
>running on a different platform, so we would like the data file to
>be platform independent. There is a large quantity of output data so
>we practically need a binary file. We are interested in netCDF as
>a means to achieve an easily read platform independent binary data
>file. We would also enjoy being able to use the tools which have
>been developed for examining netCDF files.
>
>Unfortunately, our data is not necessarily regularly patterned, and
>it seems that it may not fit well within a standard netCDF file. Our
>natural inclination would be to use the time value as the unlimited
>dimension within the file and then define the coordinates of our
>spatial grid points using three additional dimensions.
>
>Problems:
>	Our data is sparse in both a spatial and time sense;
>	i.e. not every physical quantity is written out at each
>	time step, nor at every spatial grid point within a
>	given timestep.
>
>	Our grids themselves can vary as a function of the
>	timestep, i.e. a finer grid might be created inside
>	of a cell for a single time step for needed accuracy.
>	The finer grid would only exist for one or two timesteps
>	and would then no longer be used for the rest of the
>	simulation....

We have a somewhat similar problem in communicating data for
Kain-Fritsch-McHenry cloud events between the MM5 meteorology model
and the MAQSIP air quality model.  In that case, convective cloud events
arise at random times in random horizontal cells of the met model;
the characteristics of the vertical profile which sets off the
event, as well as the starting date, starting time, and duration
of the event must be communicated from MM5 to MAQSIP.  We do this
using structure built on top of our "usual" netCDF wrapper.  The
files have three components:

    a file header that matches our usual file header structure
    (dimensions, attributes, etc.);

    a (fixed-maximum-size) events index which permits selective
    direct access into:

    an events list dimensioned in the UNLIMITED dimension, which
    stores vertical profiles of all variables associated with an
    individual cloud event, in addition to other characteristics
    of the event.

All this is documented at URL

    http://www.iceis.mcnc.org/EDSS/ioapi/H.DATATYPES.html#kfevnt

or feel free to call me.

Carlie J. Coats, Jr.              [email-masked]    *or*    [email-masked]
MCNC Environmental Programs                             phone: (919)248-9241
North Carolina Supercomputing Center                      fax: (919)248-9245
3021 Cornwallis Road                                         P. O. Box 12889
Research Triangle Park, N. C.  27709-2889                                USA
"My opinions are my own, and I've got *lots* of them!"