Subject: Position Statement from Treinish

From: BRUCE BARGMEYER 202-260-5306 <[email-masked]>
Date: Thu, 17 Mar 1994 12:39:00 -0500 (EST)

The following was distributed to participants of an EPA/MCNC workshop 
on metadata. I think it may be of interest to recipients of this 
metadata reflector.

--Bruce Bargmeyer

Subject: Position Statement from Treinish
To: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "reagan.james" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, "namzer.john" <
Date: Wed, 16 Mar 1994 11:37:00 EST

Workshop Participants:

	I thank you for attending our data management workshop.
As promised, I am sending you a position statement I received from Lloyd
Treinish.  Other papers available in hardcopy include:

"Unifying Principles of Data Management for Scientific Visualization"
"Interactive Archives of Scientific Data"
"An Architecture for Rule-Based Visualization"
"Visualization of Stratospheric Ozone Depletion and Polar Vortex"
"Human Vision, Visual Processing, and Digital Display IV"
	
Ted

-------------------- Message from Treinish -----------------
>From [email-masked] Tue Mar  8 10:47:56 1994
Date: Tue, 8 Mar 94 10:51:11 EST
From: "Lloyd A. Treinish" <[email-masked]>
To: [email-masked]
Status: OR

Position Statement on Scientific Data Models, Structures and Access Software

Lloyd A. Treinish
Visualization Systems
IBM Thomas J. Watson Research Center
Post Office Box 704
Yorktown Heights, NY 10598 USA
914-784-5038 (voice)
914-784-5130 (facsimile)
[email-masked]

Background

There are a tremendous number of sources of scientific data, be they computed
or measured.  Even from a single source there can be a wide variety of data
sets.  Each such data set typically contains several independent variables
such as time, one or more spatial, spectral, etc. variables, and of course,
many dependent variables, where the interesting science is stored.  There can
be a bewildering range of underlying formats, structures, arrangements and
access methods for these data.  Of course, the size and complexity of these
data are growing significantly as data generators rapidly improve.
Unfortunately, an appropriate data handling infrastructure is still required,
for which there is no universally agreed upon solution.

Independently, there is a need to develop a taxonomy of such data by both
structural characteristics and application areas.  Rudimentary efforts along
these lines were begun at a SIGGRAPH '90 workshop and additional efforts have
been continued since but much work is still required, especially in the
collection of discipline-specific information.

Data Structures

To attempt to bring some simplifying order to this chaos, consider six key
attributes of data: dimensionality, parameters, data type, rank, mesh
structure, and aggregation.

Any data set may be considered as a function(s) of independent variable(s).
These independent variables may be called dimensions. The number of
independent variables may be called the dimensionality of the data.  It is the
fundamental characteristic of the data.  Such dimensions may be space (length,
width, height), time, energy, etc.  For example, zero-dimensional data are
just numbers such as sales, while two-dimensional data could depend on an area
such as barometric pressure over a state.  Some complex data may have five or
more dimensions.

The function(s) composing a data set really are dependent variable(s) -- the
data themselves, which may be called parameters.  They are dependent on the
dimensions, such as sales or temperature.  Thus, data implies a parameter or
field of one or more (dependent) values that is a function of one or more
(independent) variables, e.g.,

y1, y2, ... , ymY  =  
f1(x1, x2, ... , xn)                               (1)
                        f2(x1, x2, ... , xn)
                        .
                        .
                        .
                        fm(x1, x2, ... , xn)Y

The data type includes the physical primitive, which describes how data values
are stored on some medium (e.g., byte, int, float, etc.).  It can include
machine representations (e.g., little endian vs. big endian, IEEE vs. VAX,
etc.).  In addition, there is a category of such types, i.e., real, complex or
quaternion.

A parameter may have more than one value, which is characterized by tensor
rank.  Rank 0 is a scalar (one value), such as temperature (a magnitude -- a
single-valued function).  Rank 1 is a vector such as wind velocity (a
magnitude and a direction: two values in two dimensions, three values in three
dimensions).  Vectors of size, n, are n-valued functions.  Rank 2 is a tensor
such as stress on an airframe (four values in two dimensions, nine values in
three dimensions).  A rank 2 tensor in n-dimensional space is a n x n matrix
of functions (e.g., stress).  Dimensionality and rank are thus, related.  The
number of elements in a particular parameter is d**r, where d is the
dimensionality and r is the rank.  As with dimensionality, rank may be large
for very complex data.

There is often an association between the dimensionality of the data and its
geometry, which can be called a mesh or grid -- the size, shape and
organization of how the data relates to a physical domain.  This is perhaps
the class of characteristics for which there is the most confusion because of
the typical use of domain-specific terminology.  In general, a mesh describes
the base geometry for the mapping of the functions or dependent variables to
some (physical) coordinate system.  For example, there can be:

o    Regular grid with regular positions and regular connectivity such as a
     temperature map

o    Deformed regular or curvilinear or structured grid with irregular
     positions and regular connectivity such as the pressure on an airframe

o    Irregular "regular" or structured grid with irregular positions and
     regular connectivity such as several satellite images with gaps in
     coverage

o    Unstructured or irregular grid with regular or irregular connectivity
     such as a finite element mesh in a structural analysis problem

o    No grid with irregular positions and no connectivity -- scattered points
     such as sales figures or rainfall in specific towns

Such a mesh may be explicitly or implicitly positioned.  In the latter case,
it is preferred to store the positioning information implicitly.  Some
implementations may store it explicitly.  In most cases there is a topological
relationship or cell primitive connecting these positions.  For example,

Dimensionality      Cell Primitive
--------------      --------------
0                   not applicable

1                   line

2                   triangle, quadrilateral

3                   tetrahedra, parallelpiped, hexahedra, prism, pyramid

Often there is a need to form aggregates of data.  In dependent-variable
space, one could have a collection of parameters or fields over the same or
different grids that could be treated as single entity.  In that sense, an
aggregate or group could be composed of members that are either a single field
or another group.  This mechanism can be used to define simple tree
structures.  In independent-variable space, one could have a collection of
meshes.  For example, in aerospace fluid dynamics simulations, a computation
is often performed over several intersecting grids.  Such a multigrid solution
permits the definition of variable grid resolution and regularity around
airframe structures such as an engine nacelle or a wing.  In this case, one
could treat the collection of meshes as a single entity, although a mechanism
must be defined for accommodating regions of invalidity where grids may
intersect.  A similar problem occurs with observational data, where there may
be no data for some grid nodes.  Another type of mesh aggregate can result
from a hybrid collection of meshes of different cell primitives, where within
each mesh, the cell is the same.  A special case of aggregates can be called
series, where the tree structure has only one level of children.  The classic
example is a time series, where there is multiple instances of some field or
aggregate over a constant or changing mesh.  Generically, such a series does
not have to depend on time, but could apply to any sequencing of events.

Implementations

Traditional methods of handling scientific data such as flat sequential files
are generally inefficient in storage, access or ease-of-use for large complex
data sets particularly for input/output and floating-point-intensive
applications like signal processing (e.g., inverse problems) and
visualization. Modern, commercial relational data management systems do not
offer an effective solution because they are more oriented to business
applications. The relational model does not accommodate multidimensional,
irregular or hierarchical structures often found in scientific data sets.  In
addition, relational systems do not provide sufficient performance for the
size, complexity and type of access dictated by current and future data sets
and their potential usage.  Hence, current data base systems are not yet up to
the challenge of supporting very large data sets.

Therefore, there is a need for some type of data (base) model(s) that
possesses elements of a modern data base management system but is oriented
toward scientific data sets and applications.  This intermediate approach
should be easy to use, support large disk-based (perhaps other media as well)
data sets and accommodate multiple scientific data structures in a uniform
fashion.  In the process of providing simple access to self-describing data,
such a mechanism should match applications requirements for visualization as
well as data analysis and management, and be independent of any specific
discipline or source or visualization technique.  Hence, data management as
embodied as a data model(s) is as important to scientific applications as the
underlying computation.  Its implementation, the management of and access to
the data, should be decoupled from the actual computational software.

These classic limitations have been recognized by a few groups in the support
of a number of scientific applications.  As a result, several data models have
been defined, some within very domain-specific contexts while others being
more general.  Recently, a few of these models have been associated with
software, including well-defined language bindings that represent the
implementation of one or more abstract data types.

Common Data Format (CDF), developed at NASA/Goddard Space Flight Center, was
one of the first implementations of a scientific data model.  It is based upon
the concept of providing abstract support for a class of scientific data that
can be described by a multidimensional block structure.  Although all data do
not fit within this framework, a large variety of scientific data do.  From
the CDF effort spawned the Unidata Program Center's netCDF.  These systems are
extensible by the user, and conventions have been established in some orga-
nizations to ensure proper interpretation when data are exchanged.  Although
the data models for both CDF and netCDF are essentially the same, the
interfaces and physical storage are quite different.  Among many other
differences, CDF transparently supports multiple physical formats on all
platforms, which can be chosen for convenience or improving performance.  In
contrast, netCDF supports a single physical representation.

Another important effort has been the Hierarchical Data Format (HDF) developed
by the National Center for Supercomputing Applications (NCSA) at the
University of Illinois at Urbana-Champaign.  This activity evolved from the
need to move files of scientific data among heterogeneous machines, which grew
out of the requirement to look at images and other data on personal computers,
workstations, etc. that were generated on a supercomputer.   HDF, which is
also self-describing, uses an extensible tagged file organization to provide
access to basic data types like a raster image (and an associated palette), a
multidimensional block, etc.  In this sense, HDF provides access (i.e., via
its C and FORTRAN bindings) to a number of different flat file organizations.
Unlike CDF and netCDF, registration authority for the definition of data
structures, formats and interfaces is with the implementor.

The Visualization Systems Group at IBM Thomas J. Watson Research Center has
developed a more comprehensive data model that includes curvilinear and
irregular meshes and hierarchies (e.g., trees, series, composites), vector and
tensor data, etc. in addition to the class of scalar, multidimensional blocks
supported by the aforementioned implementations.  The IBM Visualization Data
Explorer software, developed by this same group is built upon this data model.
It is a client-server data-flow system for general visualization applications.
This software provides polymorphic operations on data, independent of a role
in generating pictures, by working with shared data structures in memory via
an uniform interface.  There is a physical disk-based format, as one
import/export mechanism but only simple sequential access is provided.

Issues

A major concern for extant systems is in their practical application to a
number of large data sets.  None of the implementations in their current state
can address the diversity and size of extant and future data sets in an
application-independent fashion.  These issues often relate to scaling to even
modest data sets by today's standards independent of addressing raw bandwidth.
The data effectiveness of a system can be measured by its ability to handle
multiple data sets simultaneously of various sizes, types, structures, etc.
without forcing artificial constraints that disrupt the fidelity of the
original data.  Systems that support different classes of data separately will
have difficulty scaling to support disparate data properly at the same time.
Systems that support different classes of data uniformly do not because they
effectively decouple the management of and access to the data from the actual
applications software.  Some of the specific issues, for which improvements
and standards are potential required, include but are not limited to:

o    Data structure residency

Are the data structures in an implementation on secondary or persistent
storage (i.e., disk) and/or in primary storage (i.e., memory)?  Efficient
disk-based data structures, access methods and interfaces are critical for
utilization of large data sets, where the aggregate size may be one or
more orders of magnitude greater than available memory.

o    Transaction-like processing

What types of access methods for data are supported?  The ability to do DBMS-
like operations on data in place on secondary storage is important for data
sets that are too expensive to even partially reproduce.  The design of
efficient caching schemes for large data blocks is required.  Support of
different physical organizations transparently may be necessary as well.

o    Physical format, file structure, protocols, etc.

How does an implementation compensate for typical limitations in conventional
file systems (e.g., blocking), operating systems (e.g., paging) and networks
(e.g., NFS, RPC, TCP/IP) for bulk data access?  An underlying mechanism geared
for bulk access is required.  Ideally, this should be available independently
of a data model implementation.  In practice, it is not.  This can be viewed
as an interface between storage systems and data management systems, since
these scientific data models are in effect a class of data management systems.
This notion is also important for effective use of high-performance and
expensive storage, communications and computation technology (e.g., parallel
disk arrays, HiPPI networks, supercomputers).  For example, the IBM Power
Visualization System (a medium-grain, shared-memory, parallel supercomputer)
utilizes a custom RPC between computational and input/output (HiPPI)
processors to achieve sustained high-speed throughput (e.g., 95 MB/sec to a
HiPPI frame buffer, 95 MB/sec to a RAID-3 HiPPI disk array or 76 MB/sec
to a 4-bank SCSI-2 fast and wide RAID-3 disk array with a simple block

64KY + extent file system)

o    Applications enabling

How should programmers view data when they develop applications?  Similarly,
how should applications view data for users?  Data model(s) and
implementations should provide uniform "object" access independent of
underlying structure, which may include representation and access to complex
meshes and hierarchies, multiresolution abstraction, compression as well as as
simple rectilinear arrays and collections of point data.  Interfaces to data
(for programmers) and operations (for users) should be polymorphic and
interoperable.

In addition, there is a need to develop data-format, structure and access
software benchmarks to investigate performance and compare implementations.
At a minimum, such benchmarks could be used to compare strengths and
weaknesses of extant systems, and to help focus the direction of future
implementations and standards.  In this context, performance is defined in
terms of ease of specification and extensibility, level of complexity that can
be represented, storage efficiency and actual access (read/write) times of
different classes of data for specific computer systems in common use, and
software and data portability.

Selected References

1.   Brown, S. A., M. Folk, G. Goucher and R. Rew.  "Software for Portable
Scientific Data Management".  Computers in Physics, 7, n.3, pp.304-308,
May/June 1993.

2.   Butler, D. M. and M. H. Pendley.  "The Visualization Management System
Approach to Visualization in Scientific Computing" and "A Visualization Model
Based on the Mathematics of Fiber Bundles", Computers in Physics, 3, n.5
Sept./Oct. 1989.

3.   Butler, D. M. and C. Hansen (ed.).  "Scientific Visualization
Environments:  A Report on a Workshop at Visualization '91".  Computer
Graphics, 26, n. 3, pp. 213-216, February 1992.

4.   Butler, D. M. and S. Bryson.  "Vector-Bundle Classes Form Powerful
Tool for Scientific Visualization".  Computers in Physics, 6, n.6, pp.
213-216, November/December 1992.

5.   Campbell, W. J., R. F. Cromp, G. Fekete, R. Wall and M. Goldberg.  Panel
on "Techniques for Managing Very Large Scientific Data Bases".  Proceedings
IEEE Visualization '92, pp. 362-365, October 1992.

6.   Faust, J. T. and D. S. Dyer.  "An Effective Data Format for Scientific
Visualization".  Proceedings of the SPIE/SPSE Symposium on Electronic Imaging,
February 1990.

7.   Fekete, G. "Rendering and Managing Spherical Data with Sphere Quadtrees".
Proceedings IEEE Visualization '90, pp. 176-186, October 1990.

8.   French, J. C., A. K. Jones, J. L. Pfaltz.  "A Summary of the NSF
Scientific Database Workshop".  Quarterly Bulletin of IEEE Computer Society
Technical Committee on Data Engineering, 13, n. 3, September 1990.

9.   Haber, R., B. Lucas and N. Collins.  "A Data Model for Scientific
Visualization with Provisions for Regular and Irregular Grids", Proceedings
IEEE Visualization '91, pp. 298-305, October 1991.

10.  Hibbard, W., C. R. Dyer and B. Paul.  "Display of Scientific Data
Structures for Algorithm Visualization".  Proceedings IEEE Visualization '92,
pp. 139-146, October 1992.

11.  "Definition of the Flexible Image Transport System".  NASA/OSSA
Office of Standards and Technology, June, 1993.

12.  Kochevar, P., Z. Ahmed, J. Shade and C. Sharp.  "Bridging the Gap
Between Visualization and Data Management:  A Simple Visualization
Management System".  Proceedings IEEE Visualization '93, pp. 94-101,
October 1993.

13.  Lang, U., R. Lang, R. R-hle.  "Integration of Visualization and
Scientific Calculation in a Software System, Proceedings IEEE Visualization
'91 Conference, October 1991.

14.  Li, Y. P., T. H. Handley, Jr., E. R. Dobinson.  "Data Hub: A Framework
for Science Data Management".  Submitted to 18th International Conference in
Very Large Data Bases, August 1992.

15.  NASA/OSSA Office of Standards and Technology.  "Definition of the
Flexible Image Transport System", June 1993.

16.  Pfau, L. M.  "The GridFile Tool Structure of System Files".
Eidgenossische Technische Hochschule, Zurich, 1990.

17.  Rew. R. K. and G. P. Davis.  "NetCDF: An Interface for Scientific Data
Access".  IEEE Computer Graphics and Applications, 10, n.4, July 1990,
pp. 76-82.

18.  Salem, K.  "MR-CDF: Managing Multi-Resolution Scientific Data".  Cent

of Excellence in Space Data and Information Systems Technical Report 928,
NASA/Goddard Space Flight Center, March 1992.

19.  Stonebraker, M., J. Chen, N. Nathan, C. Paxson, A. Su and J. Wu.
"Tioga: A Database-Oriented Visualization Tool".  Proceedings IEEE
Visualization '93, pp. 86-93, October 1993.

20.  Treinish, L. A. and M. L. Gough.  "A Software Package for the Data-
Independent Storage of Multi-Dimensional Data".  Eos Transactions American
Geophysical Union, 68, pp. 633-635, 1987.

21.  Treinish, L. A. (ed).  "Data Structures and Access Software for
Scientific Visualization", A Report on a Workshop at SIGGRAPH '90.  Computer
Graphics, 25, n. 2, April 1991.

22.  Treinish, L. A.  "Unifying Principles of Data Management for
Scientific Visualization".  Proceedings of the British Computer Society
Conference on Animation and Scientific Visualization, December 1992 and
Animation and Scientific Visualization Tools and Applications (R. Earnshaw
and D. Watson, editors), Academic Press, pp. 141-169, 1993.

23.  Treinish, L. A., M. Folk, G. Goucher, R. Kulkarni and R. Rew.  "Data
Models, Structures and Access Software for Scientific Visualization".
Proceedings IEEE Visualization '93, pp. 355-360, October 1993.

24.  Wells, D. C., E. W. Greisen and R. H. Harten.  "FITS: A Flexible Image
Transport System".  Astronomy and Astrophysics Supplement Series, 44, pp. 363-
370, 1981.

-- 
Ted Smith                                           MCNC
[email-masked]                         Information Technologies Division
Voice: 919-248-9232                  P.O. Box 12889, 3021 Cornwallis Rd.
Fax:   919-248-9245                 Research Triangle Park, NC 27709-2889

From: [email-masked]
Date: Wed, 16 Mar 1994 12:21:00 EST

RFC-822-headers:
Date: Wed, 16 Mar 1994 11:37:12 +0000
From: "William T. Smith" <[email-masked]>
Subject: Position Statement from Treinish
To: [email-masked] (John Ambrosiano),
 [email-masked] (Rajini Balay),
 [email-masked] (Tony Baraghimian),
 [email-masked] (Bruce Bargmeyer),
 [email-masked] (Bill Benjey), [email-masked] (Ed Bilicki),
 [email-masked] (Daewon Byun), [email-masked] (Jason Ching),
 [email-masked] (Carlie Coats), [email-masked] (Steve Davis),
 [email-masked] (Alison Eyth), [email-masked] (Steve Fine),
 [email-masked] (Mike Frame), [email-masked] (Ken Galluppi),
 [email-masked] (Frank Jou), [email-masked] (Carla Kennedy),
 [email-masked] (Ravi Kulkarni),
 [email-masked] (Deborah Leucken),
 [email-masked] (John Manzer), [email-masked] (Mike Neacsu),
 [email-masked] (Joan Novak),
 [email-masked] (Richard L. Peskin),
 [email-masked] (Jim Reagan),
 [email-masked] (Theresa Rhyne), [email-masked] (Lisa Roig),
 [email-masked] (Diane Sandoval), [email-masked] (Ted Smith),
 [email-masked] (Mladen Vouk),
 [email-masked] (Sandra Walther), [email-masked] (Bob Wayland),
 [email-masked] (Robert Weston)
Message-id: <[email-masked]>
X-Envelope-to: [email-masked],
 [email-masked], [email-masked],
 [email-masked]
X-Mailer: ELM [version 2.3 PL0]
Content-transfer-encoding: 7BIT