Subject: Welcome to sci.data.formats!!!

From: [email-masked] (Briand T. Sanderson)
Date: Mon, 7 Jun 1993 20:40:22 GMT

I would like to welcome everyone to the sci.data.formats news group.

The following is the "official" charter for the newsgroup:

CHARTER:

  an UNMODERATED Usenet newsgroup called "sci.data.formats"
  for discussing the modeling, storage, and retrieval of scientific data.

PURPOSE:

  The purpose of the newsgroup "sci.data.formats" would be for discussing
  information related to the use of the scientific data formats such
  as HDF, netCDF, FITS, and various others.

  Topics of discussion could include problems encountered using
  specific formats, questions about where to get the code for data
  format programs, discussions of specific uses of data format
  programs, etc.

  It is anticipated that readers of the group would be scientists (and
  software developers) who use data formats/models such as HDF,
  netCDF, and  other such programs.

RATIONALE:

  There are many data formats designed specifically for
  scientific applications.  Many have been adopted as a standard by 
  scientific organizations ranging over such diverse fields as
  astronomy, physics, chemistry, medical research, health care, space
  science, atmospheric sciences, earth sciences and environmental
  science.  

  It is hoped that the creation of the group, "sci.data.formats" will
  encourage and facilitate a greater amount of collaboration and sharing
  of information and experience between individuals and groups.

--
Briand Sanderson
NCSA HDF Research Programmer
[email-masked]

---

From: [email-masked] (Russ Rew)
Date: Wed, 16 Jun 1993 18:11:29 GMT

[email-masked] (Lloyd Harding) writes:

>Could we have a list of these formats, their intended purpose (what they
>are good for, not good for), their interchangability etc. established in a
>FAQ?  Personally, I am ignorant of the formats listed above but am fairly
>well embedded in free flowing text formats and would like to become better
>acquainted with these formats.

Here's the top of the netCDF Frequently Asked Questions file:

                 Frequently Asked Questions About netCDF

This article contains answers to some of the most frequently asked netCDF
questions on the netcdfgroup mailing list and in the email sent to
[email-masked].  The list is archived in pub/netcdf/FAQ on
unidata.ucar.edu, and is currently maintained by Russ Rew
([email-masked]).

List of Questions:

    1:  What Is netCDF?
    2:  How do I get the netCDF software package?
    3:  What does netCDF run on?
    4:  When will the next version be available?  What changes are included?
    5:  What is the connection between netCDF and CDF?
    6:  What is the connection between netCDF and HDF?
    7:  Is netCDF available for the Apple Macintosh?
    8:  What are some references to netCDF?
    9:  What are the files in pub/netcdf/ on unidata.ucar.edu?
    10: Why do I get XDR error messages when trying to write data?
    11: What is the best way to represent [some particular data] using netCDF?
    12: Are there plans to add facilities for data compression to netCDF?
    13: What utilities are available for netCDF?
    14: What is the status of implementing additional netCDF operators?
    15: Is there a mailing list for netCDF discussions and questions?
    16: How do I make a bug report?
    17: Is there any WAIS or gopher access to netCDF information?
    18: How widely is netCDF used?

1:  What Is netCDF?

A:  NetCDF (network Common Data Form) is an interface for scientific data
    access and a freely-distributed software library that provides an
    implementation of the interface.  It was developed by Glenn Davis, Russ
    Rew, and Steve Emmerson at the Unidata Program Center in Boulder,
    Colorado.  The netCDF library also defines a machine-independent format
    for representing scientific data.  Together, the interface, library, and
    format support the creation, access, and sharing of scientific data.

    netCDF data is:

    - Self-Describing.  A netCDF file includes information about the data it
      contains.

    - Network-transparent.  A netCDF file is represented in a form that can
      be accessed by computers with different ways of storing integers,
      characters, and floating-point numbers.

    - Direct-access.  A small subset of a large dataset may be accessed
      efficiently, without first reading through all the preceding data.

    - Appendable.  Data can be appended to a netCDF dataset along one
      dimension without copying the dataset or redefining its structure.
      The structure of a netCDF dataset can be changed, though this
      sometimes causes the dataset to be copied.

    - Sharable.  One writer and multiple readers may simultaneously access
      the same netCDF file.

2:  How do I get the netCDF software package?

A:  Via anonymous FTP from

	host:	unidata.ucar.edu [128.117.140.3]
	file:	pub/netcdf/netcdf.tar.Z

    Make sure the file is transmitted in BINARY mode.  This is version 2.3.2,
    last updated in June 1993.

3:  What does netCDF run on?

A:  Version 2.3.2 of netCDF has been tested on the following platforms:

	CRAY Y-MP	  UNICOS 6.1.6
	DEC Alpha	  OSF/1 1.2
	DEC VAX		  VMS 5.5-2
	DEC VAX		  Ultrix 4.3
	DECstation 3000	  Ultrix 4.3
	HP-9000/7xx	  HPUX 9.0
	IBM PS/2	  MSDOS 5.0
	IBM PS/2	  OS/2 1.2
	IBM RS-6000	  AIX 3.2
	NeXT		  NeXTOS 3.0
	SGI Iris	  IRIX 4.0.5F
	SPARCstation	  Solaris 2.1
	SPARCstation	  SunOS 4.1.3

    The experience of outside developers indicates that netCDF is relatively
    easy to port to any system that has a C compiler and an XDR library
    (eXternal Data Representation, used for NFS and usually supplied by
    vendors).

4:  What has changed since the netCDF 2.2 release in December 1991?

A:  NetCDF 2.3, made available in April 1993, contains bug fixes, portability
    enhancements, performance enhancements, and a few new capabilities.

    Some new optimizations for the library result in significant speedups
    for accessing cross-sections involving non-contiguous data.

    New capabilities include some additional interfaces that provide a more
    general form of hyperslab access.  This supports sub-sampling along
    specified dimensions and a mapping between the points of the hyperslab
    and the memory locations of the corresponding values.  In a generalized
    hyperslab, an index mapping vector is used to define the mapping between
    points in the generalized hyperslab and the memory locations of the
    corresponding values, so data values that are written or read need no
    longer be contiguous in memory.

    There are also some new interfaces that can be used to write, read, and
    inquire about records, where a record may contain multiple variables of
    different types and shapes.  Where before you had to access a record's
    worth of data using multiple calls, now you will be able to use a single
    call.

    The ncdump utility supports several new command-line options including
    the ability to specify for which variables data values will be output,
    to provide brief annotations in the form of CDL comments to identify
    data values for large multidimensional variables, or to provide full
    annotations in the form of trailing CDL comments for every data value.

    The current release also includes a prototype implementation of a C++
    interface for the netCDF data access library.  It provides all the
    functionality of the previous C interface, improves type safety by
    eliminating use of void* pointers, and is somewhat simpler to use than
    the C interface.  With the C++ interface, no IDs are needed for netCDF
    components, there is no need to specify types when creating attributes,
    and less indirection is required for dealing with dimensions.  However,
    since this is a prototype interface and implementation, it may be
    changed before a supported version is released.

5:  What is the connection between netCDF and CDF?

A:  CDF was developed at the NASA Space Science Data Center at Goddard, and
    is freely available.  It was originally a VMS FORTRAN interface for
    scientific data access.  Unidata reimplemented the library from scratch
    to use XDR for a machine-independent representation, designed the CDL
    text representation for netCDF data, wrote a User's Guide and made other
    additions including aggregate data access, single-file implementation,
    named dimensions, and variable-specific attributes.

    NetCDF and CDF have evolved independently.  CDF now supports many of the
    same features as netCDF (aggregate data access, XDR representation,
    single-file representation, variable-specific attributes), but some
    differences remain (netCDF doesn't support native-mode representation,
    CDF doesn't support named dimensions).  There is no compatibility
    between data in CDF and netCDF form, and as yet no translation software
    exists to convert data in one form to data in the other form.

6:  What is the connection between netCDF and HDF?

A:  The National Center for Supercomputing Applications (NCSA) developed the
    HDF software and makes it freely available.  HDF is an extensible data
    format for self-describing files that was developed independently of
    netCDF.  Applications and utilities based on HDF are available that
    support raster-image manipulation and display and browsing through
    multidimensional scientific data. The HDF software includes a package of
    routines for accessing each HDF data type, as well as a lower-level
    interface for building packages to support new types.  HDF supports both
    C and Fortran interfaces, and it has been successfully ported to a wide
    variety of machine architectures and operating systems.  HDF emphasizes
    a single common format for data, on which many interfaces can be built.

    NCSA has implemented software that provides a netCDF interface to HDF.
    With this software, it is possible to use the netCDF calling interface
    to place data into an HDF file.  The netCDF calling interface has not
    changed and netCDF files stored in XDR format are readable, so existing
    programs and data will still be usable (although programs will need to
    be relinked to the new library).  There is currently no support for the
    mixing of HDF and netCDF structures.  For example, a raster image can
    exist in the same file as a netCDF object, but you have to use the
    Raster Image interface to read the image and the netCDF interface to
    read the netCDF object.  The other HDF interfaces are currently being
    modified to allow multi-file access, closer integration with the netCDF
    interface will probably be delayed until the end of that project.

    Eventually, it may be possible to integrate netCDF objects with the
    rest of the HDF tool suite.  Such an integration will then allow tools
    written for netCDF and tools written for HDF to both interact
    intelligently with the new data files.

 ...

--
________________________________________________________________________
Russ Rew					Unidata Program Center
[email-masked]				UCAR, PO Box 3000
						Boulder, CO 80307-3000