Subject: Re: some specific questions about data formats and visualization tools

From: Don Wells <[email-masked]>
Date: 24 Oct 1997 17:21:39 -0400

"JT" == Jeffrey Templon <[email-masked]> writes:
JT> .. We're investigating possible scientific data formats and analysis
JT> tools to use in multiparameter (multi means on the order of
JT> hundreds to thousands of parameters) measurements of nuclear and
JT> high-energy physics experiments..
JT> What we need from the data format:
JT> 	o ability to write multiparameter data sets in a
JT> 	  platform-independent fashion.  ..
JT> 	o ability to provide annotations (text strings) to specific
JT> 	  items in the data file..
JT> 	o should be able to interface the data format from Fortran-77,
JT> 	  Fortran-90, and C
JT> 	o should be possible to read in those huge multidimensional
JT> 	  arrays in chunks to avoid running out of memory ... 

Given these specifications, FITS [Flexible Image Transport System] is
definitely a candidate. Although FITS was designed by astronomers for
astronomy applications, the architecture is generalized, and could
easily be used in other science disciplines. I suggest that you
examine the FITSIO Subroutine Library:

    http://legacy.gsfc.nasa.gov/docs/software/fitsio/fitsio.html

and the FTOOLS package of utility programs:

    http://heasarc.gsfc.nasa.gov/docs/software/ftools/ftools_menu.html

A variety of FITS application and display programs are available in
the worldwide astronomy community.  I expect that your data problems
most closely resemble those of the high energy astrophysics community,
such as the people at HEASARC [High Energy Astrophysics Archival
Research Center] at GSFC.  That is why I recommend that you look first
at the HEASARC software (FITSIO and FTOOLS).

Take a look at the 'BINTABLE' (BINary TABLE) extension of FITS, not
the primary data array and 'IMAGE' extensions, which are used to
document and transmit simple N-dimensional matricies. The term
'extension' is FITS-speak for a data object --- with its own
self-describing header --- which is appended to the basic header and
data matrix with which all FITS files begin. The rules of FITS permit
data matricies of dimensionality zero, and so the primary matrix need
not be present, and the initial FITS header (2880 bytes) can be used
as a human-readable introduction to a bytestream file containing a
collection of arbitrary data objects (which we call 'extensions'). At
the present time three types of extensions (objects) are officially
defined and in common use in the FITS community: (1) 'IMAGE'
extensions (N-dimensional matricies), (2) 'TABLE' extensions (tabular
information coded as ASCII text) and (3) 'BINTABLE' extensions. New
types of objects (basic data structures) could be defined as needed,
but BINTABLE is strong enough to support almost any conceivable
application, so there is little incentive in the astronomy community
to create new types of extensions. Of course, it would be trivial to
agree to interchange FITS files containing extensions titled
XTENSION='MPEG' or XTENSION='MSWORD' if we wished to associate such
objects with our data objects in single files; the generalized
extensions architecture of FITS can encapsulate any other
bitstreams. So far we have not exploited this possibility.

The BINTABLE architecture is able to describe and transmit arrays of
arbitrary C structures (records) whose members consists of arrays of
various elementary data types (chars, ints, floats, doubles,..).
'BINTABLE' can do anything that 'TABLE' can do, and much more. For
example, the records can contain fields which are N-dimensional
matrices plus arbitrary character strings, booleans, integers and
floats which are needed to specify the physical parameters,
dimensionality, coordinate systems, etc, associated with those
matricies. The architecture even supports variable dimension matrices
in the records; it does this by defining a heap space and using a
pointer data type to offset into the heap. The ASCII (human-readable)
header of a BINTABLE extension describes the fields of its
records. The cost in bytes of the somewhat verbose ASCII text of the
header is easily amortized for even modest sets of binary records, and
the human readability of the header simplifies documentation and
assures that FITS bytestreams in archives are likely to be interpreted
fairly easily decades hence (the header comments should contain
bibliographic references to the published FITS specifications, of
course). Because a FITS file can contain an arbitrary number of such
BINTABLE extensions, FITS is an ideal self-documenting
machine-independent format for the transmission and archiving of
relational databases (multiple tables). In addition to the usual
scalars in database records, the fields of the tables expressed in
BINTABLE style can also contain vectors ("blobs"). Here are some URLs
for the BINTABLE architecture:

    The BINTABLE design document as it was published (45 KB, 8 pages):
    http://www.cv.nrao.edu/fits/documents/standards/bintable_aa.ps.gz 

    The FITS standard, including BINTABLE rules (113 KB, 80 pages):
    http://www.cv.nrao.edu/fits/documents/standards/fits_standard.ps.gz

Note that FITS is _not_ defined by an API [Application Programming
Interface].  Rather, it is defined as a machine-independent bitstream
file format, and multiple APIs (e.g., FITSIO) in multiple computer
languages (F77, C, C++, IPL, Java, ...) and with various
functionalities are available, generally under GNU terms, from various
astronomy software sources worldwide.

JT> what we want from a visualization tool:
JT> 	o display two-and three-dimensional plots of the data in a
JT> 	  variety of styles (contour, surface, scatterplot, etc.)
JT> 	o select one (or two) variable(s) and based on "gates", "cuts",
JT> 	  "limits", or "conditions" (pick your favorite word for it)
JT> 	  on some of the other variables' values, make frequency
JT> 	  (number of occurrences) histograms for the selected
JT> 	  variables.  It would be nice to be able to save the
JT> 	  thusly-generated histogram as an array into the data file.
JT> 	o be able to do some histogram or array manipulation.

Take a look at the programs of the FTOOLS package (URL above). I
expect that you will find much of what you want there, and that
special cases that you want can be created by cloning their programs
and modifying them as needed.

In addition to FTOOLS, several of the large data analysis packages
used in astronomy (MIDAS, IRAF, ST-SDAS) contain strong support for
general manipulation and display of tabular data, such as FITS tables.

There is a newsgroup devoted to FITS discussions:

    sci.astro.fits

See also:

FITS Support Office Home Page
http://ssdoo.gsfc.nasa.gov/astro/fits/fits_home.html

FITS archive at NRAO
http://www.cv.nrao.edu/fits/
-- 
  Donald C. Wells         Associate Scientist         [email-masked]
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

---

From: Craig Markwardt <[email-masked]>
Date: 25 Oct 1997 15:38:03 -0600

Jeffrey Templon <[email-masked]> writes:
> We're investigating possible scientific data formats and analysis
> tools to use in multiparameter (multi means on the order of
> hundreds to thousands of parameters) measurements of nuclear and
> high-energy physics experiments.  We've been using the HBOOK
> package from CERN, but we have found several limitations of
> this package.

I spent some time at SLAC as an undergraduate, and they were working
on several data management and visualization packages.  At the time
they were working on the NEXT architecture, but they very likely may
have moved on to X windows.

Among the packages:
  
  Hippoplotamus - general plotting routines
  Cheetah       - data management/storage package, including documentation
                  and system portability, web page at:
    http://www.slac.stanford.edu/FIND/FHMAIN.html?cheetah

  HippoPlot     - Integrated plotting and analysis package, including
                  data cutting and selecting, function minimization, etc.

                  You can check out the web page for this package at:
    http://www.slac.stanford.edu/FIND/FREEHEP/NAME/HIPPODRAW/FULL
                  and if you seem interested, contact the authors.  There
                  is a whole set of "FREEHEP" software indexed at 
    http://www.slac.stanford.edu/FIND/FHMAIN.html
                  which may be helpful.

These were packages developed several years ago.  If you contact the
authors they may have new developments.  Since they are already
working in HE particle physics, they may be particularly sensitive and
familiar with your concerns.

Another poster has already mentioned the solution of astronomy, namely
the FITS format.  In the US, the FITS format is used for distributing
almost all HE astrophysics data, including event, image (ie, array),
and other tabular data.  You will probably be most interested in
binary tables, which store data in logical columns (max number of
columns is 999, max number of rows is "unlimited").  Individual table
elements can be any of the fundamental data types, or even
multiple-dimension arrays themselves.

HEASARC, a part of NASA at Goddard Space Flight Center, is actively
archiving data from missions (http://legacy.gsfc.nasa.gov/0/) and has
a FITS working group
(http://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/ofwg_intro.html)

  FITSIO - for reading and writing FITS data
  FV     - for viewing FITS data structures, 
            http://heasarc.gsfc.nasa.gov/docs/software/ftools/fv.html

I am sure other readers of the news group could point you to packages
like HDF, netCDF, etc, but I don't have experience with these.

Good luck

Craig

-- 
------------------------------------
Craig Markwardt, Ph.D.              
EMAIL: [email-masked] 
------------------------------------