From zender@ncar.ucar.edu Wed Oct 8 10:16:59 1997 Path: newsfeed.cv.nrao.edu!hearst.acc.Virginia.EDU!uunet!in5.uu.net!news.monmouth.com!news.maxwell.syr.edu!newsfeed.internetmci.com!192.52.106.6!ncar!not-for-mail From: Charlie Zender Newsgroups: sci.data.formats Subject: nco-1.1 netCDF operators is released. Date: Tue, 07 Oct 1997 23:34:07 -0600 Organization: National Center for Atmospheric Research Lines: 32 Message-ID: <343B1B4F.68A67592@ncar.ucar.edu> NNTP-Posting-Host: 128.117.91.216 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; I; Linux 2.0.0 i686) Xref: newsfeed.cv.nrao.edu sci.data.formats:121 I am pleased to announce the release of version 1.1 of NCO, which stands for `netCDF operators'. NCO consists of ten command line operators for netCDF files: ncks ncrename ncra ncea ncrcat ncdiff ncwa ncecat ncflint ncatted These operators work on generic netCDF files and perform rote tasks like renaming, concatentation, hyperslabbing, averaging, interpolation, and differencing. The documentation is extensive, and available in many formats, both on-line and printed. Major new features of version 1.1 are ** New operator ncatted--netCDF Attribute Editor ** Windows NT supported ** Improved documentation The source code is freely available via anonymous ftp: ftp.cgd.ucar.edu:pub/zender/nc/nco-1.1.tar.gz The documentation is now on-line: http://www.cgd.ucar.edu/cms/nco/nco.html Good luck, Charlie -- Charlie Zender Voice, FAX: (303) 497-1612, 497-1324 NCAR ASP & CGD E-mail: zender@ncar.ucar.edu P.O. Box 3000 URL: http://www.cgd.ucar.edu/cms/zender Boulder CO 80307-3000 PGP: finger -l zender@odin.cgd.ucar.edu From dwells@fits.cv.nrao.edu Fri Oct 24 17:46:55 1997 Path: newsfeed.cv.nrao.edu!not-for-mail From: Don Wells Newsgroups: sci.data.formats Subject: Re: some specific questions about data formats and visualization tools Date: 24 Oct 1997 17:21:39 -0400 Organization: nrao Lines: 133 Message-ID: References: Reply-To: dwells@nrao.edu NNTP-Posting-Host: fits.cv.nrao.edu X-Attribution: Up X-No-Archive: yes X-BOFH-Archive: yes X-Newsreader: Gnus v5.3/Emacs 19.34 Xref: newsfeed.cv.nrao.edu sci.data.formats:139 "JT" == Jeffrey Templon writes: JT> .. We're investigating possible scientific data formats and analysis JT> tools to use in multiparameter (multi means on the order of JT> hundreds to thousands of parameters) measurements of nuclear and JT> high-energy physics experiments.. JT> What we need from the data format: JT> o ability to write multiparameter data sets in a JT> platform-independent fashion. .. JT> o ability to provide annotations (text strings) to specific JT> items in the data file.. JT> o should be able to interface the data format from Fortran-77, JT> Fortran-90, and C JT> o should be possible to read in those huge multidimensional JT> arrays in chunks to avoid running out of memory ... Given these specifications, FITS [Flexible Image Transport System] is definitely a candidate. Although FITS was designed by astronomers for astronomy applications, the architecture is generalized, and could easily be used in other science disciplines. I suggest that you examine the FITSIO Subroutine Library: http://legacy.gsfc.nasa.gov/docs/software/fitsio/fitsio.html and the FTOOLS package of utility programs: http://heasarc.gsfc.nasa.gov/docs/software/ftools/ftools_menu.html A variety of FITS application and display programs are available in the worldwide astronomy community. I expect that your data problems most closely resemble those of the high energy astrophysics community, such as the people at HEASARC [High Energy Astrophysics Archival Research Center] at GSFC. That is why I recommend that you look first at the HEASARC software (FITSIO and FTOOLS). Take a look at the 'BINTABLE' (BINary TABLE) extension of FITS, not the primary data array and 'IMAGE' extensions, which are used to document and transmit simple N-dimensional matricies. The term 'extension' is FITS-speak for a data object --- with its own self-describing header --- which is appended to the basic header and data matrix with which all FITS files begin. The rules of FITS permit data matricies of dimensionality zero, and so the primary matrix need not be present, and the initial FITS header (2880 bytes) can be used as a human-readable introduction to a bytestream file containing a collection of arbitrary data objects (which we call 'extensions'). At the present time three types of extensions (objects) are officially defined and in common use in the FITS community: (1) 'IMAGE' extensions (N-dimensional matricies), (2) 'TABLE' extensions (tabular information coded as ASCII text) and (3) 'BINTABLE' extensions. New types of objects (basic data structures) could be defined as needed, but BINTABLE is strong enough to support almost any conceivable application, so there is little incentive in the astronomy community to create new types of extensions. Of course, it would be trivial to agree to interchange FITS files containing extensions titled XTENSION='MPEG' or XTENSION='MSWORD' if we wished to associate such objects with our data objects in single files; the generalized extensions architecture of FITS can encapsulate any other bitstreams. So far we have not exploited this possibility. The BINTABLE architecture is able to describe and transmit arrays of arbitrary C structures (records) whose members consists of arrays of various elementary data types (chars, ints, floats, doubles,..). 'BINTABLE' can do anything that 'TABLE' can do, and much more. For example, the records can contain fields which are N-dimensional matrices plus arbitrary character strings, booleans, integers and floats which are needed to specify the physical parameters, dimensionality, coordinate systems, etc, associated with those matricies. The architecture even supports variable dimension matrices in the records; it does this by defining a heap space and using a pointer data type to offset into the heap. The ASCII (human-readable) header of a BINTABLE extension describes the fields of its records. The cost in bytes of the somewhat verbose ASCII text of the header is easily amortized for even modest sets of binary records, and the human readability of the header simplifies documentation and assures that FITS bytestreams in archives are likely to be interpreted fairly easily decades hence (the header comments should contain bibliographic references to the published FITS specifications, of course). Because a FITS file can contain an arbitrary number of such BINTABLE extensions, FITS is an ideal self-documenting machine-independent format for the transmission and archiving of relational databases (multiple tables). In addition to the usual scalars in database records, the fields of the tables expressed in BINTABLE style can also contain vectors ("blobs"). Here are some URLs for the BINTABLE architecture: The BINTABLE design document as it was published (45 KB, 8 pages): http://www.cv.nrao.edu/fits/documents/standards/bintable_aa.ps.gz The FITS standard, including BINTABLE rules (113 KB, 80 pages): http://www.cv.nrao.edu/fits/documents/standards/fits_standard.ps.gz Note that FITS is _not_ defined by an API [Application Programming Interface]. Rather, it is defined as a machine-independent bitstream file format, and multiple APIs (e.g., FITSIO) in multiple computer languages (F77, C, C++, IPL, Java, ...) and with various functionalities are available, generally under GNU terms, from various astronomy software sources worldwide. JT> what we want from a visualization tool: JT> o display two-and three-dimensional plots of the data in a JT> variety of styles (contour, surface, scatterplot, etc.) JT> o select one (or two) variable(s) and based on "gates", "cuts", JT> "limits", or "conditions" (pick your favorite word for it) JT> on some of the other variables' values, make frequency JT> (number of occurrences) histograms for the selected JT> variables. It would be nice to be able to save the JT> thusly-generated histogram as an array into the data file. JT> o be able to do some histogram or array manipulation. Take a look at the programs of the FTOOLS package (URL above). I expect that you will find much of what you want there, and that special cases that you want can be created by cloning their programs and modifying them as needed. In addition to FTOOLS, several of the large data analysis packages used in astronomy (MIDAS, IRAF, ST-SDAS) contain strong support for general manipulation and display of tabular data, such as FITS tables. There is a newsgroup devoted to FITS discussions: sci.astro.fits See also: FITS Support Office Home Page http://ssdoo.gsfc.nasa.gov/astro/fits/fits_home.html FITS archive at NRAO http://www.cv.nrao.edu/fits/ -- Donald C. Wells Associate Scientist dwells@nrao.edu http://fits.cv.nrao.edu/~dwells National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road, Charlottesville, Virginia 22903-2475 USA From craigm@cow.physics.wisc.edu Sat Oct 25 19:33:39 1997 Path: newsfeed.cv.nrao.edu!newsgate.duke.edu!nntprelay.mathworks.com!news.maxwell.syr.edu!newsfeeds.sol.net!news.pagesat.net!news.itis.com!news.doit.wisc.edu!not-for-mail From: Craig Markwardt Newsgroups: sci.data.formats Subject: Re: some specific questions about data formats and visualization tools Date: 25 Oct 1997 15:38:03 -0600 Organization: U. Wisc. Madison Physics -- Compact Objects Lines: 68 Message-ID: References: NNTP-Posting-Host: cow.physics.wisc.edu X-Newsreader: Gnus v5.3/Emacs 19.33 Xref: newsfeed.cv.nrao.edu sci.data.formats:140 Jeffrey Templon writes: > We're investigating possible scientific data formats and analysis > tools to use in multiparameter (multi means on the order of > hundreds to thousands of parameters) measurements of nuclear and > high-energy physics experiments. We've been using the HBOOK > package from CERN, but we have found several limitations of > this package. I spent some time at SLAC as an undergraduate, and they were working on several data management and visualization packages. At the time they were working on the NEXT architecture, but they very likely may have moved on to X windows. Among the packages: Hippoplotamus - general plotting routines Cheetah - data management/storage package, including documentation and system portability, web page at: http://www.slac.stanford.edu/FIND/FHMAIN.html?cheetah HippoPlot - Integrated plotting and analysis package, including data cutting and selecting, function minimization, etc. You can check out the web page for this package at: http://www.slac.stanford.edu/FIND/FREEHEP/NAME/HIPPODRAW/FULL and if you seem interested, contact the authors. There is a whole set of "FREEHEP" software indexed at http://www.slac.stanford.edu/FIND/FHMAIN.html which may be helpful. These were packages developed several years ago. If you contact the authors they may have new developments. Since they are already working in HE particle physics, they may be particularly sensitive and familiar with your concerns. Another poster has already mentioned the solution of astronomy, namely the FITS format. In the US, the FITS format is used for distributing almost all HE astrophysics data, including event, image (ie, array), and other tabular data. You will probably be most interested in binary tables, which store data in logical columns (max number of columns is 999, max number of rows is "unlimited"). Individual table elements can be any of the fundamental data types, or even multiple-dimension arrays themselves. HEASARC, a part of NASA at Goddard Space Flight Center, is actively archiving data from missions (http://legacy.gsfc.nasa.gov/0/) and has a FITS working group (http://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/ofwg_intro.html) FITSIO - for reading and writing FITS data FV - for viewing FITS data structures, http://heasarc.gsfc.nasa.gov/docs/software/ftools/fv.html I am sure other readers of the news group could point you to packages like HDF, netCDF, etc, but I don't have experience with these. Good luck Craig -- ------------------------------------ Craig Markwardt, Ph.D. EMAIL: craigm@lheamail.gsfc.nasa.gov ------------------------------------