From topher@pobox.com Tue Oct 1 13:47:26 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!news.sprintlink.net!news-dc-9.sprintlink.net!www.nntp.primenet.com!nntp.primenet.com!howland.erols.net!newsfeed.internetmci.com!news.Traveller.COM!news From: Chris _topher_ Lakey Newsgroups: sci.data.formats,comp.object.corba Subject: Scientific Data Model and IDL Interface Date: Tue, 24 Sep 1996 19:14:08 -0500 Organization: Physitron, Inc. Lines: 28 Message-ID: <32487950.52BF@pobox.com> NNTP-Posting-Host: terminator.physitron.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; I; IRIX 5.3 IP22) CC: CORBA-DEV , Physitron Visualization Group Xref: solitaire.cv.nrao.edu sci.data.formats:1615 comp.object.corba:2324 We are in the process of developing a data model for the purpose of exchanging scientific data between different Navy applications. The data model will be specified by an English language document and a set of APIs. These APIs will be expressed in OMG IDL. Our work will not be to describe a storage format, but to define a mechanism to exchange data. It is possible that someone could use this API to move data to a file and then restore data using the API. It would also be possible to read data from an existing format (e.g. HDF, netCDF and Plot3D) using the same API, provided the proper implementation objects exist. However, the primary purpose it to provide data transportation between applications such as simulators, visualization tools, database servers, etc... Is anyone aware of any previous work using IDL to define interfaces for a scientific data model? What is the state of the Data Interchange facility and will it meet the needs of the scientific community? -topher -- . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris "topher" Lakey "Only two things that money can't buy mailto:topher@pobox.com That's true love & homegrown tomatoes" http://www.pobox.com/~topher/ http://www.pobox.com/~topher/guyclark/ (205) 534-4844 FAX: 534-4846 *** THIS SPACE FOR RENT *** . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From dwd@pmel.noaa.gov Mon Oct 7 09:44:20 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!aruba.odu.edu!news.larc.nasa.gov!news.msfc.nasa.gov!news.sgi.com!enews.sgi.com!news.mathworks.com!wizard.pn.com!news.gte.com!news-in.tiac.net!uunet!in1.uu.net!news.u.washington.edu!root From: "Donald W. Denbo" Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Tue, 01 Oct 1996 08:07:14 -0700 Organization: NOAA/PMEL/OCRD Lines: 28 Message-ID: <325133A2.871@pmel.noaa.gov> References: <52ovvv$f7n@coranto.ucs.mun.ca> NNTP-Posting-Host: merlin.pmel.noaa.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 2.02 (WinNT; I) CC: dwd@pmel.noaa.gov, nns@pmel.noaa.gov Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4931 sci.data.formats:1619 Dave Senciall wrote: > > > My questions are : > > 1) Are there any other good formats/softwares that I have missed that may be > of use. > > 2) If you have used any of these (netcdf HDF other) can you comment on their > suitability (or more importantly UNsuitability) for my purposes. > > Any suggestions comments or criticisms are welcome and appreciated.. > > Dave Senciall Dave, You might take a look at http://www.pmel.noaa.gov/epic/home.html EPIC is a collection of analysis tools that use the netCDF format. EPIC is designed for use with oceanographic data sets and new tools are being brought on line all the time. The software is freely available from anonymous ftp at ftp://ftp.pmel.noaa.gov/epic. Don Denbo NOAA/PMEL/OCRD 206 526-4487 dwd@pmel.noaa.gov or epic@pmel.noaa.gov From rmg3@access5.digex.net Mon Oct 7 09:45:03 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!news-server.ncren.net!concert!rutgers!news.sgi.com!howland.erols.net!news2.digex.net!digex.net!not-for-mail From: rmg3@access5.digex.net (Robert Grumbine) Newsgroups: sci.geo.oceanography,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 3 Oct 1996 12:40:42 -0400 Organization: Under construction Lines: 29 Message-ID: <530qaa$b4i@access5.digex.net> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <52v3ij$1se@merlin.llnl.gov> NNTP-Posting-Host: access5.digex.net Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4942 sci.data.formats:1621 In article <52v3ij$1se@merlin.llnl.gov>, Don Eliason wrote: > >In my opinion, netCDF is fine for oceanography data, but more so for 2D and >3D data (e.g., the Levitus data sets; model output) than for station data. >The JGOFS data format may be better for station data. The ECMWF uses Grib >and/or Grads (difference = ?). Grib is the World Meteorological Organization's standard for gridded fields. It isn't a very good thing for station data. For that, the WMO standard is BUFR. Grads is a freely available software package for graphic display. Among other things, it can read grib files. Bufr is probably a good design for the ocean station data since atmospheric soundings were one of the drivers in bufr's development. Having spoken of grib, one of the things about grib is that one can have a different mapping of parameter number to physical variable. I maintain such a table and would appreciate suggestions on additions. The table (and some other ocean related material) is at http://polar.wwb.noaa.gov/. The page, of course, is under development with significant changes and additions expected in the next copule of weeks. -- Bob Grumbine rmg3@access.digex.net Sagredo (Galileo Galilei) "You present these recondite matters with too much evidence and ease; this great facility makes them less appreciated than they would be had they been presented in a more abstruse manner." Two New Sciences From hrodstein@aol.com Mon Oct 7 09:45:54 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!math.ohio-state.edu!howland.erols.net!news-peer.gsl.net!news.gsl.net!portc01.blue.aol.com!newstf01.news.aol.com!newsbf02.news.aol.com!not-for-mail From: hrodstein@aol.com (HRodstein) Newsgroups: sci.data.formats Subject: Re: MS Excel Data format Date: 4 Oct 1996 00:06:13 -0400 Organization: America Online, Inc. (1-800-827-6364) Lines: 16 Sender: root@newsbf02.news.aol.com Message-ID: <5322fl$ql6@newsbf02.news.aol.com> References: Reply-To: hrodstein@aol.com (HRodstein) NNTP-Posting-Host: newsbf02.mail.aol.com > I would like to know the internal data format of Microsoft > Excel 5.0. I am actually developing a software and am trying to > introdue a feature to save my text data in Excel format. This is described in the Excel Developer's Kit from Microsoft Press. It is a fairly complicated format made much more complicated by the fact that Excel files are OLE compound documents. Thus, in addition to understanding the Excel file format, you need to know a bit about OLE and be able to call OLE libraries to write the compound file. Good luck! Howard Rodstein WaveMetrics (makers of IGOR Pro - Mac scientific graphing and data analysis) www.wavemetrics.com From topher@pobox.com Mon Oct 7 09:46:52 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!news-server.ncren.net!concert!rutgers!news.sgi.com!news.mathworks.com!newsfeed.internetmci.com!news.Traveller.COM!news From: Chris _topher_ Lakey Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Thu, 03 Oct 1996 17:51:34 -0500 Organization: Physitron, Inc. Lines: 31 Message-ID: <32544376.4B76@pobox.com> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> NNTP-Posting-Host: terminator.physitron.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; I; IRIX 5.3 IP22) Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4946 sci.data.formats:1625 Another option would be WMO GRIB. The navy seems to be pretty heavy into GRIB right now. It's worth checking out. I've been researching this lately and I think the netCDF data model is much better than the HDF SDS data model, but I don't know much about the netCDF binary data format. HDF is fairly standard (adopted by EOS) so you might want to consider using the netCDF features of HDF. The NCAR Graphics package uses the netCDF data model as the basis for their data model, so if you have access to NCAR Graphics (version 4+ I think) you might want to consider netCDF. Also, netCDF has an ASCII format (netCDL) that's very readable. If you're doing any type of distributed stuff, check out http://dods.gso.uri.edu/DODS/ The Distributed Oceanographic Data System They provide some distributed versions of the netCDF API. Ok, while I'm rambling, netCDF has APIs for several languages including C++. Good luck! -topher -- . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris "topher" Lakey "Only two things that money can't buy mailto:topher@pobox.com That's true love & homegrown tomatoes" http://www.pobox.com/~topher/ http://www.pobox.com/~topher/guyclark/ (205) 534-4844 FAX: 534-4846 *** THIS SPACE FOR RENT *** . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From davis@unidata.ucar.edu Mon Oct 7 09:47:27 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!uunet!in3.uu.net!news.sgi.com!news.mathworks.com!howland.erols.net!vixen.cso.uiuc.edu!uchinews!ncar!usenet From: "Glenn P. Davis" Newsgroups: sci.data.formats Subject: netcdf-3.1a binary for 32 bit Windows available Date: Thu, 03 Oct 1996 17:57:16 -0600 Organization: National Center for Atmospheric Research/Boulder, CO Lines: 26 Message-ID: <325452DC.41C6@unidata.ucar.edu> NNTP-Posting-Host: binnie.unidata.ucar.edu Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; U; IRIX64 6.2 IP26) Versions of netcdf-3.1 "C" library for 32 bit Windows platforms are available. These were built on an NT 4.0 system using Microsoft Visual C++ 4.2. The resulting binaries and the include file are available as ftp://ftp.unidata.ucar.edu/pub/netcdf/nc31a.zip This includes 4 compiles: a debug library (.LIB), a debug .DLL, an optimized library and an optimized .DLL You can find out more about netcdf at http://www.unidata.ucar.edu/packages/netcdf and more about the 3.1a release at http://www.unidata.ucar.edu/packages/netcdf/3.1a.html Enjoy. -glenn Glenn P. Davis davis@unidata.ucar.edu UCAR / Unidata PO Box 3000 3300 Mitchell Lane, Suite 170 Boulder, CO 80307-3000 Boulder, CO 80301 (303) 497 8643 From senciall@sunny.nwafc.nf.ca Wed Oct 9 09:54:57 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!swrinde!hookup!news.nstn.ca!coranto.ucs.mun.ca!usenet From: Dave Senciall Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 2 Oct 1996 19:02:39 GMT Organization: Physcial Oceanography,NWAFC,DFO, Gov of Canada Lines: 60 Message-ID: <52ue8f$2h1@coranto.ucs.mun.ca> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> NNTP-Posting-Host: hobbes.nwafc.nf.ca Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 1.1N (X11; I; SunOS 4.1.3 sun4c) X-URL: news:3252A213.3047@ios.bc.ca Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4950 sci.data.formats:1628 rich pawlowicz wrote: >Dave Senciall wrote: >> >> I'm in the process of evaluating options for reworking our oceanographic >> data archives, (CTD, current meter, T-chain, ADCP, timeseries thermographs >> etc). > > >> 3) compact binary storage. >> 4) some degree of platform independence for the files/software (we use Sun - >> unix for are archives but do a lot of work on the PC, also DEC ALPHA >> /openVMS may creep into the picture) > >Just curious - but why binary storage? The amount of ascii data >you have in even a very comprehensive >CTD/current meter dataset is not all that large, comparatively. The storage >breakpoint between binary CTD + (e.g.) netCDF libraries vs. ascii CTD is fairly >high. > >For example, I have in one directory 250 CTD files in IOS header-file format >(more header info than data in this format!); they take up 2M. The HDF >distribution (which I happen to have around for some reason or other) >appears to take up something like 13M...and as a user, I much >prefer being able to look at ascii files. > > > > >-- >Rich Pawlowicz, ph: (604) 363-6339 fax: (604) 363-6798 >Acoustical Oceanography Research Group, Institute of Ocean Sciences, >PO Box 6000, 9860 West Saanich Road, Sidney, B.C. CANADA V8L 4B2 >email: rich@ios.bc.ca WWW: http://pinger.ios.bc.ca/people/rich Rich, I have approximately 450,000 Ctd/xbt/bottle files (not counting other instrument types(30 ADCP cruises,c-meter,t-graphs)) ranging in size from a a single level to 10,000 levels, at ,say, 6 bytes per value in ascii (eg 32.456) for pressure temp sal plus derived channels that adds up to a lot of disk space, the same value (32.456) could be stored as 32456 in a 2 byte unsigned int, giving a saving of 66% not counting the white spaces between data columns. One of my concerns is that netcdf or hdf would be gready since they are general. And I might be better served using my own formats if space and speed were my biggest concerns. One of my goals however is to bring a 'standard theme' to the data sets. I hope that netcdf or similar might provide that. Additionally since there are various public and commercial software that can read netcdf format this might prove an added advantage. I agree that ascii is nice to view, but if you provide a utility to let you view the binary file then the effect is the same. Dave Senciall Physical Oceanography Section NWAFC St.John's NF senciall@sunny.nwafc.nf.ca From topher@pobox.com Wed Oct 9 09:55:09 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!news-server.ncren.net!concert!rutgers!news.sgi.com!news.mathworks.com!newsfeed.internetmci.com!news.Traveller.COM!news From: Chris _topher_ Lakey Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Thu, 03 Oct 1996 17:51:34 -0500 Organization: Physitron, Inc. Lines: 31 Message-ID: <32544376.4B76@pobox.com> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> NNTP-Posting-Host: terminator.physitron.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; I; IRIX 5.3 IP22) Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4946 sci.data.formats:1625 Another option would be WMO GRIB. The navy seems to be pretty heavy into GRIB right now. It's worth checking out. I've been researching this lately and I think the netCDF data model is much better than the HDF SDS data model, but I don't know much about the netCDF binary data format. HDF is fairly standard (adopted by EOS) so you might want to consider using the netCDF features of HDF. The NCAR Graphics package uses the netCDF data model as the basis for their data model, so if you have access to NCAR Graphics (version 4+ I think) you might want to consider netCDF. Also, netCDF has an ASCII format (netCDL) that's very readable. If you're doing any type of distributed stuff, check out http://dods.gso.uri.edu/DODS/ The Distributed Oceanographic Data System They provide some distributed versions of the netCDF API. Ok, while I'm rambling, netCDF has APIs for several languages including C++. Good luck! -topher -- . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chris "topher" Lakey "Only two things that money can't buy mailto:topher@pobox.com That's true love & homegrown tomatoes" http://www.pobox.com/~topher/ http://www.pobox.com/~topher/guyclark/ (205) 534-4844 FAX: 534-4846 *** THIS SPACE FOR RENT *** . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From saunders@aps.anl.gov Wed Oct 9 09:55:53 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!cs.utexas.edu!www.nntp.primenet.com!nntp.primenet.com!howland.erols.net!vixen.cso.uiuc.edu!milo.mcs.anl.gov!usenet From: Claude Saunders Newsgroups: sci.data.formats Subject: Re: continuous archiving with HDF Date: Mon, 07 Oct 1996 10:36:18 -0500 Organization: Argonne National Laboratory Lines: 26 Message-ID: <32592372.485D@aps.anl.gov> References: <32558603.52AF@aps.anl.gov> NNTP-Posting-Host: poseidon.aps.anl.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (X11; I; SunOS 5.5.1 sun4m) CC: saunders@anl.gov > > Au contraire! We use HDF 4.0 to append acquired data (ultrasound > data, to be precise) to a file. It even works with several channels > opened at the same time. > While the old DF interface for SDSs did not provide for appending, > the newer SD interface does. You can keep one dimension of each > data set open, using the SD_UNLIMITED specifier. Works quite well. > > Andreas Schumm > Electricite de France True! But my point was different. I need the ability to read >from an HDF file WHILE it is currently being appended to by some other tool. In other words, the HDF interface has to support simultaneous producer and consumer clients. Presumably, in your case, you cannot plot or otherwise analyze your data until you have completed collecting all of it, and have closed the file. I don't see anything in the interface that supports any finer level of control than "open" and "close". The data (and updated offsets, etc) are not flushed to the file until a complete file close is done. The interface needs to support some form of explicit buffer management, and file locking. Thank you for responding! From ejh@larry.gsfc.nasa.gov Mon Oct 14 10:35:45 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!news.msfc.nasa.gov!elroy.jpl.nasa.gov!lll-winken.llnl.gov!ames!newsfeed.gsfc.nasa.gov!usenet From: ejh@larry.gsfc.nasa.gov (Edward Hartnett) Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 04 Oct 1996 15:04:58 -0400 Organization: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA Lines: 35 Message-ID: References: <52ovvv$f7n@coranto.ucs.mun.ca> NNTP-Posting-Host: larry.gsfc.nasa.gov In-reply-to: Dave Senciall's message of 30 Sep 1996 17:28:31 GMT X-Newsreader: Gnus v5.1 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4956 sci.data.formats:1633 I don't think you should worry about the overhead too much. A properly written HDF or netCDF file ill not have significant amounts of overhead. THat is, it will be almost as small as you could get it with any custom definition. Certainly I think you should stay far far away from your own formats. It's just a dead end. If you're really smart and spend a lot of time on it you'll end up with a format almost as useful as HDF, except without all the free tools etc. I prefer HDF myself, and it does have a good way to store station data. I believe netCDF is more often used in the oceanagraphic field and it does have one nifty feature. With a recompile of your code you can get it to write HDF files instead wiothout changing any code. This is because the HDF people stole the netCDF library and incorporated it. Note that the two file formats are very different. But with a recompile the exact same code that output netCDF will output HDF files. You will be tempted to do your own custom formats when you start because it will seem like too much of a pain to use any standard format. But beliieve me this is a mistake that you and future programmers who work with you will regret. Pick a standard, don't make up a new one. Heaven knows we have enough "universal" data formats already! -- Edward Hartnett ejh@larry.gsfc.nasa.gov * I don't speak for NASA or ARC, I just work for them! * Goddard Distributed Active Archive Center (DAAC) Check out cool Earth Science data at: http://daac.gsfc.nasa.gov/ From senciall@sunny.nwafc.nf.ca Mon Oct 14 10:38:17 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!news-peer.gsl.net!news.gsl.net!news.mathworks.com!newsfeed.internetmci.com!in3.uu.net!ott.istar!istar.net!tor.istar!east.istar!news.nstn.ca!coranto.ucs.mun.ca!usenet From: Dave Senciall Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 10 Oct 1996 12:55:57 GMT Organization: Physcial Oceanography,NWAFC,DFO, Gov of Canada Lines: 76 Message-ID: <53irot$77m@coranto.ucs.mun.ca> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53g8vd$ibi@highway.leidenuniv.nl> <325BCDED.2461@pmel.noaa.gov> NNTP-Posting-Host: hobbes.nwafc.nf.ca Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 1.1N (X11; I; SunOS 4.1.3 sun4c) To: senciall@sunny.nwafc.nf.ca X-URL: news:325BCDED.2461@pmel.noaa.gov Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4960 sci.data.formats:1635 "Donald W. Denbo" wrote: >I've been following this thread for while now and one aspect of data >management that has not been addressed it how do you keep track of >what processing has been done to the data? We use netCDF to store our >CTD data (over 50,000 stations) and the processing programs >automatically >include comment lines documenting what has been done. We had a >in house developed format (ASCII headers, binary data) and have >developed a library (epslib >http://www.pmel.noaa.gov/epic/eps-overview.html ) to provide a file >independent way of accessing the data. The eps library allows us to >continue to use the old data format with the new netCDF format! > >One big problem of ASCII data ... People can pull it into an editor and >change values as well as header information. Those changes are very >difficult to find. If you have a lot of data and are doing a lot of >interactive processing large ASCII data files are a lot slower to read. >If you store them compressed, not only do you have to uncompress them, >but you have to convert the ASCII the ascii during a formated read. > >The final advantage that netCDF provides for us... we have a fairly >large user group, and a self-describing data format with the appropriate >well designed software, means a user doesn't have to know which of >dozens of formats the data is written in. > I agree, actually issues of logging of processing activities,quality control etc. are factors that I must consider, Often if an internation study program,or nation archive reqests you provide data to them they want to know what you have done to it, the processing and manipualtion history is often as important as the actual data . Another issue is that of 'indexing' or cataloguing the data for searching, It's fine to stick 500,000 profiles into a archive but if you can't readily search and rapidly extract data you can't accomplish much. When I first posted my question I had no idea of that number of responces that I would recieve ,nor the variety of opions aired. I think the fundamental thing that distingushes the respondents is the 'scope' of the projects that they work on. If your data set is small and/or personnel then 'internation standard' quality data formats are likely more of an inconvience than an asset. However if your group is large, your datasets large and varied, if you have to exchange data with other institutes and you have to manage data that is not part of 'your person project' but is more of an institutional asset then formats like netCDF and HDF are seriously worth considering. My currant archives are ascii based using unix compress for storage efficieny for low usage data. Space and performance are not excessively pressing issues, though in 5 years when the data sets have doubled in size they will be. My concern centered more on access, as it is now a researcher looking at ,say, water temperature in a region can access ctd data. If he wants to get temperature data from a current meter sensor in the same region he has to access a different data set that is in a totally unrelated format for meta data and sensor data that is controlled by total different software, similarly if they wanted to get the thermister data from the ADCP head in a ship, the software and data formats are totally un-related. My goal is to, at the least, get a common software interface to the data. ie point your program at desired dataset and read out temperature data, with little effort having to be put into 'knowing' the format of that data. Beyond that it would be nice that the 'style' of each data set have a common 'theme' so that when a new instrument comes along you have a 'formula' for designing the data storage rather that the add-hock method that people tend to fall into where the archieve ends up begining a version of the acquistion software format, or someones pet format that they dreamed up over a beer. Dave Senciall Physical Oceanography Section senciall@sunny.nwafc.nf.ca NorthWest Atlantic Fisheries Centre ph. 709-772-4883 Dept. Fisheries & Oceans, Gov of Canada fax 709-772-3207 Box 5667 St.John's,NF Canada A1C-5X1 From pjc2@newton.npl.co.uk Mon Oct 14 10:38:42 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!uunet!in3.uu.net!feed1.news.erols.com!howland.erols.net!EU.net!usenet2.news.uk.psi.net!uknet!usenet1.news.uk.psi.net!uknet!uknet!nplpsg!news From: pjc2@newton.npl.co.uk (Peter Cumpson) Newsgroups: sci.data.formats Subject: Re: MS Excel Data format Date: 9 Oct 1996 10:26:37 GMT Organization: National Physical Laboratory Lines: 25 Message-ID: <53fukt$qr8@lightning.cise.npl.co.uk> References: NNTP-Posting-Host: tees.surf.npl.co.uk Mime-Version: 1.0 X-Newsreader: WinVN 0.93.11 In article , vittalk@bit.csc.lsu.edu says... > >Hi, > I do not know if this is the proper newgroup to post my query. >I would like to know the internal data format of Microsoft >Excel 5.0. I am actually developing a software and am trying to >introdue a feature to save my text data in Excel format. >Pls help > > >Au Revoir >-Vittal Vittal, Microsoft publish an Excel 5.0 Software Developers Kit (SDK). This is a book (about 30 pounds in the UK if I remember correctly) with a floppy in the back cover with some useful bits of C. One of the chapters is entirely devoted to the format in which workbooks are saved. It tells you all you'll ever need to know. Email me if you need the ISBN number. Peter Cumpson From mmartini@usgs.gov Mon Oct 14 10:39:44 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!spool.mu.edu!howland.erols.net!www.nntp.primenet.com!nntp.primenet.com!cam-news-hub1.bbnplanet.com!cam-news-feed2.bbnplanet.com!dilbert.whoi.edu!news@whoi.edu From: mmartini@usgs.gov (Marinna Martini) Newsgroups: sci.geo.oceanography,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Thu, 10 Oct 1996 15:08:24 GMT Organization: Woods Hole Oceanographic Institution Lines: 36 Message-ID: <325d1014.10806109@netnews.whoi.edu> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53g8vd$ibi@highway.leidenuniv.nl> <325BCDED.2461@pmel.noaa.gov> <53irot$77m@coranto.ucs.mun.ca> NNTP-Posting-Host: mmartini.er.usgs.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Newsreader: Forte Agent .99f/16.299 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4964 sci.data.formats:1637 ejh@larry.gsfc.nasa.gov (Edward Hartnett) wrote: >Yes, but I will point out that the number of cases where a data set >starts as a personal thing and ends up being shared is not trivial. So >I think you should amend the above to say that if the data are goiing >to be shared, now, or ever, than a universal format is probably >better. > This exact thing has happened to me, and it is only because I was using netCDF that I was able to provide data with any sort of reasonable turn around time to someone who wanted it. I have also found that when using a computer with limited power (laptop in the field) I can still look at parts of large data sets stored in netCDF format while ASCII would be very unweildy if not impossible for my little underpowered machine to deal with. >Actually if you just learn HDF or netCDF and start using it for all >your data, personal or otherwise, the learning curve will quickly be >paid back and you can stick to that one format for all data. I can attest to that. >Sinc the essence of science is colaberation and sharing, and since the >net maakes the sharing of data easy and fun, I don't think it makes >sense to plan on personal data. You should probably figure that almost >all of your data will need to be shared at some point. Never assume you are the only one... sooner or later you will be collaborating! Marinna ------------ "Be seeing you..." - The Prisoner -------------- Marinna Martini | U.S. Geological Survey Ocean Engineer | Marine & Coastal Geology Program mmartini@usgs.gov | Woods Hole, MA 02543 (508) 548-8700 x326 | http://marine.usgs.gov/~mmartini ------------- Earth Science is a Public Trust ---------------- From dwd@pmel.noaa.gov Mon Oct 14 10:41:00 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!cs.utexas.edu!swrinde!howland.erols.net!newsfeed.internetmci.com!news-in2.uu.net!netnews.nwnet.net!news.u.washington.edu!root From: "Donald W. Denbo" Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Thu, 10 Oct 1996 08:46:54 -0700 Organization: NOAA/PMEL/OCRD Lines: 27 Message-ID: <325D1A6E.54D7@pmel.noaa.gov> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53g8vd$ibi@highway.leidenuniv.nl> <325BCDED.2461@pmel.noaa.gov> <53ibcv$ss5@highway.leidenuniv.nl> Reply-To: dwd@pmel.noaa.gov NNTP-Posting-Host: merlin.pmel.noaa.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (WinNT; I) CC: Nancy Soreide Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4965 sci.data.formats:1639 Jeroen Schaap wrote: > > Donald, if you want to tell me, how large is your group? > > Jeroen The EPIC format was originally started about 10 years ago by Nancy Soreide and at that time stood for "Equatorial Pacific Information Collection". The number of users probably was about a dozen. Since then data from all over the world has been brought into the format (we no longer pretend that the initials E.P.I.C. stand for anything!) and the number of user sites is probably about 10. Included several in foreign countries and at several Oceanographic institutions. Dave, We too have the difficulty of finding the data we want to process. EPIC has connections to both miniSQL and Ingress databases. Several command line and X/Motif tools are available for selecting data >from a large number of sites. When I first read your message I suggested EPIC because there already exists a large number of programs in existance that were designed to be robust. We are always looking for new groups with which we can collaborate! -- Donald W. Denbo | Pacific Marine Environmental Laboratory dwd@pmel.noaa.gov | 7600 Sand Point Way NE Fax: (206) 526-6744 | Seattle, Washington 98115 Phone: (206) 526-4487 | From russ@unidata.ucar.edu Mon Oct 14 10:41:22 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!aruba.odu.edu!news.larc.nasa.gov!lerc.nasa.gov!magnus.acs.ohio-state.edu!math.ohio-state.edu!uwm.edu!cs.utexas.edu!howland.erols.net!newsfeed.internetmci.com!csn!nntp-xfer-1.csn.net!ncar!usenet From: Russ Rew Newsgroups: sci.geo.oceanography,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 11 Oct 1996 09:14:50 -0600 Organization: UCAR Unidata Program Lines: 40 Sender: russ@buddy.unidata.ucar.edu Message-ID: References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca><52ue8f$2h1@coranto.ucs.mun.ca> <32544376.4B76@pobox.com> <01bbb631$897cf0a0$b638cb83@gust> NNTP-Posting-Host: buddy.unidata.ucar.edu Cc: m.hadfield@niwa.cri.nz X-Newsreader: Gnus v5.3/Emacs 19.33 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4968 sci.data.formats:1641 "Mark Hadfield" writes: > NetCDF version 3 introduces a new file format (which will eventually > allow a more general data model) and a completely reworked API. A > pre-release of version 3 (C only) is available at the netCDF site > (http://www.unidata.ucar.edu/packages/netcdf/index.html). Well, you're right about the reworked API, but netCDF version 3 will use the same file format used by all previous versions of netCDF, so files written with the new version can be read with previous versions and vice versa. Rewriting the library offered an opportunity to implement a new interface that provided significant benefits to programs that use netCDF: type-safety, automatic type conversions, improved readability, and simpler error behavior. The new interface also removes some obstacles to adding future enhancements, such as packed data and enhanced concurrency. The netCDF-3 library will include support for all netCDF-2 function interfaces and behavior. The benefits of the new interface will be an incentive to use it in future applications, but current applications that use the netCDF-2 interface will continue to work. Programs may be converted to the new interface incrementally, since a mixture of netCDF-2 and netCDF-3 calls is permitted. Version 4 will require a new format in order to support packed external data (e.g. an array of 10-bit values), but the netCDF library will still support reading and writing data in the current format. More details about the alpha (prerelease) version of the C-language-only part of netCDF-3 are available at --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@unidata.ucar.edu http://www.unidata.ucar.edu From russ@unidata.ucar.edu Mon Oct 14 10:42:19 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!aruba.odu.edu!news.larc.nasa.gov!lerc.nasa.gov!magnus.acs.ohio-state.edu!math.ohio-state.edu!uwm.edu!cs.utexas.edu!howland.erols.net!newsfeed.internetmci.com!csn!nntp-xfer-1.csn.net!ncar!usenet From: Russ Rew Newsgroups: sci.geo.oceanography,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 11 Oct 1996 09:14:50 -0600 Organization: UCAR Unidata Program Lines: 40 Sender: russ@buddy.unidata.ucar.edu Message-ID: References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca><52ue8f$2h1@coranto.ucs.mun.ca> <32544376.4B76@pobox.com> <01bbb631$897cf0a0$b638cb83@gust> NNTP-Posting-Host: buddy.unidata.ucar.edu Cc: m.hadfield@niwa.cri.nz X-Newsreader: Gnus v5.3/Emacs 19.33 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4968 sci.data.formats:1641 "Mark Hadfield" writes: > NetCDF version 3 introduces a new file format (which will eventually > allow a more general data model) and a completely reworked API. A > pre-release of version 3 (C only) is available at the netCDF site > (http://www.unidata.ucar.edu/packages/netcdf/index.html). Well, you're right about the reworked API, but netCDF version 3 will use the same file format used by all previous versions of netCDF, so files written with the new version can be read with previous versions and vice versa. Rewriting the library offered an opportunity to implement a new interface that provided significant benefits to programs that use netCDF: type-safety, automatic type conversions, improved readability, and simpler error behavior. The new interface also removes some obstacles to adding future enhancements, such as packed data and enhanced concurrency. The netCDF-3 library will include support for all netCDF-2 function interfaces and behavior. The benefits of the new interface will be an incentive to use it in future applications, but current applications that use the netCDF-2 interface will continue to work. Programs may be converted to the new interface incrementally, since a mixture of netCDF-2 and netCDF-3 calls is permitted. Version 4 will require a new format in order to support packed external data (e.g. an array of 10-bit values), but the netCDF library will still support reading and writing data in the current format. More details about the alpha (prerelease) version of the C-language-only part of netCDF-3 are available at --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@unidata.ucar.edu http://www.unidata.ucar.edu From ROsborn@anl.gov Tue Oct 15 11:23:42 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!hunter.premier.net!news.mathworks.com!howland.erols.net!math.ohio-state.edu!newsfeed.acns.nwu.edu!news.ece.nwu.edu!dis.anl.gov!milo.mcs.anl.gov!usenet From: Ray Osborn Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Wed, 09 Oct 1996 09:24:00 -0500 Organization: Argonne National Laboratory Lines: 46 Message-ID: <325BB57E.4AA@anl.gov> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53dj9t$180@kwuz.nerc-keyworth.ac.uk> Reply-To: ROsborn@anl.gov NNTP-Posting-Host: osborn.msd.anl.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (Macintosh; I; PPC) Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4976 sci.data.formats:1646 Alistair Murray wrote: > > > > I agree that ascii is nice to view, but if you provide a utility to let you > >view the binary file then the effect is the same. > > > > Why not use ASCII in a standard layout and then use a freely available compression > program such as GNU gzip or zip? This gives easy readability of data when > uncompressed and someone else writes and maintains the compression software. > It is also cross-platform portable at least from DOS to UNIX (I just checked!). > Binary formats tend to run into portability problems with the dreaded byte-swapping > fiasco (some operating systems / languages expect least significant byte first - > others vice-versa!!) > > We have large data sets of the kind you have (plus associated biological and > acoustic data) and we find that gzip gives extremely high levels of compression. > Often better than 66%. > I have always had a problem with the idea that ASCII is somehow easier to read. Of course, any operating system allows you to print out ASCII files, but viewing lists of numbers are not usually the best way of visualizing the data, and reading such data in programs can be fraught with difficulty if we do not have access to the precise format used to write the file. Are the columns tab-delimited, (multiple-)white space delimited, free format etc? We also normally want the data to be plotted with suitable axes and labelling, titles and other annotations so we need unambiguous ways of associating column headers or header block information with the remaining data. These are not serious problems if the format is properly documented, but become a major headache with more complex data sets if it is not. It seems to me that portable self-describing binary formats such as HDF or netCDF should make it easier to view the data since there is no ambiguity about what is being stored. HDF are in the process of writing Java code to access HDF files, including some visualization tools, which I hope means that it may soon be possible to browse through these HDF files with our favourite web browsers. Then binary data would become as easy to view as ASCII for most of us. ------------------------------------------------------ Dr Ray Osborn Tel: +1 (630) 252-9011 Materials Science Division Fax: +1 (630) 252-7777 Argonne National Laboratory E-mail: ROsborn@anl.gov Argonne, IL 60439-4845 [Please note change in telephone area code] From dwd@pmel.noaa.gov Tue Oct 15 11:24:40 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!news.sprintlink.net!news-dc-9.sprintlink.net!www.nntp.primenet.com!nntp.primenet.com!howland.erols.net!news.mathworks.com!uunet!in1.uu.net!netnews.nwnet.net!news.u.washington.edu!root From: "Donald W. Denbo" Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: Wed, 09 Oct 1996 09:08:13 -0700 Organization: NOAA/PMEL/OCRD Lines: 55 Message-ID: <325BCDED.2461@pmel.noaa.gov> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53g8vd$ibi@highway.leidenuniv.nl> Reply-To: dwd@pmel.noaa.gov NNTP-Posting-Host: merlin.pmel.noaa.gov Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0 (WinNT; I) CC: Nancy Soreide Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4975 sci.data.formats:1645 Jeroen Schaap wrote: > > > Interchanging data between various programs and platforms during the > last year of a research project, I've got some experience with ASCII. > I've tried some binary formats, but they were all laborious to program > and hard to interchange. Although my dataset is considerably large > (Estimation of 1 Gb uncompressed) with compression of data I am not > accessing frequently I don't use too much space. Moreover, I can > transport and read my formats (about 10 ASCII formats) really easily > between programs, universities and platforms. To me it's very > convenient, but most of all, time-effective. I don't want to spend weeks > on programming, since I have to do all the programming myself. Programs, > macro's and routines must be completed within a few hours. > > Success > > Jeroen I've been following this thread for while now and one aspect of data management that has not been addressed it how do you keep track of what processing has been done to the data? We use netCDF to store our CTD data (over 50,000 stations) and the processing programs automatically include comment lines documenting what has been done. We had a in house developed format (ASCII headers, binary data) and have developed a library (epslib http://www.pmel.noaa.gov/epic/eps-overview.html ) to provide a file independent way of accessing the data. The eps library allows us to continue to use the old data format with the new netCDF format! One big problem of ASCII data ... People can pull it into an editor and change values as well as header information. Those changes are very difficult to find. If you have a lot of data and are doing a lot of interactive processing large ASCII data files are a lot slower to read. If you store them compressed, not only do you have to uncompress them, but you have to convert the ASCII the ascii during a formated read. The final advantage that netCDF provides for us... we have a fairly large user group, and a self-describing data format with the appropriate well designed software, means a user doesn't have to know which of dozens of formats the data is written in. I agree that ASCII is faster to work with than netCDF or our eps library, but your only talking about the initial development time, time required to add new features, time spent debugging, robustness, all of these are improved using a self-describing data format (and for us) a library that allows multiple file formats to be read with the same library. -- Donald W. Denbo | NOAA/PMEL/OCRD dwd@pmel.noaa.gov | 7600 Sand Point Way NE Fax: (206) 526-6744 | Seattle, Washington 98115 Phone: (206) 526-4487 | From m.hadfield@niwa.cri.nz Tue Oct 15 11:25:08 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!howland.erols.net!newspump.sol.net!spool.mu.edu!munnari.OZ.AU!comp.vuw.ac.nz!HERMES!niwa.cri.nz!news From: "Mark Hadfield" Newsgroups: sci.geo.oceanography,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 9 Oct 1996 22:37:01 GMT Organization: NIWA Lines: 21 Message-ID: <01bbb631$897cf0a0$b638cb83@gust> References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca><52ue8f$2h1@coranto.ucs.mun.ca> <32544376.4B76@pobox.com> NNTP-Posting-Host: 131.203.56.182 X-Newsreader: Microsoft Internet News 4.70.1155 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4978 sci.data.formats:1647 andreas schumm wrote in article ... > > I prefer the netCDF data model to the HDF SDS model, but i like the > HDF API. With HDF 3.3 and higher, however, you can have both, if you > don't mind losing the physical netCDF file format. > > Andreas Schumm NetCDF version 3 introduces a new file format (which will eventually allow a more general data model) and a completely reworked API. A pre-release of version 3 (C only) is available at the netCDF site (http://www.unidata.ucar.edu/packages/netcdf/index.html). -- ============================================================== Mark Hadfield NIWA (Taihoro Nukurangi) PO Box 14-901 m.hadfield@niwa.cri.nz Wellington, New Zealand From ejh@larry.gsfc.nasa.gov Wed Oct 16 10:04:51 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!news.msfc.nasa.gov!centauri.hq.nasa.gov!newsfeed.gsfc.nasa.gov!usenet From: ejh@larry.gsfc.nasa.gov (Edward Hartnett) Newsgroups: sci.geo.oceanography,,sci.data.formats Subject: Re: Oceanographics data archival: netCDF,HDF,etc suggestions sought Date: 10 Oct 1996 11:34:28 -0400 Organization: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA Lines: 113 Message-ID: References: <52ovvv$f7n@coranto.ucs.mun.ca> <3252A213.3047@ios.bc.ca> <52ue8f$2h1@coranto.ucs.mun.ca> <53g8vd$ibi@highway.leidenuniv.nl> <325BCDED.2461@pmel.noaa.gov> <53irot$77m@coranto.ucs.mun.ca> NNTP-Posting-Host: larry.gsfc.nasa.gov In-reply-to: Dave Senciall's message of 10 Oct 1996 12:55:57 GMT X-Newsreader: Gnus v5.1 Xref: solitaire.cv.nrao.edu sci.geo.oceanography:4982 sci.data.formats:1649 >>>>> "D" == Dave Senciall writes: D> I agree, actually issues of logging of processing activities,quality control D> etc. are factors that I must consider, Often if an internation study program,or D> nation archive reqests you provide data to them they want to know what you have D> done to it, the processing and manipualtion history is often as important as D> the actual data . D> Another issue is that of 'indexing' or cataloguing the data for searching, D> It's fine to stick 500,000 profiles into a archive but if you can't readily D> search and rapidly extract data you can't accomplish much. D> When I first posted my question I had no idea of that number of responces that D> I would recieve ,nor the variety of opions aired. Hahahahaha! Welcome to the Format Wars! As a Format Warrior for the last ten years I know all the arguments and aat one time or another have supported all different sides. Years ago ASCII was my format of choice for small data files, and I thought HDF was a giant pain in the butt. Now I've swung 180 degrees on the issue. D> I think the fundamental thing that distingushes the respondents is the 'scope' D> of the projects that they work on. If your data set is small and/or personnel D> then 'internation standard' quality data formats are likely more of an D> inconvience than an asset. However if your group is large, your datasets large D> and varied, if you have to exchange data with other institutes and you have to D> manage data that is not part of 'your person project' but is more of an D> institutional asset then formats like netCDF and HDF are seriously worth D> considering. Yes, but I will point out that the number of cases where a data set starts as a personal thing and ends up being shared is not trivial. So I think you should amend the above to say that if the data are goiing to be shared, now, or ever, than a universal format is probably better. Actually if you just learn HDF or netCDF and start using it for all your data, personal or otherwise, the learning curve will quickly be paid back and you can stick to that one format for all data. The amount of effort scientists spend on data formats is tremendous! I think we would all like to see this problem go away so that you-all could spend that time doing science. HDF and netCDF are not the perfect answer, but are as close as we've come so far. Sinc the essence of science is colaberation and sharing, and since the net maakes the sharing of data easy and fun, I don't think it makes sense to plan on personal data. You should probably figure that almost all of your data will need to be shared at some point. For example the group I work with aat NASA has some data that, 5 years ago, they would have sworn would never be of interest to anyone. Of course I wouldn't be raising this example if the data were not of tremendous interest to several other groups at the moment. For the next iteration of their processing this group is doing almost everything in HDF. D> My currant archives are ascii based using unix compress for storage efficieny D> for low usage data. Space and performance are not excessively pressing issues, D> though in 5 years when the data sets have doubled in size they will be. My Probably not really. 5 years ago the largest disk size you could get was what, 1.2 GB? Now 9 GB disks are common. This trend will only continue. Data doubling every 5 yeas is not worth worrying about. In 5 years you'll just buy a disk that's ten times the size of your current one and you'll be good to go. D> concern centered more on access, as it is now a researcher looking at ,say, D> water temperature in a region can access ctd data. If he wants to get D> temperature data from a current meter sensor in the same region he has to D> access a different data set that is in a totally unrelated format for meta data D> and sensor data that is controlled by total different software, similarly if D> they D> wanted to get the thermister data from the ADCP head in a ship, the software D> and D> data formats are totally un-related. My goal is to, at the least, get a common D> software interface to the data. ie point your program at desired dataset and D> read out temperature data, with little effort having to be put into 'knowing' D> the format of that data. Beyond that it would be nice that the 'style' of each HDF and netCDF come with a program called ncdump (or something of that kind) that can give you an ascii dump of any variable (or all of them) >from the data file. This gives you the best of both worlds. Visualisation packages like grads can now read HDF files and netCDF files. This trend will continue. HDF and netCDF are here to stay and more and more tools will deal with them well. D> data set have a common 'theme' so that when a new instrument comes along you D> have a 'formula' for designing the data storage rather that the add-hock method D> that people tend to fall into where the archieve ends up begining a version of D> the acquistion software format, or someones pet format that they dreamed up D> over a beer. This of course is the big savings when dealing with HDF or netCDF. Smart format people have already thought of all the different types of data and provided for them in the formats, so you don't have to invent a way to store your data, just read the manuals and find the way already provided. I urge you to let those smart format people work for you! Good luck, and let us know what your decision is. -- Edward Hartnett ejh@larry.gsfc.nasa.gov * I don't speak for NASA or ARC, I just work for them! * Goddard Distributed Active Archive Center (DAAC) Check out cool Earth Science data at: http://daac.gsfc.nasa.gov/ From georgev@news.ncsa.uiuc.edu Fri Oct 18 09:57:25 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!spool.mu.edu!munnari.OZ.AU!news.ecn.uoknor.edu!feed1.news.erols.com!howland.erols.net!vixen.cso.uiuc.edu!not-for-mail From: georgev@news.ncsa.uiuc.edu (George Velamparampil) Newsgroups: sci.data.formats,sci.image.processing Subject: Re: Looking for tiled raster image format i/o libraries Date: 17 Oct 1996 14:43:25 GMT Organization: National Center for Supercomputing Applications Lines: 47 Message-ID: <545gmd$m2q@vixen.cso.uiuc.edu> References: <543742$2odk@elmo.cadvision.com> <543ppu$6uu@vixen.cso.uiuc.edu> NNTP-Posting-Host: dogbert.ncsa.uiuc.edu X-Newsreader: TIN [UNIX 1.3 950824BETA PL0] Xref: solitaire.cv.nrao.edu sci.data.formats:1664 sci.image.processing:23886 George Velamparampil (georgev@news.ncsa.uiuc.edu) wrote: : Tom Spoering (spoering@cadvision.com) wrote: : : I'm looking for library code for reading and writing tiled raster image data of : : multiple types (8, 16, 32 bit signed and unsigned int, 32 and 64 bit float) for : : very large (100s of MB) files. Hopefully something in C or C++. We need : : tiling for efficiency and also multiple images per file to store a pre-zoomed : : tier of resolution factors of the same image. : : : : I've come across FITS and CCITT, but neither seem to fill all requirements. : : I've heard of TRIF, but haven't found a web page for it yet. : : : : In HDF4.1 there will be support for chunking(2D-tiling) with compression : using the Scientific Dataset(SDS) API. I don't know if the chunking will : make into the General Raster(GR) API by HDF4.1 release but I'm hoping it : will. Currently the HDF4.1 release is scheduled for sometime in January or : Febuary. A Beta release should be available in December. : : I currently have a pre-beta release that has support for chunking without : compression. If you are interested I can give you a copy. I have : not settled on the final SDS API routines and would like to get some : early feedback. : : -georgev : : -- : ----------------------------------------------------------------------------- : George Velamparampil -- HDF Developer(mostly) -- email:georgev@ncsa.uiuc.edu : Software Development Group -- NCSA -- University of Illinois, Urbana I forgot to mention that another format the allows tiling and supports the data types above is TIFF. The unofficial home page for TITF is http://www-mipl.jpl.nasa.gov/~ndr/tiff/ The home page for HDF is http://hdf.ncsa.uiuc.edu/ -georgev -- ----------------------------------------------------------------------------- George Velamparampil -- HDF Developer(mostly) -- email:georgev@ncsa.uiuc.edu Software Development Group -- NCSA -- University of Illinois, Urbana From georgev@news.ncsa.uiuc.edu Fri Oct 18 16:22:57 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!uunet!news-in2.uu.net!www.nntp.primenet.com!nntp.primenet.com!howland.erols.net!vixen.cso.uiuc.edu!not-for-mail From: georgev@news.ncsa.uiuc.edu (George Velamparampil) Newsgroups: sci.data.formats,sci.image.processing Subject: Re: Looking for tiled raster image format i/o libraries Followup-To: sci.data.formats,sci.image.processing Date: 16 Oct 1996 23:06:38 GMT Organization: National Center for Supercomputing Applications Lines: 28 Message-ID: <543ppu$6uu@vixen.cso.uiuc.edu> References: <543742$2odk@elmo.cadvision.com> NNTP-Posting-Host: dogbert.ncsa.uiuc.edu X-Newsreader: TIN [UNIX 1.3 950824BETA PL0] Xref: solitaire.cv.nrao.edu sci.data.formats:1667 sci.image.processing:23897 Tom Spoering (spoering@cadvision.com) wrote: : I'm looking for library code for reading and writing tiled raster image data of : multiple types (8, 16, 32 bit signed and unsigned int, 32 and 64 bit float) for : very large (100s of MB) files. Hopefully something in C or C++. We need : tiling for efficiency and also multiple images per file to store a pre-zoomed : tier of resolution factors of the same image. : : I've come across FITS and CCITT, but neither seem to fill all requirements. : I've heard of TRIF, but haven't found a web page for it yet. : In HDF4.1 there will be support for chunking(2D-tiling) with compression using the Scientific Dataset(SDS) API. I don't know if the chunking will make into the General Raster(GR) API by HDF4.1 release but I'm hoping it will. Currently the HDF4.1 release is scheduled for sometime in January or Febuary. A Beta release should be available in December. I currently have a pre-beta release that has support for chunking without compression. If you are interested I can give you a copy. I have not settled on the final SDS API routines and would like to get some early feedback. -georgev -- ----------------------------------------------------------------------------- George Velamparampil -- HDF Developer(mostly) -- email:georgev@ncsa.uiuc.edu Software Development Group -- NCSA -- University of Illinois, Urbana From philk@sgi.com Mon Oct 21 10:00:14 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!news-server.ncren.net!taco.cc.ncsu.edu!gatech!csulb.edu!news.sgi.com!mr.net!feeder.chicago.cic.net!wolverine.hq.cic.net!news.worldpath.net!iag.net!enews.sgi.com!fido.asd.sgi.com!news From: Phil Keslin Newsgroups: sci.data.formats,sci.image.processing Subject: Re: Looking for tiled raster image format i/o libraries Date: Wed, 16 Oct 1996 16:18:34 -0700 Organization: Silicon Graphics Computer Systems Lines: 27 Message-ID: <32656D4A.41C6@sgi.com> References: <543742$2odk@elmo.cadvision.com> NNTP-Posting-Host: rampage.asd.sgi.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0C-SGI (X11; I; IRIX 6.2 IP22) To: Tom Spoering Xref: solitaire.cv.nrao.edu sci.data.formats:1669 sci.image.processing:23904 Tom Spoering wrote: > > I'm looking for library code for reading and writing tiled raster image data of > multiple types (8, 16, 32 bit signed and unsigned int, 32 and 64 bit float) for > very large (100s of MB) files. Hopefully something in C or C++. We need > tiling for efficiency and also multiple images per file to store a pre-zoomed > tier of resolution factors of the same image. > > I've come across FITS and CCITT, but neither seem to fill all requirements. > I've heard of TRIF, but haven't found a web page for it yet. Have you checked out TIFF? It supports all of the features that you require. The source for the library is readily available from the following URL. ftp://ftp.sgi.com/graphics/tiff There is a README and HOWTO file that contain lots of information. Hope this helps... - Phil -- Phil Keslin Silicon Graphics, Inc. E-Mail: philk@asd.sgi.com 2011 N. Shoreline Blvd. M/S 8U-590 Phone: (415)933-1959 Mountain View, CA 94043 From ggoodrum@perigee.ncdc.noaa.gov Wed Oct 23 10:34:17 1996 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!newsfeed.internetmci.com!in1.uu.net!rdc.noaa.gov!perigee.ncdc.noaa.gov!ggoodrum From: Geof Goodrum Newsgroups: sci.data.formats Subject: Re: ps research Date: Thu, 17 Oct 1996 09:20:37 -0400 Organization: National Climatic Data Center Lines: 22 Message-ID: References: <3264F21C.4970@essex.ac.uk> NNTP-Posting-Host: perigee.ncdc.noaa.gov Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII To: Daniele Malleo In-Reply-To: <3264F21C.4970@essex.ac.uk> On Wed, 16 Oct 1996, Daniele Malleo wrote: > I'm doing some kind of research about postscript.. > Which company developed it? Are there any x-Windows Postscript viewers? > How is aPostscript file printed? Can PS files contain images? What books > are written about PS? What are PS Fonts? is a PS readable (by humans)? > any help will be appreciated. > Please post it to : dmalle@essex.ac.uk Visit the Ghostscript Home Page at http://www.cs.wisc.edu/~ghost/index.html. There is also a link to the PostScript FAQ, which has answers to most of your questions. DISCLAIMER: The comments above are my own and may not represent the views of my employer. +-------------------------------+-------------------------------------------+ : Geoffrey P. Goodrum : US Department of Commerce : : +1-301-457-5100 : NOAA/NESDIS National Climatic Data Center : : ggoodrum@perigee.ncdc.noaa.gov: Satellite Services Branch : +-------------------------------+-------------------------------------------+ Geof's Primordial Soup