From jeff@lodur.srl.caltech.edu Wed May 7 12:15:33 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!europa.clark.net!newsfeed2!news.ridgecrest.ca.us!nntp-server.caltech.edu!lodur.srl.caltech.edu!jeff From: jeff@lodur.srl.caltech.edu (Jeff Hammond) Newsgroups: sci.data.formats Subject: HDF shortcuts Date: 2 May 1997 17:56:28 GMT Organization: California Institute of Technology, Pasadena Lines: 20 Message-ID: <5kd9sc$nhj@gap.cco.caltech.edu> NNTP-Posting-Host: lodur.srl.caltech.edu X-Newsreader: TIN [version 1.2 PL0] We have developed some software for our internal use that makes using HDF much easier and faster. It also reduces the learning curve significantly normally required to read and write HDF files. We decided that our basic unit of data would be C structures, and have routines that will read and write C structures by making the appropriate calls to HDF. Our software (written in perl) creates these C structure read/write routines using a file containing the C structure as input. We have also incorporated this into our Makefile so that when the structures change (as they have done many times in development) the read/write routines are automatically updated. A copy of our latest version with an example is available at http://www.srl.caltech.edu/ACE/ASC/HDF_ex.html. Cheers, Jeff From matthew.rice@ftlsol.com Wed May 7 12:15:38 1997 Message-ID: <336C0EF9.1B5302FB@ftlsol.com> Date: Sun, 04 May 1997 00:22:18 -0400 From: Matthew Rice Organization: FTL Solutions Inc. X-Mailer: Mozilla 3.01 (X11; I; Linux 2.0.18 i586) MIME-Version: 1.0 Newsgroups: sci.data.formats To: Bartek Rajwa Subject: Re: Float to HDF References: <336628B6.41C6@aragorn.mol.uj.edu.pl> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit NNTP-Posting-Host: shiva4-n4-ip38-toronto.ica.net Lines: 14 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!news-peer.sprintlink.net!Sprint!news-pull.sprintlink.net!news.sprintlink.net!news.rns.net!lightning.ica.net!shiva4-n4-ip38-toronto.ica.net Bartek Rajwa wrote: > I have a data set stored as 32-bit floating point data array. There is > no information about dimensions in the set. This info has to supplied by > the user when the file is read by our software. Now the problem: I would > like to convert it to HDF. Where can I find a public domain package > which could ask me for dimensions and then transform the array into > HDF-like? You could start with fp2hdf. It's included in the HDF distribution: ftp://ftp.ncsa.uiuc.edu/HDF/HDF_Current -- Matthew Rice e-mail: matthew.rice@ftlsol.com From TECHman@matrix.kapatel.gr Sat May 10 22:58:43 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!rill.news.pipex.net!pipex!oleane!pasteur.fr!jussieu.fr!news.forth.gr!news-ath.forthnet.gr!news.kapatel.gr!news From: "Byron S. Veras" Newsgroups: sci.data.formats Subject: Re: MS-Word, Excel file format Date: Thu, 08 May 1997 16:57:06 +0200 Organization: TECHmanŽ Co. - Electronic Applications / Automations Lines: 26 Message-ID: <3371E9C1.785E@matrix.kapatel.gr> References: <5kl6hi$3n1@gazette.omnilink.net> Reply-To: TECHman@matrix.kapatel.gr NNTP-Posting-Host: www.kapatel.gr Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-7 Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.0Gold (Win16; I) To: Timo.Engelke@frankfurt.netsurf.de Hi! Timo Engelke wrote: > > Hi ! > > I hardly need the specification of the MS_Word 6.0 and 7.0 for Windows > file format and Excel in the newest versions ... > Where can I get such files ? > or can anyone send them to me ? Take a look at http://www.microsoft.com/msdn/ or under SDK area for docs. > > thanks > Timo -- ""8"" 8"""" 8""""8 8 8 8 8 8 " 8 8 eeeeeee eeeee eeeee 8e 8eeee 8e 8eee8 8 8 8 8 8 8 8 88 88 88 88 8 8e 8 8 8eee8 8e 8 88 88 88 e 88 8 88 8 8 88 8 88 8 88 88eee 88eee8 88 8 88 8 8 88 8 88 8 2:410/104.7@FidoNet & 30:3001/136.7@GrUNet From jbeal@nvmedia.com Wed May 21 09:56:24 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!news-peer.sprintlink.net!news.sprintlink.net!Sprint!howland.erols.net!news2.digex.net!news5.digex.net!haven.umd.edu!purdue!yuma!csn!nntp-xfer-2.csn.net!news-2.csn.net!not-for-mail From: jbeal@nvmedia.com (Jeremy Beal) Newsgroups: sci.data.formats Subject: Difficulties with netCDF and Sparse Data Date: Mon, 19 May 1997 22:41:57 GMT Organization: SuperNet Inc. +1.303.296.8202 Denver Colorado Lines: 101 Message-ID: <3380d683.340402963@news-2.sni.net> NNTP-Posting-Host: 198.233.40.11 X-Newsreader: Forte Free Agent 1.1/32.230 Forgive me for bringing up a difficulty which has been encountered before, but I'm interested in seeing if anybody has made any headway. We have an interest in storing a variety of numerical data as output >from a software simulation of a physical system. The data which must be stored are physical quantities which depend on spatial coordinates and time. The output data must be read by another program, possibly running on a different platform, so we would like the data file to be platform independent. There is a large quantity of output data so we practically need a binary file. We are interested in netCDF as a means to achieve an easily read platform independent binary data file. We would also enjoy being able to use the tools which have been developed for examining netCDF files. Unfortunately, our data is not necessarily regularly patterned, and it seems that it may not fit well within a standard netCDF file. Our natural inclination would be to use the time value as the unlimited dimension within the file and then define the coordinates of our spatial grid points using three additional dimensions. Problems: Our data is sparse in both a spatial and time sense; i.e. not every physical quantity is written out at each time step, nor at every spatial grid point within a given timestep. Our grids themselves can vary as a function of the timestep, i.e. a finer grid might be created inside of a cell for a single time step for needed accuracy. The finer grid would only exist for one or two timesteps and would then no longer be used for the rest of the simulation. As I understand the structure of the netCDF file, the only way that we could have a file contain all of the quantities would be to set up the dimensions to enumerate every possible grid location and timestep which is ever used within the simulation and store within these dimensions. Nulls will be written for any values which are not explicitly placed in the file. This would waste a tremendous amount of space in the file due to the sparseness of our data, so much so as to make it unusable. I've seen from the archives that people have used some tricks to get around the time sparseness issue, including sub-record schemes (good if the data is regularly patterned in time) and using separate netCDF files for quantities which are written at different frequencies. These won't work easily for our problem because we can't generally predict in advance when the quantities may need to be written, and because they don't address the issue of spatial data sparseness. Our current code writes a platform dependent binary file which must be run through a conversion prior to being loaded by our second program. The binary file is written efficiently using our own data format, which takes advantage of subheaders at the beginning of each time record indicating exactly what has been stored within the time record. We see three possibilities for writing a platform independent binary data file with a reasonable size: 1. Filter the output from our existing routines through the XDR library in order to write a platform independent binary file. The output side should be easy, just one different step in writing to the file. It would require some amount of coding on the input stage to the second program, as we'd need to have it dissect the proprietary binary (but platform independent) file. Here we keep the efficient file size but lose the benefits of netCDF like external utilities, simple function calls to retrieve values, etc. 2. Do something clever using the existing netCDF routines. This would be something like the sub-record scheme or multiple file workaround, but would need to address all of our sparseness problems. I haven't thought of anything too great yet...? 3. Modify the netCDF library to allow for sub-headers at each record explicitly showing what is stored within that record. I'm not sure how difficult this would be yet. We would still lose the benefits of compatibility with netCDF with respect to utilities, etc. In addition we would need to maintain the code with respect to updates in netCDF if we wanted to take advantage of benefits of the updates. However, it would provide the benefits of nice standard functions to retrieve arbitrary pieces of data. I'm sure that there would be a performance hit on random access reads/writes because you would no longer have a nice fixed record size. I don't know how much of a hit it would be. Is anybody else facing a data storage problem with sparse and general data? Any suggestions? Thanks sincerely, Jeremy Beal jbeal at nvmedia dot com From johnecarter@mindspring.com Sun May 25 18:13:31 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!cam-news-hub1.bbnplanet.com!cpk-news-hub1.bbnplanet.com!news.bbnplanet.com!mindspring!usenet From: johnecarter@mindspring.com (John Carter) Newsgroups: sci.data.formats Subject: Re: Book of data file formats Date: Sat, 24 May 1997 04:48:39 GMT Organization: MindSpring Enterprises Lines: 20 Message-ID: <33867205.13042803@news.mindspring.com> References: <337949A8.761A@jupiter.otago.ac.nz> Reply-To: johnecarter@mindspring.com NNTP-Posting-Host: user-37kbm78.dialup.mindspring.com X-Server-Date: 24 May 1997 04:48:00 GMT X-Newsreader: Forte Free Agent 1.1/32.230 There were two volumes published on older software "File Formats for Popular PC Software" and "More File Formats for Popular PC Software" that covered the DOS versions of the mots popular programs. Most of the vendors offer some information on *most* (but not all) of their products (Microsoft Project is proprietary and they won't share under a non-disclosure agreement - makes retrieval of a damaged fileimpossible). For graphics, there is a fairly recent volume on graphics file formats with a CD of samples and sample code for reading/displaying the graphics files - the book is at work and I don't remember the title. There are also some excellent Web sites with graphics references. Again, all my refernces are at work. John C Richard wrote: >Hi, does anyone know of a reasonably complete book that provides the >file formats of various applications (such as Microsoft Excel and Lotus >123 with various versions) and standards (such as GIF and JPEG)? From thompson@orpheus.nascom.nasa.gov Tue May 27 16:33:10 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.mathworks.com!europa.clark.net!news.msfc.nasa.gov!centauri.hq.nasa.gov!newsfeed.gsfc.nasa.gov!orpheus.nascom.nasa.gov!thompson From: thompson@orpheus.nascom.nasa.gov (William Thompson) Newsgroups: sci.data.formats Subject: Re: Floating point binary portability Date: 27 May 1997 14:01:17 GMT Organization: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA Lines: 19 Message-ID: <5mepfd$68b@post.gsfc.nasa.gov> References: <33857426.41C6@rumba.cnb.uam.es> NNTP-Posting-Host: orpheus.nascom.nasa.gov X-Newsreader: NN version 6.5.0 #4 (NOV) Monica Chagoyen Quiles writes: >Could someone tell me which is the state of portability between the >different hardware platforms in order to interchange floating point >binary files? >Does everybody use the IEEE current standards? >Thank you very much for your help, The only modern platform that I'm aware of that doesn't use IEEE floating point notation is OpenVMS from Digital. However, for us, that's still an important platform to support--many scientific institutions still use it. Even on other platforms, there are really two kinds of IEEE numbers, depending on whether one has little-endian or big-endian numbers. It's best to assume that some kind of translation needs to be applied. Bill Thompson From breiter@mathematik.Uni-Osnabrueck.DE Wed May 28 13:02:03 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!agate!spool.mu.edu!uwm.edu!chi-news.cic.net!su-news-hub1.bbnplanet.com!cpk-news-hub1.bbnplanet.com!news.bbnplanet.com!newsfeed.internetmci.com!news-was.dfn.de!news-fra1.dfn.de!news-koe1.dfn.de!news-han1.dfn.de!newsserver.rrzn.uni-hannover.de!not-for-mail From: breiter@mathematik.Uni-Osnabrueck.DE (Bernhard Reiter) Newsgroups: sci.data.formats Subject: Re: HDF, CDF or NetCDF. Witch one? Date: 28 May 1997 13:41:38 GMT Organization: RRZN - Newsserver Lines: 45 Message-ID: <5mhcmi$8l3$1@newsserver.rrzn.uni-hannover.de> References: <338C47BF.1932281F@termo1.mmt.upc.es> NNTP-Posting-Host: ranec.mathematik.uni-osnabrueck.de Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Newsreader: knews 0.9.8 In article <338C47BF.1932281F@termo1.mmt.upc.es>, Ramiro Alba Queipo writes: > At this point, I have been looking for a format on the free > comunity, used but as much people as possible and suitable for our > simulations on Heat Transfer and Fluid Dinamics. Witch one is > the most suitable: HDF, CDF or NetCDF?. Any other? I do not know. Maybe you should read the FAQs and decide yourself. There are several systems dealing with large data. I heard of ROOT, which was announced as "OO framework for large scale scientific data analysis and data mining. It has been developed at CERN with the sponsorship of HP and is currently being used by a number of large high energy physics experiments." References: ROOT: Message-ID: HDF: netCDF: CDF: > Wath about making movies?: gif, cgm, jpeg... I am a novell > in this field. Look for "mpeg" quicktime or avi are other formats. "xanim" and "mpeg_encode" are examples for free software which plays and can encode pictures to "movies". If you want professional results you need other programms which can create real videos. I rember that there were some nearly free Scientific Visualisation Applications around..... May start your search here: You might want to have allok into "VRML" if you want to make 3D model, which can be explored interactively instead of a movie. Send me a mail if this info has helped you. ;-) Bernhard Reiter From russ@unidata.ucar.edu Thu May 29 10:10:56 1997 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!uunet!in3.uu.net!207.172.3.52!feed1.news.erols.com!cpk-news-hub1.bbnplanet.com!news.bbnplanet.com!newsfeed.internetmci.com!ncar!not-for-mail From: Russ Rew Newsgroups: sci.chem.analytical,sci.data.formats,sci.chem.labware Subject: Re: AIA datafile format, specifications? Followup-To: sci.data.formats Date: 28 May 1997 13:25:46 -0600 Organization: UCAR Unidata Program Lines: 20 Message-ID: References: <01bc69fe$fc0cacb0$844bed82@ask> NNTP-Posting-Host: 128.117.140.29 X-Newsreader: Gnus v5.3/Emacs 19.33 Xref: solitaire.cv.nrao.edu sci.chem.analytical:8744 sci.data.formats:1929 sci.chem.labware:3370 "TMD" writes: > I am intrested in more information about the AIA (Analytical Instruments > Association) datafile format. This datafile format is used for > chromatography > data interchange (GC, HPLC..). > > Were can I find the description of the AIA datafile format? Information about the AIA Andi standards (Andi stands for "Analytical data interchange") and information about how to order Andi specifications from AIA can be obtained from http://www.ultranet.com/~lasf/andi.html --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program russ@unidata.ucar.edu http://www.unidata.ucar.edu