From sandro@news.unige.it Mon Feb 6 13:28:59 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!newsxfer.itd.umich.edu!gatech!howland.reston.ans.net!math.ohio-state.edu!jussieu.fr!univ-lyon1.fr!ghost.dsi.unimi.it!news.unige.it!not-for-mail From: sandro@news.unige.it (Sandro Magri') Newsgroups: sci.data.formats Subject: Re: What is JCAMP format? Date: 6 Feb 1995 17:25:13 +0100 Organization: Univ. of Genoa, Italy Lines: 39 Message-ID: <3h5ih9$fh9@alpha.cisi.unige.it> References: <3h057a$2sl@lucy.infi.net> NNTP-Posting-Host: alpha.cisi.unige.it X-Newsreader: TIN [version 1.2 PL2] William Dawes (wmdawes@infi.net) wrote: : I'm investigating making a CD-ROM with perhaps several million : color spectra. Two other scientists tell me that they believe the : best format is JCAMP. Looking for details in the FAQ here, I see : that it isn't mentioned. It's a draft standard for spectra data (IR & NMR) and chemical stuff, References: 1) JCAMP-DX for NMR, A. N. Davies, P. Lampen, Applied Spectroscopy, 1993, 47, 1093-1099; 2) A proposed European Implementation of the JCAMP-DX Format, D. N. Rutledge, P. Mcintyre, Chemometrics and Intelligent Laboratory Systems, 1992, 16, 95-101; 3) JCAMP-DX, A standard format for exchange of infrared-spectra in computer readable form, J. G. Grasselli, Pure and Applied Chemistry 1991, 63, 1781-1792; 4) JCAMP-CS A standard exchange format for chemical-structure information, J.Gasteiger, B. M. P. Hendriks, P. Hoever, C. Jochum, H. Somberg, Applied spectroscopy, 1991, 45, 4-11 A viewer for Windows: Rober Lancaster's JCAMP viewer, for spectra. http://wwwchem.uwimona.edu.jm:1104/software/jcampdx.html The chairman of the suitable IUPAC committee for JCAMP 5: srheller@probe.nalusda.gov and : I'm wondering if it's a subset of some other type that is mentioned : here. Has anyone here heard of JCAMP? : Bill : wmdawes@infi.net Chester, VA 804-748-8639 Fax: 804-768-8170 -- Sandro Magri' E-Mail : sandro@ibf.unige.it Biophysical Inst.,Univ. of Genoa magri@cisi.unige.it Via Giotto 2,16153 Genova, Italy Tel. +39-10-6516052 Fax +39-10-6513106 From dilg@fishery.hitc.com Mon Feb 6 13:29:16 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!lll-winken.llnl.gov!ames!newsfeed.gsfc.nasa.gov!newsroom.gsfc.nasa.gov!fishery.hitc.com!dilg From: dilg@fishery.hitc.com () Newsgroups: sci.data.formats Subject: Re: portable binary format? [CUJ reference needed] Date: 6 Feb 1995 17:28:09 GMT Organization: Applied Reseach Corporation Lines: 17 Distribution: world Message-ID: <3h5m79$l71@newsroom.gsfc.nasa.gov> References: <3h1g8e$g4h@news.doit.wisc.edu> Reply-To: dbuto@eos.hitc.com NNTP-Posting-Host: fishery.hitc.com In article <3h1g8e$g4h@news.doit.wisc.edu>, khan@xraylith.wisc.edu (Mumit Khan) writes: |> I've been looking into writing a portable I/O interface (callable from |> C/C++ and FORTRAN) that's based on IEEE floating pt format. XDR looks |> promising, but I don't know how readily I can get XDR routines for |> non-UNIX OS's (and something we can redistribute for free). There was a |> posting that mentioned some C User's Journal article dedicated to the |> topic ... anyone have a ref? Are there established pckgs that already |> solved this? Yes, there are several established packages out there that have solved your problem to various degrees, in different ways. I'd suggest taking a look at the FAQ for sci.data.formats: ftp://rtfm.mit.edu/pub/usenet/news.answers/sci-data-formats I'd pay particular attention to CDF, netCDF, and HDF (my personal favorite). They seem to cover the widest range of data on the widest range of platforms. -Doug Ilg From gohel@csee.usf.edu Mon Feb 6 17:37:52 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!zip.eecs.umich.edu!panix!news.mathworks.com!news.alpha.net!uwm.edu!vixen.cso.uiuc.edu!howland.reston.ans.net!swrinde!gatech!darwin.sura.net!mother.usf.edu!news From: gohel@csee.usf.edu (Himanshu Gohel (DMIP)) Newsgroups: sci.image.processing,sci.data.formats Subject: Re: sun visualization file format Followup-To: sci.image.processing,sci.data.formats Date: 6 Feb 1995 18:52:11 GMT Organization: Geometric Modeling & Computer Graphics Group, USF, Tampa, FL. Lines: 36 Distribution: world Message-ID: <3h5r4r$khv@mother.usf.edu> References: <3gnocs$ptu@mogan.cc.metu.edu.tr> Reply-To: gohel@csee.usf.edu NNTP-Posting-Host: bruiser.rad.usf.edu Summary: Re: sun visualization file format Keywords: Re: sun visualization file format Xref: solitaire.cv.nrao.edu sci.image.processing:12140 sci.data.formats:791 In article <3gnocs$ptu@mogan.cc.metu.edu.tr>, serhat@rorqual.cc.metu.edu.tr (serhat akin) writes: > hi I would like to convert sun visualization file format (vff) to tiff > or targa. I recently converted them to sun raster format and then to tiff > or targa. Ok it works, but the problem is while converting sun vff to > sun ras it reduces 24 bit info to 8 bits. Any software or info will be > greatly appreciated. Hmmm...SunVision's VFF image format doesn't have a 24-bit support, so I'm not sure whether you're dealing with 16-bit or 32-bit images here. Anyway, basically SunVision's VFF image file format is very simple - it consists of an ASCII header, which is very self explanatory, and which is followed by a ctrl-L (^L) to indicate end of header, and then binary data (could be ASCII too, but I don't think many people use that format - the files would be too big). So if you had a 16-bit file, all you have to do is to cut everything above the ^L and then you have a flat file which contains your image. In case of 32-bit images, the data is stored as such (according to the man page): ABGRABGR... (Alpha, Blue, Green, Red...) Sun's Raster Image format will be 8-bit, so you will certainly lose info if you use that as an intermediate step. Hope that helps you get started. -- Himanshu Gohel, gohel@rad.usf.edu WEB URL: http://www.csee.usf.edu/~gohel Digital Medical Imaging Program (DMIP) University of South Florida, Tampa, FL. USA From sandro@news.unige.it Tue Feb 7 10:25:40 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!europa.eng.gtefsd.com!news.mathworks.com!udel!gatech!howland.reston.ans.net!math.ohio-state.edu!jussieu.fr!univ-lyon1.fr!ghost.dsi.unimi.it!news.unige.it!not-for-mail From: sandro@news.unige.it (Sandro Magri') Newsgroups: sci.data.formats Subject: Re: What is JCAMP format? Date: 7 Feb 1995 10:26:49 +0100 Organization: Univ. of Genoa, Italy Lines: 63 Message-ID: <3h7ecp$5eu@alpha.cisi.unige.it> References: <3h057a$2sl@lucy.infi.net> <3h5ih9$fh9@alpha.cisi.unige.it> NNTP-Posting-Host: alpha.cisi.unige.it X-Newsreader: TIN [version 1.2 PL2] Sandro Magri' (sandro@news.unige.it) wrote: : William Dawes (wmdawes@infi.net) wrote: : : I'm investigating making a CD-ROM with perhaps several million : : color spectra. Two other scientists tell me that they believe the : : best format is JCAMP. Looking for details in the FAQ here, I see : : that it isn't mentioned. : It's a draft standard for spectra data (IR & NMR) and chemical stuff, : References: : 1) JCAMP-DX for NMR, A. N. Davies, P. Lampen, Applied Spectroscopy, : 1993, 47, 1093-1099; : 2) A proposed European Implementation of the JCAMP-DX : Format, D. N. Rutledge, P. Mcintyre, Chemometrics and Intelligent : Laboratory Systems, 1992, 16, 95-101; : 3) JCAMP-DX, A standard format for : exchange of infrared-spectra in computer readable form, J. G. Grasselli, : Pure and Applied Chemistry 1991, 63, 1781-1792; : 4) JCAMP-CS A standard exchange format for chemical-structure information, : J.Gasteiger, B. M. P. Hendriks, P. Hoever, C. Jochum, H. Somberg, : Applied spectroscopy, 1991, 45, 4-11 : A viewer for Windows: : Rober Lancaster's JCAMP viewer, for spectra. : http://wwwchem.uwimona.edu.jm:1104/software/jcampdx.html I want add some other notes: Today I have received this messages from CheMIME mailing-list >> From: jimw@COM.PE-Nelson (Jim Watt) >> Subject: Spectral file format standardization >> There has been a standardization effort under way for several years in >> the chromatography, infrared, and mass spectrometry communities. These >> share a common technology - Unidata's netCDF. I served on the committee >> for the mass spectrometry standardization effort. As of now all of the >> major chromatography and mass spectrometry system vendors offer data >> interchange products. >> >> The specification for the chromatography standard is available from >> the Analytical Instruments Association in Washington, DC. I belive >> the cost of the standard is about $120US. The mass spectrometry standard >> is available at our site via anonymous FTP to "ftp.pe-nelson.com", or >> 192.52.153.11 if the name causes any problems. The relevant file >> is in /pub/andi-MS, and is named ms_doc.zip. >> >> >>From: dr.bill@genie.geis.com >>To: Multiple recipients of list >>Subject: JCAMP-DX format >>.... >>Also, see the DEC 1994 issue of Applied Spectroscopy for JCAMPDX for MS data >>and GC-MS data. The editorial has a discussion of netCDF format, approved >>by AIA, as contrasted to JCAMPDX. >>..... >> -- Sandro Magri' E-Mail : sandro@ibf.unige.it Biophysical Inst.,Univ. of Genoa magri@cisi.unige.it Via Giotto 2,16153 Genova, Italy Tel. +39-10-6516052 Fax +39-10-6513106 From khan@xraylith.wisc.edu Thu Feb 9 14:07:29 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!vixen.cso.uiuc.edu!uwm.edu!news.doit.wisc.edu!khan From: khan@xraylith.wisc.edu (Mumit Khan) Newsgroups: sci.data.formats Subject: SUMMARY and followup questions: portable binary format Date: 9 Feb 1995 16:35:18 GMT Organization: Center For X-ray Lithography, UW-Madison Lines: 44 Message-ID: <3hdg86$5ko@news.doit.wisc.edu> References: <3h1g8e$g4h@news.doit.wisc.edu> NNTP-Posting-Host: brahma.xraylith.wisc.edu I had posted the following query asking for info/suggestion on a portable binary format that uses IEEE standard floating point format. In article <3h1g8e$g4h@news.doit.wisc.edu>, I wrote: I've been looking into writing a portable I/O interface (callable from C/C++ and FORTRAN) that's based on IEEE floating pt format. XDR looks promising, but I don't know how readily I can get XDR routines for non-UNIX OS's (and something we can redistribute for free). There was a posting that mentioned some C User's Journal article dedicated to the topic ... anyone have a ref? Are there established pckgs that already solved this? thanks mumit -- khan@xraylith.wisc.edu http://www.xraylith.wisc.edu/~khan/ All the responses I've received so far have pointed out the following packages: NCSA HDF, UCAR netCDF, and UCAR CDF These packages have been designed specifically for this type of task. Carlie Coats, Jr suggests writing a domain-specific wrapper/interface on top of either UCAR netCDF or NCS HDF, which would be very much what my application needs. I've built both HDF and netCDF since my posting and found them very powerful and that prompts the next 2 questions: - What're the (dis)advantages of netCDF over HDF? Why would you pick one over the other? - What kind of runtime overhead can I expect to incur (over straight binary reads and writes using fread/fwrite, for example)? I'd like to thank the following people for their help: Carlie Coats, Jr Steve Emmerson Doug Ilg From acheng@ncsa.uiuc.edu Thu Feb 9 16:28:51 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!vixen.cso.uiuc.edu!acheng From: acheng@ncsa.uiuc.edu (Albert Cheng) Newsgroups: sci.data.formats Subject: Re: SUMMARY and followup questions: portable binary format Date: 9 Feb 1995 20:39:51 GMT Organization: Nat'l Ctr for Supercomp App (NCSA) @ University of Illinois Lines: 23 Distribution: world Message-ID: <3hduin$kb7@vixen.cso.uiuc.edu> References: <3h1g8e$g4h@news.doit.wisc.edu> <3hdg86$5ko@news.doit.wisc.edu> NNTP-Posting-Host: xongmao.ncsa.uiuc.edu Originator: acheng@xongmao.ncsa.uiuc.edu In article <3hdg86$5ko@news.doit.wisc.edu>, khan@xraylith.wisc.edu (Mumit Khan) writes: |> I've built both HDF and netCDF since my posting and found them very |> powerful and that prompts the next 2 questions: |> |> - What're the (dis)advantages of netCDF over HDF? Why would you pick |> one over the other? They each has their strength and drawbacks. But HDF also supports a netCDF interface--it can read and write netCDF files (but not to create a brand new netCDF file.) So, you have the best of both worlds. :-) |> |> - What kind of runtime overhead can I expect to incur (over straight |> binary reads and writes using fread/fwrite, for example)? We are doing a study on the overhead cost. There *maybe* a report released later but no promise since I am not the one doing it. ==== Albert Cheng Research Programmer HDF Team From ejh@larry.gsfc.nasa.gov Thu Feb 9 17:27:05 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!ncar!ames!newsfeed.gsfc.nasa.gov!gsfc.nasa.gov!ejh From: ejh@larry.gsfc.nasa.gov (Edward Hartnett) Newsgroups: sci.data.formats Subject: Re: SUMMARY and followup questions: portable binary format Date: 09 Feb 1995 20:40:32 GMT Organization: NASA Goddard Space Flight Center -- Greenbelt, Maryland USA Lines: 24 Message-ID: References: <3h1g8e$g4h@news.doit.wisc.edu> <3hdg86$5ko@news.doit.wisc.edu> NNTP-Posting-Host: larry.gsfc.nasa.gov In-reply-to: khan@xraylith.wisc.edu's message of 9 Feb 1995 16:35:18 GMT >>>>> "M" == Mumit Khan writes: M> I've built both HDF and netCDF since my posting and found them very M> powerful and that prompts the next 2 questions: M> - What're the (dis)advantages of netCDF over HDF? Why would you pick M> one over the other? HDF has been chosen by NASA's EOSDIS project, which is *BIG* project. I expect that within the next few years it will increase in popularity. Since it's latest release HDF includes all the features of netCDF anyway. M> - What kind of runtime overhead can I expect to incur (over straight M> binary reads and writes using fread/fwrite, for example)? Pretty negligible. Biggest inventment is in programmer time. I've been slamming my head against HDF for about a month now. Using any of these formats is more complicated then working with binary data files. -- From wmdawes@infi.net Sun Feb 12 21:06:37 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!vixen.cso.uiuc.edu!howland.reston.ans.net!news.sprintlink.net!news.infi.net!usenet From: wmdawes@infi.net (William Dawes) Newsgroups: sci.data.formats Subject: Neat Windows Spectra viewer for JCAMP files Date: 12 Feb 1995 20:14:41 GMT Organization: SpectraServe Lines: 29 Message-ID: <3hlq7h$6lm@lucy.infi.net> NNTP-Posting-Host: h-olive.richmond.infi.net X-Newsreader: WinVN 0.92.6+ Thanks to Sandro' Magri, U. of Genoa -- I've been looking for information on spectral data formats, for a project to put a few million color spectra on a CD-ROM. A format called JCAMP was recommended. Further surfing found a scaleable Windows viewer for spectra that allows scaling, data clipping, and other tricks. Sample data files of IR, UV, and MS are included with the software in the home page. Dr. Lancashire, the author, said: "I make available a viewer for WINDOWS, so pull down a copy and pull down a data file and see how you go. The worst looking data files are those encoded by our PE FT-IR. The NMR files given are much simpler to convert." Dr. Robert Lancashire rjlanc@uwimona.edu.JM Department of Chemistry University of the West Indies, JAMAICA The Chem Dept, U of West Indes, Jamaica home page is: http://wwwchem.uwimona.edu.jm:1104/index.html The JCAMP-DX viewer program is available from: http://wwwchem.uwimona.edu.jm:1104/software/jcampdx.html A dozen or so IR, MS, NMR spectra (MS, IR, NMR) in JCAMP format will be found at: http://wwwchem.uwimona.edu.jm:1104/spectra/index.html Try it -- I predict you'll like it! Bill From dilg@fishery.hitc.com Fri Feb 24 16:16:47 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!portal.gmu.edu!europa.eng.gtefsd.com!news.mathworks.com!news.alpha.net!uwm.edu!lll-winken.llnl.gov!ames!newsfeed.gsfc.nasa.gov!newsroom.hitc.com!fishery.hitc.com!dilg From: dilg@fishery.hitc.com () Newsgroups: sci.data.formats Subject: Re: Packing floating point data using HDF Date: 24 Feb 1995 15:19:59 GMT Organization: Hughes STX Corporation Lines: 44 Distribution: world Message-ID: <3iktev$3ef@newsroom.hitc.com> References: <3i3g8a$mbl@ncar.ucar.edu> Reply-To: dilg@ulabsgi.gsfc.nasa.gov NNTP-Posting-Host: fishery.hitc.com Keywords: HDF,packing In article <3i3g8a$mbl@ncar.ucar.edu>, rosinski@isis.cgd.ucar.edu (Jim Rosinski) writes: |> |> I would like to pack my floating point SDS data generated by HDF on a UNICOS |> operating system. Obviously you can get effective 2 to 1 packing by |> specifying DFNT_FLOAT32 when the data get written if you're willing to live |> with the roughly 7 decimal digits of precision that this provides (Crays use |> a 64-bit floating point representation). Be careful with this! If you have a float64 array which you supply it to the HDF library and claim that it's a float32 array, HDF will *NOT* convert your data from 64 to 32 bits. It will simply take each consecutive 32 bits and interpret them as a float32. So, if you have a 100x100 array of float64 (that's 100 x 100 x 8 = 80,000 bytes) and tell HDF that it's a 100x100 array of float32 (100 x 100 x 4 = 40,000 bytes), HDF will store the first 10,000 32-bit "words" into your SDS. The result is that half your data has just disappeared. The way to safely accomplish your 2:1 compression is to use a loop to assign and coerce each element of your float64 array to an element of a float32 array. |> Does HDF provide the capability to |> pack/unpack the data, with (obviously) some loss of precision which provides |> even more than this 2 to 1 factor? Standard compression utilities which |> provide no loss of precision (ex: gzip) are useless on the type of data which |> I'm generating--there aren't many repeated values. The current version of HDF does not provide any compression, as such, for SDSs. Version 4.0 (currently in alpha release) will provide some compression techniques for SDSs. What you *can* do with the current version is to "de-calibrate" your data >from a 64-bit float to a 16-bit integer and store an int16 SDS with standard "calibration" information (through a routine called SDsetcal). The calibration information includes a scale, offset, and resultant data type. Using the formula given in the HDF Reference Manual, you can then convert the int16 data set into the resultant float64, using the stored scale and offset. Note that HDF does not do the scaling operation for you. It just provides a standard way of storing the parameters for the scaling operation. Of course, this strategy will only make sense if your data can stand some loss of precision *and* dynamic range. -Doug Ilg Hughes STX From dilg@fishery.hitc.com Fri Feb 24 16:17:53 1995 Path: solitaire.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!uwm.edu!lll-winken.llnl.gov!ames!newsfeed.gsfc.nasa.gov!newsroom.hitc.com!fishery.hitc.com!dilg From: dilg@fishery.hitc.com () Newsgroups: sci.data.formats Subject: Re: Who uses HDF? Date: 24 Feb 1995 14:06:10 GMT Organization: Hughes STX Corporation Lines: 53 Distribution: world Message-ID: <3ikp4i$pov@newsroom.hitc.com> References: Reply-To: dilg@ulabsgi.gsfc.nasa.gov NNTP-Posting-Host: fishery.hitc.com In article , cdodge@edvs11.awi-bremerhaven.de (Chris Dodge) writes: |> |> In the netCDF FAQ there is a list of who uses netCDF, but we havn't |> been able to find a similar list for HDF. Does one exist? |> |> We are trying to make a decision as to what data format to use for |> Ocean flow model output; current contenders are netCDF, HDF or |> GRIB. Anyone working in this line got any thought on what would be |> best? From our research so far, it looks like netCDF is probably the |> best supported and most widely used format...... Chris/Arne/Heinz, I'm sure that NCSA can give you a more current list, but here are some of the folks who are known to use HDF: Spyglass, Intergraph, Research Systems, Inc (IDL), Silicon Graphics, Inc (Explorer), PCI. All of these companies have commercial applications that use HDF. NASA's Jet Propulsion Laboratory, Phillips Petroleum, Massachusetts General Hospital. Each of these organizations has adopted HDF to fulfill some or all of its internal data storage needs. In addition, over the next several years, the biggest single user of HDF will be the Earth Observing System. EOS will be creating and archiving several petabytes of remote sensing data, which will be made available to the Earth Science community using HDF as the standard data format. As part of this effort, NASA will be encouraging more commercial software vendors to incorporate HDF capabilities into their data analysis and visualization tools. |> Also: What about performance of reading/writing speed of netCDF |> vs. HDF. Back when netCDF was first being investigated by the HDF team at NCSA (I was on the team at the time), HDF was found to be significantly faster than netCDF. The difference was due in large part to the reliance of the netCDF software on the XDR library, which was used to provide cross-platform portability via IEEE standard data representations. HDF software includes its own custom data conversion routines, which produce the same IEEE standard data representations, but are much faster (and are even specially optimized on certain platforms like Cray). Of course, this work was done several years ago. I'm sure that netCDF has improved in comparison to HDF since then. I seem to recall that they have even replaced the XDR library with their own custom code. Hope this helps you with your decision. -Doug Ilg Hughes STX EOS Data and Information System