From coats@robin.mcnc.org Tue Jun 3 23:05:54 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!nntprelay.mathworks.com!news.mathworks.com!cam-news-hub1.bbnplanet.com!cpk-news-hub1.bbnplanet.com!news.bbnplanet.com!gatech!ultra150.ncren.net!news-server.ncren.net!not-for-mail From: coats@robin.mcnc.org (Carlie Coats) Newsgroups: sci.data.formats Subject: Re: Difficulties with netCDF and Sparse Data Date: 3 Jun 1997 09:21:04 -0400 Organization: MCNC Lines: 70 Message-ID: <5n15o0$dl2@robin.mcnc.org> References: <3380d683.340402963@news-2.sni.net> NNTP-Posting-Host: robin.mcnc.org In article <3380d683.340402963@news-2.sni.net>, Jeremy Beal wrote: > >Forgive me for bringing up a difficulty which has been encountered >before, but I'm interested in seeing if anybody has made any headway. > >We have an interest in storing a variety of numerical data as output >from a software simulation of a physical system. The data which must >be stored are physical quantities which depend on spatial coordinates >and time. The output data must be read by another program, possibly >running on a different platform, so we would like the data file to >be platform independent. There is a large quantity of output data so >we practically need a binary file. We are interested in netCDF as >a means to achieve an easily read platform independent binary data >file. We would also enjoy being able to use the tools which have >been developed for examining netCDF files. > >Unfortunately, our data is not necessarily regularly patterned, and >it seems that it may not fit well within a standard netCDF file. Our >natural inclination would be to use the time value as the unlimited >dimension within the file and then define the coordinates of our >spatial grid points using three additional dimensions. > >Problems: > Our data is sparse in both a spatial and time sense; > i.e. not every physical quantity is written out at each > time step, nor at every spatial grid point within a > given timestep. > > Our grids themselves can vary as a function of the > timestep, i.e. a finer grid might be created inside > of a cell for a single time step for needed accuracy. > The finer grid would only exist for one or two timesteps > and would then no longer be used for the rest of the > simulation.... We have a somewhat similar problem in communicating data for Kain-Fritsch-McHenry cloud events between the MM5 meteorology model and the MAQSIP air quality model. In that case, convective cloud events arise at random times in random horizontal cells of the met model; the characteristics of the vertical profile which sets off the event, as well as the starting date, starting time, and duration of the event must be communicated from MM5 to MAQSIP. We do this using structure built on top of our "usual" netCDF wrapper. The files have three components: a file header that matches our usual file header structure (dimensions, attributes, etc.); a (fixed-maximum-size) events index which permits selective direct access into: an events list dimensioned in the UNLIMITED dimension, which stores vertical profiles of all variables associated with an individual cloud event, in addition to other characteristics of the event. All this is documented at URL http://www.iceis.mcnc.org/EDSS/ioapi/H.DATATYPES.html#kfevnt or feel free to call me. Carlie J. Coats, Jr. coats@mcnc.org *or* xcc@hpcc.epa.gov MCNC Environmental Programs phone: (919)248-9241 North Carolina Supercomputing Center fax: (919)248-9245 3021 Cornwallis Road P. O. Box 12889 Research Triangle Park, N. C. 27709-2889 USA "My opinions are my own, and I've got *lots* of them!" From jbeal@nospam.nvmedia.com Sat Jun 7 22:28:50 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!zombie.ncsc.mil!alnews.ncsc.mil!uunet!uunet!199.60.229.3!newsfeed.direct.ca!su-news-hub1.bbnplanet.com!news.bbnplanet.com!csn!nntp-xfer-1.csn.net!csn!nntp-xfer-2.csn.net!news-2.csn.net!not-for-mail From: jbeal@nospam.nvmedia.com (Jeremy Beal) Newsgroups: sci.data.formats Subject: Re: Floating point binary portability Date: Fri, 06 Jun 1997 16:06:43 GMT Organization: SuperNet Inc. +1.303.296.8202 Denver Colorado Lines: 19 Message-ID: <3398349c.73940130@news-2.sni.net> References: <33857426.41C6@rumba.cnb.uam.es> <5mccrg$qe$2@newsserver.rrzn.uni-hannover.de> <5mk0jl$qip$1@newsserver.rrzn.uni-hannover.de> <33972ebf.6903837@news-2.sni.net> NNTP-Posting-Host: 198.233.40.11 X-Newsreader: Forte Free Agent 1.1/32.230 On Thu, 05 Jun 1997 21:27:33 GMT, jbeal@nospam.nvmedia.com (Jeremy Beal) wrote: > >Does anybody know the location of the XDR source? I've been searching >unsuccessfully on the net, plenty of mention of the RFC, but I can't >locate the source. > >Jeremy Beal >jbeal at nvmedia.com XDR is actually a part of the Sun RPC mechanism which seems to be implemented on nearly all Unix systems. A port of Sun RPC for Windows NT/95 can be found at http://set.gmd.de/~mfg/download.htm There is a description page somewhere there too but I can't get the URL off the top of my head. Some documentation is included in the distribution. From zender@ncar.ucar.edu Sat Jun 7 22:29:05 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!news.ysu.edu!news.radio.cz!news.vol.cz!hammer.uoregon.edu!news.oru.edu!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!newsfeed.internetmci.com!ncar!not-for-mail From: Charlie Zender Newsgroups: sci.data.formats Subject: nco-1.0 is released: netCDF Operators Date: Fri, 06 Jun 1997 17:17:04 -0600 Organization: National Center for Atmospheric Research/Boulder, CO Lines: 42 Message-ID: <33989A6F.2D25@ncar.ucar.edu> NNTP-Posting-Host: 128.117.56.8 Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Mozilla 3.01 (X11; I; SunOS 5.5.1 sun4m) I am pleased to announce the release of version 1.0 of NCO, which stands for `netCDF operators'. NCO consists of nine command line operators for netCDF files: ncks ncrename ncra ncea ncrcat ncdiff ncwa ncecat ncflint These operators work on generic netCDF files and perform rote tasks like renaming, concatentation, hyperslabbing, averaging, interpolation, and differencing. The documentation is nearly complete. The source code is freely available via anonymous ftp: ftp.cgd.ucar.edu:pub/zender/nc/nco-1.0.tar.gz The documentation is now on-line: http://www.cgd.ucar.edu/cms/nco/nco.html Good luck, Charlie >From the NEWS file: ^L * Changes in NCO version 1.0: ^L ** All known bugs in version 0.9 have been fixed. ** The manual is nearly complete, and describes all operators. ** All operators should now work on machines with sizeof(long) > sizeof(int). ** Implemented ARM time=base_time+time_offset convention in ncrcat. ** Improved support for NCAR CSM conventions ** ncks allows specification of hyperslab stride ** ncks works with wrapped coordinates (e.g., longitude) ^L -- Charlie Zender Voice, FAX: (303) 497-1612, 497-1324 NCAR ASP & CGD E-mail: zender@ncar.ucar.edu P.O. Box 3000 URL: http://www.cgd.ucar.edu/cms/zender Boulder CO 80307-3000 PGP: finger -l zender@heinlein.cgd.ucar.edu From jbeal@nospam.nvmedia.com Tue Jun 10 22:26:33 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!zombie.ncsc.mil!fozzie.mercury.net!news-dc-2.sprintlink.net!news-east.sprintlink.net!news-dc-26.sprintlink.net!news-peer.sprintlink.net!news.sprintlink.net!Sprint!newsfeed.internetmci.com!southwind.net!symbios.com!csn!nntp-xfer-2.csn.net!news-2.csn.net!not-for-mail From: jbeal@nospam.nvmedia.com (Jeremy Beal) Newsgroups: sci.data.formats Subject: Re: Floating point binary portability Date: Tue, 10 Jun 1997 22:22:39 GMT Organization: SuperNet Inc. +1.303.296.8202 Denver Colorado Lines: 43 Message-ID: <339dd01f.3040391@news-2.sni.net> References: <33857426.41C6@rumba.cnb.uam.es> <5mccrg$qe$2@newsserver.rrzn.uni-hannover.de> <5mk0jl$qip$1@newsserver.rrzn.uni-hannover.de> <33972ebf.6903837@news-2.sni.net> <3398349c.73940130@news-2.sni.net> NNTP-Posting-Host: 198.233.40.11 X-Newsreader: Forte Free Agent 1.1/32.230 On Fri, 06 Jun 1997 16:06:43 GMT, jbeal@nospam.nvmedia.com (Jeremy Beal) wrote: > >XDR is actually a part of the Sun RPC mechanism which seems to be >implemented on nearly all Unix systems. A port of Sun RPC for Windows >NT/95 can be found at http://set.gmd.de/~mfg/download.htm >There is a description page somewhere there too but I can't >get the URL off the top of my head. Some documentation is included >in the distribution. > More followup: A port of the Sun RPC (ONC RPC) for Windows NT and 95 exists at: http://set.gmd.de/~mfg/SRPC112.ZIP Unfortunately, I'm having quite a bit of trouble trying to get a small test program to execute under NT 3.51. I can get their example program to run fine, but it isn't XDR oriented. When I try to use the XDR routines directly, the program is able to create the XDR stream, but generates an exception as the program tries to filter the first value (an integer). The callback stack indicates that the routine call goes through two levels in the oncrpc static library, then the Microsoft Visual C Run Time library, then KERNEL32, then dies at the second level of NTDLL. I tried recompiling with only the XDR source parts of the library included. It crashes in the same place. Debugging the code indicates that everything runs fine until it needs to interpret a #define which seems to map a pseudo-function call to a function which is part of the XDR struct. It's starting to seem like it would be easier to write my own damn routines to implement the XDR RFC than to keep wrestling with undocumented code which doesn't seem to run correctly. Has anybody ever seen any implementation of XDR on a PC which actually works? How the hell does netCDF work if there isn't even an implementation of XDR... I guess I'll start poring through the netCDF source now. Jeremy Beal jbeal at nvmedia.com From jbeal@nospam.nvmedia.com Wed Jun 11 10:28:13 1997 Path: solitaire.cv.nrao.edu!newsgate.duke.edu!zombie.ncsc.mil!fozzie.mercury.net!news-dc-2.sprintlink.net!news-east.sprintlink.net!news-dc-26.sprintlink.net!news-peer.sprintlink.net!news.sprintlink.net!Sprint!newsfeed.internetmci.com!southwind.net!symbios.com!csn!nntp-xfer-2.csn.net!news-2.csn.net!not-for-mail From: jbeal@nospam.nvmedia.com (Jeremy Beal) Newsgroups: sci.data.formats Subject: Re: Floating point binary portability Date: Tue, 10 Jun 1997 22:22:39 GMT Organization: SuperNet Inc. +1.303.296.8202 Denver Colorado Lines: 43 Message-ID: <339dd01f.3040391@news-2.sni.net> References: <33857426.41C6@rumba.cnb.uam.es> <5mccrg$qe$2@newsserver.rrzn.uni-hannover.de> <5mk0jl$qip$1@newsserver.rrzn.uni-hannover.de> <33972ebf.6903837@news-2.sni.net> <3398349c.73940130@news-2.sni.net> NNTP-Posting-Host: 198.233.40.11 X-Newsreader: Forte Free Agent 1.1/32.230 On Fri, 06 Jun 1997 16:06:43 GMT, jbeal@nospam.nvmedia.com (Jeremy Beal) wrote: > >XDR is actually a part of the Sun RPC mechanism which seems to be >implemented on nearly all Unix systems. A port of Sun RPC for Windows >NT/95 can be found at http://set.gmd.de/~mfg/download.htm >There is a description page somewhere there too but I can't >get the URL off the top of my head. Some documentation is included >in the distribution. > More followup: A port of the Sun RPC (ONC RPC) for Windows NT and 95 exists at: http://set.gmd.de/~mfg/SRPC112.ZIP Unfortunately, I'm having quite a bit of trouble trying to get a small test program to execute under NT 3.51. I can get their example program to run fine, but it isn't XDR oriented. When I try to use the XDR routines directly, the program is able to create the XDR stream, but generates an exception as the program tries to filter the first value (an integer). The callback stack indicates that the routine call goes through two levels in the oncrpc static library, then the Microsoft Visual C Run Time library, then KERNEL32, then dies at the second level of NTDLL. I tried recompiling with only the XDR source parts of the library included. It crashes in the same place. Debugging the code indicates that everything runs fine until it needs to interpret a #define which seems to map a pseudo-function call to a function which is part of the XDR struct. It's starting to seem like it would be easier to write my own damn routines to implement the XDR RFC than to keep wrestling with undocumented code which doesn't seem to run correctly. Has anybody ever seen any implementation of XDR on a PC which actually works? How the hell does netCDF work if there isn't even an implementation of XDR... I guess I'll start poring through the netCDF source now. Jeremy Beal jbeal at nvmedia.com