Subject: Re: Standards for exchange of scientific data via WWW?

From: [email-masked] (Marc Andreessen)
Date: 2 Apr 94 06:06:37

> BTW, you are correct that Mosaic "understands" HDF files in that
   > it can read the files and produce HTML docs showing the contents
   > in a browse-like fashion.

Yup.  This includes representing hierarchical datasets through virtual
(generated on the fly) hypermedia trees, displaying images and
palettes contained within datasets, displaying full information about
dataset structure and contents in human-readable form, and providing a
facility to transfer pieces of the dataset to external programs like
AVS via a data IPC mechanism called "Data Transfer Mechanism", or DTM,
which is another product of NCSA.  More information should be in the
Mosaic docs.  (This applies to the X version of Mosaic only, so far as
I know.)

This is really neat stuff if you're into scientific data -- even if it
doesn't suit your exact needs, I think it vividly demonstrates how
things can be done well in a general, flexible environment like
Mosaic.  (I can say that because I didn't write Mosaic's HDF support,
Chris Houck of NCSA did.)

   > Actually, I'm pretty sure that the same is true netCDF files,
   > which are interpreted as if they were HDF SDSs.

Yup -- completely transparently.

   I can think of applications (meaning uses, not programs) where this
   wouldn't be desirable behavior.  Of course one could turn off this
   feature, I assume, and just save the data to a file.

Yup -- just activate "Load To Local Disk".

Cheers,
Marc

--
Marc Andreessen
Enterprise Integration Technologies
Palo Alto, California
[email-masked], [email-masked]

---

From: [email-masked] (Ilana Stern)
Date: 1 Apr 1994 00:36:27 GMT

In article <[email-masked]>, Jeff E Mandel <[email-masked]> writes:
 
> I have the following questions:
> 
> 1) Has anyone else discussed the possibility of distributing datasets via
> WWW?

There are two issues here:  the format of the dataset, and the method
of distribution.  We (NCAR) distribute some datasets by ftp, and we
have a quasi-WWW site (it's hypertext served by ftp, until we get a
real http server), so in a sense you could say we distribute datasets
via WWW.  But the data format is not MIMEish or anything -- it's mostly
just ascii (or occasionally packed binary) data.

See ftp://ncardata.ucar.edu/catalogs/.html/README.html for examples.

> 2) Are there any standards for dataset formats? I need something more
> efficient than tab-delimited text.

See the sci.data.formats FAQ (I've crossposted to that group) at
http://www.ncsa.uiuc.edu/SDG/Software/HDF/SciDataFormatsFAQ.html
I think HDF is understood by Mosaic.  Basically, packed binary data is
the way to go for most scientific datasets, because it's 1) efficient
and 2) not machine-dependent.

> 3) Is there a way to build a WAIS index of this, so that only the text is
> indexed, and not the data? Consider the file containing the 200 word 
> abstract of the paper "Creatine
> blood levels in 500 patients with chronic renal failure sampled at 10
> minute intervals for 3 years" followed by the gazillion values

Michael Schwartz at the University of Colorado has done some work
creating a Wais indexing scheme called Essence which does just this.  

-- 
/\     "This technique probably only makes the difference | dod#0009
  \_][     between serious injury and total disaster."    | [email-masked]
      \__<a href=ftp://ncardata.ucar.edu/catalogs/.html/me.html>Ilana Stern</a>