Subject: Re: Data Exchange in HTML

From: [email-masked] (Thomas Gee)
Date: 23 Jan 1996 17:13:40 -0500

In article <[email-masked]>,
Ed Carty <[email-masked]> wrote:
>
>Does anyone have any experience (or ideas) about implementing data exchange
>in HTML?  While this could apply to any spreadsheet-like data, I'm currently
>more interested in scientific collection data submission and publication.

Well, you seem to have some good ones.  Why don't we start with your
suggestions and move out from there?  Here at DCIEM, like in many (all?)
research institutes, the bookkeeping task of keeping track of your data
is rapidly becoming more of a challenge than the research itself.  So
any method that might address this issue is welcome.

I've just started keeping onlines copies of our papers and asset databases
via the Web, and it's proving very useful.  So I like the thought of
extending this ability.

>I'm assuming that the data can be organized in tabular fashion (2-dimensions),
>mostly because it will usually end up in database tables anyhow, but I suppose
>it could be extended to 3-or-more dimensions and maybe a self-describing
>approach like netCDF or HDF, or even include referential cell formulas like
>spreadsheets... (yikes!).

This is where life becomes awkward.  Mosaic claims to have builtin support
for HDF files, but either I've got the wrong version or something is
misconfigured, because I can't seem to get it to work.  I've found very
little documentation from the NCSA that even mentions HDF support, much
less describes how it works.  However, if we can get it working well, this
would be a good way to provide access to large or complex data structures.

I like the spreadsheet idea for more refined and less bulky data, though.
I think the 2D table should be quite adequate for 70-80% of our data
representation.

>Say I wanted to exchange database table data using HTML syntax (and say I
>wanted to keep it simple so other people wouldn't mind supporting it) would
>it be a good idea to;
>
>   1) Use standard HTML (v3);
...
>   
>   2) Implement some proprietary tags and render data in a simple
>      spreadsheet-ready format (like comma-separated-values).
>             
...
>   3) Other designs?

Well, I think we can have both.  I like the idea of using standard HTML
so that people can browse through the data quickly and see if the
information they need is available, but you're correct that the syntax
is more complex than most spreadsheet programs are able to handel.

I think the best answer is to store the data in the lowest common denominator,
with single character delimiters, and convert this to HTML for browsing.
I've written a very short, Perl-based CGI program to test this
idea.  It allows you to define simply a title for the table and the filename
containing the data, and it creates the HTML data right on the spot.

If you want to check it out, it's at
"http://epochms.dciem.dnd.ca/cgi-bin/sheet-cgi".

>This is pretty preliminary stuff, but I think the concept is valid...
>All ideas or comments are welcome!
>
>Thanks,
>Ed Carty

Good thinking, Ed.  Well, anyone have additional input?  Where do we go
>from here?

Thoughtfully yours,
	Tom
--
[email-masked]
Aircrew Performance/Protective Systems Section
Aerospace Life Support Technology Sector

---

From: [email-masked] (Jeff Becklehimer)
Date: 25 Jan 1996 20:56:31 GMT

Thomas Gee ([email-masked]) wrote:
: Ed Carty <[email-masked]> wrote:
[snip]
: >I'm assuming that the data can be organized in tabular fashion (2-dimensions),
: >mostly because it will usually end up in database tables anyhow, but I suppose
: >it could be extended to 3-or-more dimensions and maybe a self-describing
: >approach like netCDF or HDF, or even include referential cell formulas like
: >spreadsheets... (yikes!).

: This is where life becomes awkward.  Mosaic claims to have builtin support
: for HDF files, but either I've got the wrong version or something is
: misconfigured, because I can't seem to get it to work.  I've found very
: little documentation from the NCSA that even mentions HDF support, much
: less describes how it works.  However, if we can get it working well, this
: would be a good way to provide access to large or complex data structures.

I know that UNIX version of Mosaic 2.6 does support HDF. I'm not real sure
if HDF support ever made it to the non-UNIX versions. It is a really neat
feature. If you load an HDF dataset into MOSAIC it will print out the
headers and any images that may be in the file. You can check this out
for yourself by going to http://www7430.nrlssc.navy.mil/cgi-bin/etopo5_grid.html
If you do an area extract and request an HDF file it will be loaded into
your MOSAIC browser. Since I put html code into the comments section it
will even render the html with hotlinks back to the form where the data
came from.

What's even neater is that we have built an HDF viewer using X. In the
widget that request a file name to open is an option that says "Etopo5".
When you select this button it will ask you for the latitudes and 
longitudes you wish to see. Then using a CCI it will do the extraction
>from the web server and plot the data. Very Cool!

--
<A HREF="http://www7430.nrlssc.navy.mil/~beck/"><em>Jeff Becklehimer</em></A>

---

From: [email-masked] (Thomas Gee)
Date: 26 Jan 1996 14:40:54 -0500

In article <[email-masked]>,
Jeff Becklehimer <[email-masked]> wrote:
>your MOSAIC browser.

Ah, very nice.  I did do this, and received all the high-level description
of the data file, but no image was produced (using xmosaic 2.7b2).

Does anyone know if Mosaic supports the Vset capability of HDF?  Using
HDF3.3r3, Vset has been subsumed into HDF (as opposed to being an add-on
package like it used to be), but you still end up using an entirely
different API for manipulating them, so perhaps that part has not been
implemented into Mosaic.

My test HDF file contains quite a number of Vset constructs, and when I 
load one into Mosaic, it comes up with simply a blank screen.  I know the
file format is HDF-compatible correct, because it is created entirely with
HDF/Vset commands, and some programs (like hdfls) recognize its structure.

Curiouser and curiouser...

Tom