Subject: Re: Self-describing data files

From: [email-masked] (Joel Plutchak)
Date: 9 Jun 1993 19:28:43 GMT

In article <[email-masked]> [email-masked] (Joe Panico) writes:
>
>Hi,
>
>   Can anyone provide any information about self-describing file formats
>for scientific data? The general idea: the contents and format of the
>data are quite flexible, much more so than netCDF or AVS Fields. A library
>of functions exist for querying the data file about the types and structures 
>of data stored (not the file format itself), and other functions can then
>extract data from the file and present it to the querying program, according
>to an agreed upon protocol. Any clues/leads/hints much appreciated. Thanks.

   CD-ROM disks issued by NASA's Planetary Data System (PDS) supposedly
are self-documenting datasets.  The library of functions to access the
disks are being worked on, but last time I checked, were incomplete and
somewhat buggy (e.g. didn't work for quite a few of the disks I use).
--
Joel Plutchak, Research Programmer/Analyst
"Dies ist meine 12305te Nacht/Die erste paar tausend,
 die kannst du vergessen/ich hab' es auch getan" - EN

---

From: [email-masked] (Walter Douglas Gross)
Date: 10 Jun 1993 18:42 EDT

>for scientific data? 

                Information on CCSDS and SFDUs

CCSDS stands for Consultative Committee for Space Data Systems.  The CCSDS was 
established in 1982, as a result of an international workshop, sponsored by 
NASA, to address various issues regarding space data systems.  The CCSDS is a 
volunteer organization, composed of representatives of 18 space agencies.  

Included among the purposes of the CCSDS are to:

1) Provide a forum for technical discussions and interchange, regarding common 
   data data systems interests,

2) Promote interoperability among space exploration missions, and

3) Facilitate scientific data interchange.

The CCSDS has, working within it, a number of technical panels.  One of the 
panels is:

Panel 2 - Standard Data Interchange Structures

Panel 2 of the CCSDS is the panel whose area includes the topic of Standard 
Formatted Data Units (SFDUs).  The center of activity for NASA's interaction 
with Panel 2 is the National Space Sciences Data Center (NSSDC) at the Goddard 
Space Flight Center.  The Goddard Space Flight Center is a NASA facility 
located in Greenbelt, Maryland. 

SFDUs are a standard that is being developed for the labeling of archived data
(space exploration data has been the initial fodder for SFDUs.)  More 
specifically, this standard is a system of labels, themselves only 20 
characters long, which serve as marking-points to allow data to be organized 
into logical structures.

Some of these structures will contain the actual data, and, please, by all 
let this be heard:

---------> SFDUs impose no restrictions on what sort of actual 
---------> data structures or formats can be used.

Others of these structures contain what is often called "metadata" - which is 
"data about the data".  This is intended to be, in the aggregate, text 
containing a broad range of descriptions of the data: including, and not 
limited to (and please remember, the experience so far has been with space 
exploration data, so that's where these examples are coming from), written 
accounts of the missions and instruments that collected the data, 
documentation of the reliability of the data and instrument calibration 
concerns, references to further information in published scientific 
literature, descriptions of the format of the "actual" data structures 
- and even software, specific to the data, generated by scientists 
associated with that particular instrument or project (both source-code and 
compiled can be accommodated) that might be used to unpack, manipulate, 
or evaluate the "actual" data. 

The concept of this metadata, combined with (and demarcated from) the "actual" 
data by means of this system of labels, is effectively grasped by supposing 
that an extensive text description of some archived space data, that formerly 
might have been typed on paper and tied with string to some box of tapes or 
platters, in the hope that it would through subsequent years remain associated 
with those tapes or platters, is instead converted to electronic representation 
and stored with (yet is clearly demarcated from) the data on those tapes or 
platters; the metadata is stored on the same physical piece of media that holds 
the data it is describing.  Convenient, eh?

(Remote referencing is do-able also.  But there's so much room on CD-ROMs, 
that it hasn't been a big issue.  The descriptions take up minimal capacity 
compared to the data.)

There is the additional intention that these labels be planned in such a way 
that software may be written to examine the whole of an SFDU object, find 
certain parts of it, and automatically provide services upon, or regarding, 
the information in those certain parts.  This embodies what is often referred 
to as the requirement that SFDUs shall be "human-readable and machine-
parsible"; in other words, the labels will not be of such size or complexity 
as to significantly complicate a scanning by eye of the SFDU's contents, but 
the labels will also contain enough information so that software can usefully 
dissect the SFDU object.  Various of the labels will, of course, respond to 
different aspects of the technical requirement.  Some of the labels will be 
oriented towards demarcating the plain "text" metadata, and others will be 
oriented more at demarcating areas that the intended software will parse 
closely, to obtain information as to how to parse the SFDU as a whole or its 
contained "actual" data. 

The SFDU standard includes the description of the activities of specialized 
personnel, whose work can be broadly characterized as combining that of 
librarians, advisors, and administrators.  These are the personnel who staff 
what are called Control Authority Offices (CAOs).  Another term often used for 
such an office is Member Agency Control Authority Office (MACAO).  The 
functions of these offices include:

- Assisting scientists with the archival and retrieval of data that has been 
stored using SFDUs,

- Advising and assisting people as they use SFDUs,

- Continuing to improve and develop the SFDU standard, by means of the 
practical experience that is acquired.

The writer of this message is a member of the staff of the SFDU Support Office 
(which is associated with the NOST (NASA Office of Standards and Technology)) 
and is also a member of the staff of the CAO at NASA/Goddard (which, formally, 
is called the NASA Primary CAO.)

---

From: [email-masked] (Philip.M. Sargent)
Date: Fri, 11 Jun 1993 14:49:38 GMT

>>   Can anyone provide any information about self-describing file formats

I had a go at this because I was so disgusted with the overly rigid formats
suggested by various materials database producers. I have a number of
reports published on the subject.

Although the SFDU approach is woderful, there is a fundamental problem in
that any totally general system is too complicated for most of the people
who want to use these things - especially non-computer-science-literate
people such as metallurgists. Upwards compatibility with rigidly-formatted
systems (based on tables) is also a useful aim.

I could post some reports on the internet if someone suggests a good place
to put them which is writable:

Materials information in integrated design environments EDRC 05-52-90 (CMU)
data models and data dictionaries for materials info. EDRC 05-51-90 (CMU)
A survey of interchange technologies foir materials property data CUED/E-MANUF/TR.1 Sept.89
A tabular materials data interchange format CUED/C-MATS/TR.163 Sept.89
Associativity in materials property data CUED/C-MATS/TR.162 August 1989

and a few earlier ones, later published in Eng. with COmputers.
See also Technical data interchange using tabular formats, J.Chem.Information
and computer sciences, vol.31 (1991) 297-300.

Enghineering with computers vol.6 (1990) 237-247 and vol.8 (1992) 243-252
Computer Aided Design J. vol.24 (1992) 477-490

and forthcoming chapter "Materials data transfer" in ASTM book "An introduction
to materials databases" to be published during 1993.

I can't post published articles due to copyright, but I can post the tech.
reports.
--Philip Sargent
--
-Philip Sargent
Cambridge University Engineering Dept., Design Centre & Materials Group

---

From: [email-masked] (Walter Douglas Gross)
Date: 11 Jun 1993 12:36 EDT

In article <[email-masked]>, [email-masked] 
(Philip.M. Sargent) writes...

>Although the SFDU approach is wo[n]derful

First, I _would_ like to simply say "thank you" for the appreciative 
comment.  

>                                        there is a fundamental problem in
>people such as metallurgists.

Ah, well ... that's why we do have that SFDU Support Office.

---

From: [email-masked] (Malcolm Slaney)
Date: 11 Jun 1993 11:24:46 -0700

>Ah, well ... that's why we do have that SFDU Support Office.

You have obviously never read "The Design of Everyday Things" by Donald
Norman.  Your data format sounds like many examples from his book.  He likes
to talk about really cool doors that are so non-obvious that you have to put 
a sign telling people whether to push or pull.

Telling people that your really wonderful (and complicated data) format
is ok because you have a full support office isn't very reassuring.

						Malcolm Slaney
						Apple ATG Perception Group

P.S.  Don Norman's book "The Design of Everyday Things" should be required
reading for anybody that designs or builds things (including programs or
data formats) that will be used by other people!  After all, scientists
are everyday people too.

---

From: [email-masked] (Nigel Tzeng)
Date: 11 Jun 93 17:00:24

In article <[email-masked]> [email-masked] (Malcolm Slaney) writes:

>(Walter Douglas Gross) writes:

[Stuff deleted]

>is ok because you have a full support office isn't very reassuring.

SFDU complexity is merely being consistant with the rest of the CCSDS
interfaces...CCSDS isn't anything close to intuitive in any of thier
data formats.  I really like how MSB is bit 0 under CCSDS...*sigh*.

Yes, I do understand why they did it...so that the first bit clocked
down is always bit 0 but it sure makes talking about telemetry points
very difficult when some organizations describe points as bit x at
byte y without telling you if they mean bit 0 to be LSB or MSB.

Now CCSDS did do one wonderful thing...which was to move NASA away
>from TDM and toward packetized data formats...and to be honest SFDUs
aren't all that complex...after all you're only putting a header
around your data.  For most people this means a subroutine call to
either strip off the SDFU header or put one on.

>						   Apple ATG Perception Group

Nigel Tzeng

--
Nigel Tzeng
.sig under construction

---

From: [email-masked] (S. A. Levin [Stewart])
Date: 11 Jun 1993 21:23:09 GMT

The API standards group is working on rev 2 of their RP66 standard which allows
for both public and private dictionary encoding of data.  This format is meant
to allow self-describing data transmittal on a sequential medium.  In addition,
it permits templates to be described which can then be used to map bulk volumes
of data.

-- [email-masked]