Subject: Metadata comments

From: [email-masked] (Dale Lancaster)
Date: Mon, 7 Mar 94 13:42:38 -0600

Hi Bruce,

I think you have done an excellent job summarizing some of the key
aspects and problems for data access and management, or what we have
been calling the metadata problem.  Its clear you have spent a lot of
time thinking about this.

 > Our primary technology goal is the design, implementation,
 > maintenance, and evolution of a scientific information system which
 > provides integrated access, processing and display of scientific data.
 > The foundation of the system is a data management subsystem which
 > integrates data from MBARI acquisition systems and other external data
 > sets.  The underlying database is designed to support long term,
 > interdisciplinary research.
 > 
This is a clear goal of many of the attendees at the workshop.

 > Further, we adopted the philosophy that there is no data management
 > partition between data and metadata; data entities and their inherent
 > relationships are modeled as a logically contiguous realm.  By
 > modeling at a high enough level of abstraction, we feel that we can
 > represent changes in technology, practices, procedures, etc as changes
 > in database content and NOT in database structure.  This provides the
I believe we all came to this conclusion as well.  Data is metadata and
vice versa.

 > We have implemented this model within our MBARI Observations Data Base
 > and will implement a variation under a collaborative project with the
 > University of California at Santa Cruz (Real-Time Environmental
 > Information Network & Analysis System - REINAS).  This approach has
 > provided us an opportunity to learn about the data/metadata problem.
 > 
Do you have a whitepaper/email that you can make available to us
describing this system?

 > 1.) Metadata is a heavily overloaded term.  Computer scientists and
 > domain scientists think of these along two overlapping dimensions.
 > Another term would be better, but since we seem stuck with this one we
We had the same problem when trying to nail down the definition.

 > decided to make all components of the definition visible.  METADATA
 > REPRESENTS INFORMATION WHICH SUPPORTS THE EFFECTIVE USE OF DATA FROM
 > CREATION THROUGH LONG TERM USE.  IT SPANS FOUR ANCILLARY REALMS:
 > CONTENT, STRUCTURE, REPRESENTATION AND CONTEXT.  The CONTENT REALM
I like your definition, its more from the point of view of how its used,
rather than what it is.

 > 2.) The modeling of the domain of discourse (data & metadata)
 > indicates that most of the data management complexity is driven by
 > metadata relationships (this may be why this has been traditionally
 > handled by unstructured text).  However, discussions with other
 > scientific information management efforts, from other disciplines,
 > indicates broad topological agreement in the respective information
 > models. (e.g. botanical museum collections and oceanography)
 > 
We all agreed that metadata relationships are the most important thing,
not so much that metadata exists.  I guess metadata without relationships
is the same as collecting small bits of paper (and shiny objects? :-))

 > 3.) The nature of the metadata problem may be summarized as follows:
 > 
 > Metadata must support the sharing of data and information across
 > groups which: operate under different paradigms; have different
 > terminology for similar concepts; must collaborate across time...
 > compare data collected in different contexts with different
 > technology.
 > 
 > There are different metadata needs for current and future users of
 > primary data.
 > 
 > There are different metadata needs for expert/familiar users and other
 > users.
 > 
 > Thers is a COST/BENEFIT imbalance for the implementation of an
 > effective metadata mechanism.... Creators must supply the metadata and
 > bear most of the burden; where distant users enjoy the benefits.
 > 
 > 
Agreed 100% to all the above.

 > 4.) Therfore, any metadata mechanism must fit seemlessly into an
 > integrted data management environment to support the following:
 > 
The key is to integrate into existing environments, this is sometimes
not trivial.  This problem was highlighted as something to be considered.

 > The scientific community cannot expect technology to solve this
 > problem...  Instead they must do a better job in understanding and
 > specifying metadata needs AND establish policies, practices, and
 > technical infrastructure to implement the need solution.
 > 
In particular, how do you enforce someone to give you metadata about
an object?  If my archiving system takes your file, I want you to
fill out a form stating as much information as possible.  Some at
the workshop felt their users (scientist) would generally not fill
out the form adequately.  However I agree with your assertion that
if you reform the policies and practices, it will happen. My idea
was that if you archive a file into my system and it doesn't have
adequate metadata (not jibberish, blanks or whatever), then I delete
the file or I refuse to take it.  Kinda harsh, but it works.

 > I will send a follow-up message which contains the postscript file of
 > a graphical depiction of our information model.. You will note that
 > most of the complex structure represented comes from what is typically
 > defined as metadata.  I hope this will help communicate our work in
 > this area.
 > 
I printed it out, but I wasn't able to follow it too well :-))

 > A few other activities that I am involved in are also looking at this
 > problem: I was the originator of the NEONs system at NRL in Monterey:
 > this system does well with technology independent access to point,
 > grid and image data but is weak in the CONTEXT REALM of the metadata.
 > I am also involved in an NRC study for the National Archives for long
 > term retention of scientific data, and an advisory panel to the EOSDIS
 > project.  Needless to say, these efforts are struggling with the same
 > problems.
 > 
We heard a bit about this at the workshop.

 > I hope this contributes something to this very important discussion
 > and look forward to hard hitting feedback.
 > 
It contributes a boatload in my opinion.  I will keep this mail around
and when we get the metadata WWW home page started, would like to make
a specific link to it.

thanks again! :-)
dml