Subject: Some comments on the CEOS Format

From: [email-masked] (Mark Giovannetti)
Date: Wed, 29 Sep 1993 22:32:10 GMT

Lessons Learned from the CEOS Format

  This article summarizes the lessons learned from using and supporting
  the CEOS (Committee on Earth Observation Satellites) Computer Compatible
  Tape (CCT) Format at the Canada Centre for Remote Sensing (CCRS). A
  little background may be necessary. CCRS has been active in the design
  and specification of the CEOS CCT Format since its inception around
  1980 and still maintains it as the standard for satellite and airborne
  imagery data exchange.

  CCRS's main use of the standard is for data distribution to research
  scientists who are our main user group. Almost all of the products
  available from CCRS or through CCRS are sent in the CEOS format in
  one form or another on 9-track tape. Exabyte tapes have also been
  adopted in the last few years.

Pros:

 - The format was designed around tape media and this has the advantage
   of being easily shipped to the user. However with the advent of the
   current and future network technology, it will be necessary to
   include this in the design considerations for a future formatting
   system.

 - The format contains all the information required to successfully
   analyze, correct and interpret the imagery data.

 - The overall logical structure of the format is simple to grasp from a
   tape file point of view.

 - All Canadian data downlink processing facilities produce products
   conforming to the CEOS standard. These products can range from RAW
   data to highly processed scenes of imagery and are supported in the
   CEOS format document for a given sensor.

Cons:

 - The format is well suited for tape media but does not lend itself
   well for disk media. The drawbacks to this are access times for
   reading the data (obviously disk is faster than tape) and the simple
   fact of the need for some tape-to-disk translation software.

 - The format is highly complex at the record level and requires careful
   interpretation of the format documents.

 - There are many variations of the format documents. Each sensor
   (visible, thermal, SAR...) on each platform (satellite, aircraft)
   requires a specific version of the standard. There are currently more
   than ten separate documents. (I may plan to standardize these in an
   electronic form and make them available for FTP or somesuch. They
   currently reside either on paper or on various computing platforms in
   incompatible forms.)

 - The specification of a CEOS document for a new sensor and/or product
   may have errors through propagation, carelessness or unfamiliarity
   with the standard. These errors become apparent when a research
   scientist attempts to read a new product with previously reliable
   software.

 - There was no consideration given to the design of software tools for
   accessing the CEOS format. It is up to the research scientist to
   create `one-of' programs for reading the data.

 - The format is not self-describing and the research scientist must
   have a copy of the format document in order to comprehend the
   contents of a CEOS product.

 - The format is not suitable for archiving large amounts of data.

Summary:

  The CEOS format has been with us for years and will continue to be a
  standard to maintain. However, it must be upgraded or incorporated
  into a complete formatting system which includes software tools and
  more modern methods of format specification. These include the use of
  data dictionaries, metadata for self-description, and the production of
  object classes for object oriented software to list a few.
  Consideration must be given not only to the end user (research
  scientists) but also to those who maintain archives, offer browse
  products and maintain catalogs.

Possible Solutions:

  It is important to build on the work done with the CEOS format.
  Many of the requirements for an imagery data product have been
  addressed by the format.  Therefore, when designing a new
  formatting system the following may be taken into consideration.

 - Remove the binary based record type codes and replace then with an
   explanatory ASCII text message coupled with a sensible tagging scheme
   much like that used in the HDF format.

 - Tabulate all common record fields of the CEOS format and create
   standard text descriptions coupled with a tagging scheme. This could
   also be used to create software data structures with elements using
   the same name/tag scheme. The majority of these fields will contain
   ancillary data of some form. Included in the metadata for these
   fields should be units and data type information. This is where a
   data dictionary is needed.

 - Remove the tape media dependent records from the CEOS format while
   adopting a similar record/data structure as that used for the
   ancillary, auxillary and other supplemental data.

                                    *

-- 
Mark Giovannetti              Canada Centre for Remote Sensing 
[email-masked]                   Ottawa, Canada