Subject: Draft FAQ

From: [email-masked] (Ilana Stern)
Date: Thu, 26 Aug 1993 20:26:13 GMT

Here's the first draft of a FAQ for sci.data.formats.  Note that it
focuses less on *what* the answers are, than on *where* to get the
answers.  Please send me your comments.

When we are happy with this document, it will be regularly posted to
news.answers and sci.answers as well as here.  Chris Houck will try to 
also make it available as an HTML document in the NCSA archive somewhere.

--------cut here--------

This is a DRAFT FAQ for the sci.data.formats newsgroup.  Contents:

-2)  How to use this document
-1)  How to get a current copy of this document
 0)  Resources for format information
 1)  How to use the data retrieval methods
 2)  Why isn't my favorite format on this list?

Each (major) section has a "Subject:" line, so you can search on the
subject title above to find the section quickly.

This article is copyright (c) 1993 by Ilana Stern.  It may be freely
distributed provided that this copyright notice and the information
on retrieving a current copy are not removed. 

---------------
Subject:  How to use this document

  Most FAQ (Frequently Asked Questions) documents list many questions and 
their answers.  This FAQ is (mostly) devoted to answering only one question:

"Where can I find documentation and software for <X> data format?" 

  As the amount of information available over the networks has been 
increasing, so have the methods by which this information can be obtained.  
No longer is direct usage of FTP the only, or even the most frequent, method 
of obtaining data;  we now have Gopher, Wais, and WWW, as well as many
site-specific interfaces.  Because the information itself may be accessible
in many different ways, this FAQ will identify resources in terms of
URLs (Uniform Resource Locators).  This will also help us convert this
FAQ to a hypertext document, so that it can be used with a WWW browser
to go directly to any of the listed sources.
  Here's a glossary, so you can decode the URLs if necessary to reach 
the sites: 

ftp://host.name.domain/directory/[subdirectory/[filename]]	ftp site
http://host.name.domain/directory/[subdirectory/[filename]]	www server
telnet://host.name.domain					telnet site
gopher://host.name.domain					gopher server
wais://host.name.domain						wais server
news:newsgroup.name						newsgroup

  So, for example, if a document is available at ftp://ncardata.ucar.edu/
it means that you should ftp to ncardata.ucar.edu, and the information is
in the top-level directory.
  If you don't know what these information retrieval methods are, see
section 1 of this document.

---------------
Subject:  How to get a current copy of this document

[Note:  this FAQ hasn't been registered with news.answers yet, so
you can't really get a copy from rtfm.mit.edu yet.] 

A current copy of this document can be obtained by anonymous FTP from
ftp://rtfm.mit.edu/pub/usenet/news.answers/data-formats. 
If you don't know what FTP is, see section 1 of this document.

If you can't use FTP, send email to [email-masked] with
send /pub/usenet/news.answers/data-formats
as the only text in the message (leave the subject blank).

This document is also archived at [other archival sites].  If
you would like to put this article in an archive and want to receive
a new copy automatically at every update, please send me email.
---------------

Subject:  Resources for format information

FITS	Flexible Image Transport System		(Astronomy)

  FITS is the standard data interchange and archival format of the worldwide 
astronomy community.  Format description and software is available from 
ftp://fits.cv.nrao.edu/fits.  
  Another site with FITS information is ftp://nssdca.gsfc.nasa.gov.  The 
newsgroup news:sci.astro.fits is devoted to discussion of the FITS format.  
  If you've searched all these resources and still have questions, you 
can direct them to Barry Schlesinger at [email-masked].

GRIB	GRid In Binary				(Meteorology)

  GRIB is the World Meteorological Organization (WMO) standard for gridded
meteorological data.  Unfortunately it is still not very "standard", as
some organizations use their own versions.  A format description for WMO 
GRIB can be found at ftp://ncardata.ucar.edu/datasets/ds084.5/format_grib,
and read code is in the file access_grib.f in the same directory.
  If you need GRIB to read ECMWF data, the above format description, along 
with the ECMWF-specific parameter table, and a list of differences between
WMO and ECMWF GRIB, is in ftp://ncardata.ucar.edu/datasets/ds111.2/format.
Read code can be found in ftp://ncardata.ucar.edu/datasets/ds111.2/software.
  If all else fails, contact Ilana Stern at [email-masked].

HDF	Hierarchical Data Format		(Various)

  HDF is a self-defining file format for transfer of various types of data 
between different machines. The HDF library contians interfaces for storing 
and retrieving compressed or uncompressed raster images with palettes, and
an interface for storing and retrieving n-Dimensional scientific datasets 
together with information about the data, such as labels, units, formats, 
and scales for all dimensions.
  Source code and documentation are on ftp://ftp.ncsa.uiuc.edu/HDF.  
Some general information on HDF, including a FAQ, is available from
http://www.ncsa.uiuc.edu/SDG/Software/HDF/HDFIntro.html.  

netCDF	Network Common Data Format		(Various)

  NetCDF is an interface for scientific data access which implements a 
machine-independent, self-describing, extendible file format.  Source code
and documentation for the netCDF data access library is available from
ftp://unidata.ucar.edu/pub/netcdf or gopher://gopher.unidata.ucar.edu:70.  
A FAQ is here also, as well as pointers to other available software packages 
which use netCDF data.  Questions and answers about netCDF can be searched in 
the index wais://wais.unidata.ucar.edu:210/unidata-support-netcdf.src.
  A recent paper (Jenter and Signell, 1992) which provides a good introduction 
to netCDF is available as ftp://crusty.er.usgs.gov/pub/netcdf.asce.ps.  
  A mailing list, [email-masked], exists for discussion of the 
netCDF interface, and for announcements of netCDF news:  to subscribe, send 
a request to [email-masked].  The archives of netcdfgroup 
are available from ftp:unidata.ucar.edu/mail-archives/netcdfgroup, and also 
in the searchable index wais://wais.unidata.ucar.edu:210/netcdf-group.src.
  For more information, contact [email-masked].

Miscellaneous graphics formats			(Graphics)
  
  These formats for storing graphics files -- TIFF, GIF, JPEG, FLI, CGM,
and so on -- are more properly discussed in news:comp.graphics.  A small
amount of documentation on these and other graphics formats is on 
ftp://zamenhof.cs.rice.edu/pub/graphics.formats.

---------------
Subject:  How to use the data retrieval methods

  This section only describes FTP and telnet in any detail;  for other
methods, FTP sites are given, so you can get information on them yourself.

1. FTP
2. Telnet
3. Gopher
4. Wais
5. WWW

1. How to use FTP

FTP (File Transfer Protocol) allows transfer of files between two computers
which are on the Internet.  To access the FTP areas listed here, at your
system prompt type "ftp" followed by the name or IP number (the number
in brackets next to the system name in the listing below) of the desired
system.  For example, to access ncardata.ucar.edu you'd type

        ftp ncardata.ucar.edu
or
        ftp 128.117.8.111

Use "anonymous" as your login and your email address as the password (if
requested).

[Note: quotes ("like this") are used to set off names of directories and
files, or commands you'd type, and are not part of these names.]

Not all FTP systems accept the same commands, but here's a list of the
most useful:

        ls      list files in the current directory.
        cd      change directory, e.g. "cd wx" changes to the wx directory.
        binary  sets binary mode
        ascii   sets ascii mode (the default).  Use for retrieving text.
        get     retrieves a file, e.g. "get readme" gets a file called readme.
        bye     exits FTP.

If you can't seem to connect to the site, check to see if it is a telnet
site.  If it is, follow the instructions in the following section instead.

If you can't FTP from your site, use one of the following ftp-by-mail servers:

  [email-masked]          [email-masked]
  [email-masked]           [email-masked]

Send an e-mail message to the closest address, with the lines:

reply [email-masked]     <- replace with your real email address
connect ncardata.ucar.edu         <- for example
cd datasets/ds111.2/software
get access_sun.f
quit

For complete instructions, send a one-line message reading "help" to the
server.  Please don't ask me for help!

2. How to use telnet

Type "telnet" followed by the name or IP number of the desired system.  These
publicly accessible systems generally allow you to log in but put you in
a restricted shell, from which only a certain menu of commands is available.
The description for the site will include the login to use.

If you can't seem to connect to the site, re-check its description in the
document; if it's an FTP site, follow the instructions in the previous 
section instead.

3. Gopher information

Available by ftp at ftp://rtfm.mit.edu/pub/usenet/news.answers/gopher-faq.

4. Wais information

Available by ftp at 
ftp://rtfm.mit.edu/pub/usenet/news.answers/wais-faq/getting-started.

5. WWW information

Supposedly also available from rtfm, but so far only available *on* WWW
as far as I can tell.

---------------
Subject:  Why isn't my favorite format on this list?

  If you don't see a format you're interested in here, it could be one
of three reasons.  First of all, there are a lot of formats which are
out of the scope of this newsgroup:  it ain't named *sci*.data.formats
for nuthin', you know.  Formats used in commercial spreadsheet and
word-processing software aren't scientific data formats, and aren't
discussed in this group.

  Second, it may be that nobody has given the FAQ organizer any information
on sources for information on that format.  So ask the newsgroup -- and
if you do get a response, please let me know what it is!

  Finally, you may ask on the net, and hear nothing, because the 
data format description just *isn't* publicly available.  For most 
scientific data formats, this is a Bad Thing, and most archivists and
scientists *want* to have their format information available.  If
you have such information, but don't have resources to make it
available, please ask around and see if you can get it into an FTP
area or other resource.  Please don't publicize private or proprietary
formats without the permission of the author, though.

-- 
/\
  \_][ <--NCAR Ilana Stern dod#009 r.b. cliff swallow [email-masked]    
      \______________________________________________________________________