Subject: Need database program for organizing and managing many fits

From: herzog at ux5.lbl.gov (Hanan Herzog)
Date: 2 Mar 1993 04:21:31 GMT

Hi,

  The following was sent to me by a colleague: 

"We will soon encounter the problem of many images (requested and
archived) and no overview over them. We would like to set up some kind
of database considering the image header information and the directory
in which the image is. This program should enable us to do searches
for one or several properties of the image. [stuff deleted]. The
header information of the images should be saved and completeted in
ways to make it possible to search after object types, exposure times
and also several properties at a time. (f.e. result all galaxies with
120s exposure times or result the exposure times and coordinates for
all spiral galaxies) [stuff deleted]. Do you know a database program
which could do that or could you search FTP sites for a program like
that?"

  I replied to the original message stating that I did not know of a
program that would do what was needed. I added I would post a message
to UseNet asking, though.

  Any information would be helpful. FTP sites, vendors, anything,
would be appreciated.

Please reply by e-mail. I will forward reponses directly to my
associate.

Hanan Herzog (herzog at lbl.gov)
Lawrence Berkeley Laboratory

---

From: bartho at scsun.unige.ch (BARTHOLDI Paul)
Date: Wed, 3 Mar 1993 13:48:28 GMT

Hanan Herzog (herzog at ux5.lbl.gov) wrote:
: Hi,

:   The following was sent to me by a colleague: 

: "We will soon encounter the problem of many images (requested and
: archived) and no overview over them. We would like to set up some kind
: of database considering the image header information and the directory
: in which the image is. ...

:   Any information would be helpful. FTP sites, vendors, anything,
: would be appreciated.

I would very strongly sugest that you look at the /rdb 'unix' relational
data base.  It runs both on unix machines and on PC using any unix shell
emulator.  The basic idea is to work with pure ascii files and use the
unix facilities as the environment/shell, in particular the i/o redirections
and pipes.

The net result is a fast to very fast RdB, that is VERY easy to use (both
to set up new dB and for interrogations), and that is not costly by any
means.  It is so easy to use that since more than a year it as become one
of my basic tool for most of my work.

You can contact the /rdb maker at  rsw at rsw.rsw.com
Bye the way, I am an astronomer and have nothing to do with them, except
that I bought their product some time ago, and am now more than satisfied.

Best regards,                            Paul Bartholdi.

     +-------------------------------------------------------------------+
     |  Dr Paul Bartholdi             bartho at obs.unige.ch                |
     |  Observatoire de Geneve        bartho at scsun.unige.ch              |
     |  51, chemin des Maillettes     bartho at cgeuge54.bitnet             |
     |  CH-1290 Sauverny              02284682161350::bartho (X.25-psi)  |
     |  Switzerland                   20579::ugobs::bartho (decnet-span) |
     |                                +41 22 755 26 11     (phone)       |
     |  +41 22 755 39 83   (fax)      +45 419 209 obsg ch  (telex)       |
     +-------------------------------------------------------------------+

---

From: thompson at serts (William Thompson)
Date: Thu, 4 Mar 1993 23:32:05 GMT

>unix facilities as the environment/shell, in particular the i/o redirections

	(rest deleted)

It seems to me that the original poster was looking for something already set
up to do more-or-less what he needed to do.  Not to denigrate /rdb or anything,
but once you start talking about doing programming with a generic database,
then there is an incredibly large world of software available to the user.  I
think that it's fair to say that *any* database software should be able to do
the simple job described here.

The questions that come into my mind are:

	1) How large is the database going to be, i.e. how many entries and how
	   much data per entry?  I would then make sure that the selected
	   product could handle that amount of data.

	2) How many people are going to be accessing this data, and how often?
	   In other words, how much traffic will there be in accessing the
	   database.

It may be that these are not going to be a problem, but you should at least
think about it.

I've been looking at database software for doing something very similar for
keeping track of some spacecraft data.  If this is a mission critical
application, and you have a budget that can handle it (i.e. $10,000 and up), I
suggest that you look at the following products:

	Oracle
	Ingres
	Sybase
	Informix
	Interbase
	Progress

Bill Thompson

---

From: leb at gsfc.nasa.gov (Lee E. Brotzman)
Date: Fri, 5 Mar 1993 06:28:16 GMT

>which could do that or could you search FTP sites for a program like

Look into WAIS (Wide Area Information Server).  Jim Fullton, when he was at the
University of North Carolina used WAIS to build a query database of assorted
FITS images.  We have used it here at NASA experimentally to index, search, and
retrieve PDS images, FITS images, and a variety of full text databases,
including a reasaonbly complete set of abstracts from the major astronomical
journals (ApJ, AJ, A&A, PASP, MNRAS, and JGR).

The abstract search is publically available as the first product of Project
STELAR, the STudy of Electronic Literature for Astronomical Research, recently
demonstrated at the Phoenix AAS Meeting.  The WAIS source file for accessing
this database, containing about 60,000 abstracts, is the following:

-----------------------------< abstracts.src >-------------------------------
(:source
 :version 3
 :ip-address "128.183.36.18"
 :ip-name "ndadsb.gsfc.nasa.gov"
 :tcp-port 210
 :database-name "abstracts"
 :cost 0.00 
 :cost-unit :free 
 :maintainer "stelar-info at Hypatia.gsfc.nasa.gov"
 :description "

This database is made available by the STELAR Project, part of the
Astrophysics Data Facility (ADF) at Goddard Space Flight Center in
Greenbelt, MD.  It currently provides access to machine-readable
abstracts for eight leading academic journals of interest to the
astronomical community (ApJ, ApJS, AJ, PASP, A&A, A&AS, MNRAS and JGR).
These abstracts have been supplied by NASA/STI from a database prepared
for NASA's RECON system by an independent abstraction service.  The
RECON system database contains abstracts from as early as the
mid-1960's.  The ADF will update the set of available abstracts on a
regular basis.

Comments, suggestions and/or corrections should be directed to the
STELAR Project directly.  Our electronic mail address is
stelar-info at hypatia.gsfc.nasa.gov.

                       COPYRIGHT STATEMENT

No copying, reproduction or downloading of any portion of this database
in machine readable form, or resale of search results or multicopy
distribution is allowed, except that data my be temporarily stored (for
up to one month) in machine-readable form for reformatting or editing.
Some information in this database is copyrighted.  Copyright is not
claimed in any part of original work prepared by a U. S. Government
officer or employee as part of that person's official duties.

                  DOCUMENT DELIVERY SERVICES

You may obtain copies of referenced reports or articles through your
local library, or by contacting the NASA STI Program.  You may submit
your requests for reports to the STI Program User Services by telephone
using (301)621-0390; or FAX your request to (301)621-0134; or mail your
request to 800 ELkridge Landing Road, Linthicum Heights, MD 21090; or
via INTERNET using ACCESS at MAIL.CASI.NASA.GOV.  Please include the
accession number of the report in your request for copies.  Your
requests for journal articles should be submitted to the STI Program's
contractor for published literature, the American Institute of
Aeronautics and Astronautics (AIAA).  Requests to the AIAA can be made
using (212)247-6500; or FAX your request to (212)582-4861.
"
)
---------------------------------< CUT HERE >--------------------------------

Cut out everything between the dashed lines and save as file abstracts.src
in your wais-sources directory.  For queries just use a sentence that describes
the topic you are interested in, like "ultraviolet galaxies in Virgo".

There are a few other WAIS source files for smaller databases (the AAS Jobs
Register, for instance) available by anonymous FTP from host
hypatia.gsfc.nasa.gov.

-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS
--
-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS

---

From: warnock at gsfc.nasa.gov (Archie Warnock)
Date: Fri, 5 Mar 1993 14:45:59 GMT

>journals (ApJ, AJ, A&A, PASP, MNRAS, and JGR).

WAIS can index just the names of the files, but the indexer is also
smart enough to index the contents of the files and to stop when it hits
the first non-ASCII character, so you can actually index the _headers_
of the FITS files, and not have to worry about hosing the database with
the non-header data.  Very clever.  Very slick.

I've actually done this with some sample Voyager images - indexing the
PDS labels.
--
_______________________________________________________________________
-- Archie Warnock              Internet:  warnock at hypatia.gsfc.nasa.gov
-- Hughes STX                  "Unix --- JCL For The 90s"
-- NASA/GSFC