Subject: FITS manipulation software / elementary joins?

From: benning at smoked.cse.ogi.edu (Paul Benninghoff)
Date: 2 Nov 1994 23:52:36 GMT

The following is a quote from a paper from the recent IEEE SSDM
working conference (Modeling Complex Astrophysics Data, by A. Farris
at the Space Telescope Science Institute):

"FITS binary tables are widely use in astronomical data analysis
systems.  Many of these systems contain functions for defining and
manipulating binary tables, including performing elementary join
operations." 

I'm interested in finding out more about such software.  I'm
particularly interested in any DBMS-style operations such as an
"elementary join" operation.  Does anyone know what software is being
referred to here, and where I can find out more about it?  

I've tried the FITS Web pages without much luck, so I thought I'd try
here.

Thanks in advance.

- paul

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Thu, 3 Nov 1994 10:28:42 -0500

benning at smoked.cse.ogi.edu (Paul Benninghoff) wrote:
 
> The following is a quote from [...] A. Farris

The FTOOLS software package available at ftp site legacy.gsfc.nasa.gov
in directory /software/ftools/release contains a number of programs
that perform database type functions on FITS tables (either ASCII or
binary tables).  A new release of this software is expected by the 
end of this month.   FTOOLS has been written and maintained by
a small group of programmers at the HEASARC (High Energy Astrophysics
Science Archive Research Center) at NASA/GSFC.

Currently there is no 'join' function available, but there are tasks
to do the following:

fcreate - create a new FITS table based on ASCII template files
fmerge  - merge rows from several FITS tables into one FITS table
fsort and fmemsort - tasks to sort a table on a given column(s)
fcalc   - calculate values for a column based on user-defined arithmetic
          expression based on the values in other columns
fselect - create a new table containing selected rows from an existing
          table.  The user-supplied selection criterion can be
          an almost arbitrarily complex boolean expression which
          is a function of the values in any columns of the table.
fdump   - display the values in selected rows and columns of a table
fplot   - plot the values in one or more columns of a table

There are more than 100 other tasks in the FTOOLS package that perform
other types of manipulation or analysis of FITS format files (many of
which are oriented towards high-energy astrophysics applications.)

A long-standing limitation in using FITS tables for database
applications has been the difficulty in adding or deleting rows in an
existing FITS table.  The next release of the FITSIO interface library
(due out next week and available in directory /software/fitsio/ at the
above ftp site) will solve this problem by providing subroutines to
insert or delete an arbitrary number of rows in existing FITS tables,
even if the table is followed by other extensions in the FITS file.
This capability will allow FITS tables to be used in very dynamic
situations where the number of rows in a table must be frequently
modified.

-Bill Pence
 NASA/GSFC

---

From: bushouse at stsci.edu (Howard A. Bushouse)
Date: 4 Nov 1994 13:36:17 GMT

In article <3998o4$29k at reuter.cse.ogi.edu>, benning at smoked.cse.ogi.edu (Paul Benninghoff) writes:
|> 
|> "FITS binary tables are widely use in astronomical data analysis
|> systems.  Many of these systems contain functions for defining and
|> manipulating binary tables.
|> 
|> I'm interested in finding out more about such software.  I'm
|> particularly interested in any DBMS-style operations such as an
|> "elementary join" operation.  Does anyone know what software is being
|> referred to here, and where I can find out more about it?  
|> 

 In addition to the FTOOLS software package that Bill Pence has
 already mentioned, there is also the IRAF external package known
 as TABLES that has been developed by the STSDAS group at the Space
 Telescope Science Institute. Within the TABLES package are several
 sub-packages that contain routines for reading and writing FITS
 tables, plotting data from tables, and a whole host of tables operators
 that are capable of handling both ASCII and binary tables. Here's a
 partial list of the "ttools" tasks:

       gtedit - Graphically edit a table.
        imtab - Copy an (IRAF) image to a table column.
        tabim - Copy a table column to an (IRAF) image.
        tcalc - Perform arithmetic operations on table columns.
      tcreate - Create a STSDAS table from an ASCII descriptor table.
      tdiffer - Form a table which is the difference of two tables.
        tedit - Edit a table.
      texpand - Expand tables according to a set of rules.
   thistogram - Make a histogram of a column in a table.
   tintegrate - Numerically integrate one column with respect to another.
        tjoin - Perform a relational join of two tables.
      tlinear - Use linear regression to fit one or two table columns.
       tmatch - Find closest match between rows in two tables
       tmerge - Either merge or append tables.
       tprint - Print tables--both headers and data.
     tproduct - Form the Cartesian product of two tables.
     tproject - Create new table from selected columns in a table.
       tquery - Create a new table from selected rows and columns in a table.
        tread - Browse through a table.
       trebin - Resample a table to uniform spacing.
      tselect - Create a new table from selected rows of a table.
        tsort - Sort a table.
        tstat - Get mean, standard deviation, min, and max for a column.
        tupar - Edit table header parameters.

The TABLES package is available via anonymous ftp from node ftp.stsci.edu
in directory /software/tables. You must of course first have IRAF to be
able to install and use TABLES.

-Howard Bushouse
 STSDAS group
 STScI

---

From: Preben Grosbol <pgrosbol at eso.org>
Date: Mon, 7 Nov 94 12:16:27 +0100

In article <3998o4$29k at reuter.cse.ogi.edu>, Paul Benninghoff writes:
|> The following is a quote from a paper from the recent IEEE SSDM
|> working conference (Modeling Complex Astrophysics Data, by A. Farris
|> at the Space Telescope Science Institute):
|> 
|> "FITS binary tables are widely use in astronomical data analysis
|> systems.  Many of these systems contain functions for defining and
|> manipulating binary tables, including performing elementary join
|> operations." 
|> 
|> I'm interested in finding out more about such software.  I'm
|> particularly interested in any DBMS-style operations such as an
|> "elementary join" operation.  Does anyone know what software is being
|> referred to here, and where I can find out more about it?  
|> 
|> I've tried the FITS Web pages without much luck, so I thought I'd try
|> here.
|> 
|> Thanks in advance.
|> 
|> - paul
|> 

The ESO-MIDAS system has an extended table package with an internal
format very close to that of the BINTABLEs. A wide range of DBMS type
operations are available. The package is available through ESO (see
the ESO homepage on mosaic). A binary version for Linux can be obtained
through anonymous ftp.

Preben Grosbol
SDAG/DMD, ESO

---

From: prep at yarrow.wt.uwa.edu.au (Paul Repacholi)
Date: 08 Nov 1994 18:22:57 GMT

> the ESO homepage on mosaic). A binary version for Linux can be obtained

Please people, if you offer help like this *give the fine URL* etc...

There, i feel better now.

--
~Paul
                                                       +61 (09) 257-1001
prep at yarrow.wt.uwa.edu.au ( preferred )                1 Crescent Rd,
zrepachol at cc.curtin.edu.au                             Kalamunda,
                                                       West Aust    6076

---

From: gtr at mail.ast.cam.ac.uk (Guy Rixon)
Date: 9 Nov 1994 13:38:15 GMT

Interesting.  Are there any import-export tools to connect FITS tables with
common DMBSs like Oracle, Sybase?