Subject: Handling many small images

From: cogan at merlin.anu.edu.au (Bruce Cogan)
Date: 11 Jul 1994 06:26:04 GMT

One of our astronomers has a project in which he will be generating
a large number (100's) of small (~10x10 pixel) images. He would like
to archive these in FITS format, but dosn't want the overhead of
a separate file and its header for each image. 

I would appreciate suggestions on the best way of storing these in 
FITS files, such that they would be readily accessible by standard
reduction packages.

-- 

Bruce Cogan                                      cogan at mso.anu.edu.au
Mt Stromlo Observatory				 Tel. +61 6 249 0232
Australian National University                   Fax  +61 6 249 0233
Private Bag
Weston Creek P.O. , ACT 2611, Australia

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 11 Jul 1994 12:36:11 -0400

> FITS files, such that they would be readily accessible by standard

The most accessible way to store the images is to just put each image
into a separate FITS file with a 10x10 primary array.  Every reduction
package will be able to read them, and the total size of even a 1000
files, at 5760 bytes each (the minimum size for a FITS header + data
unit), is less than 6 MBytes which is not much to worry about these
days on most systems.  But this is not very efficient in terms of disk
space and in I/O time if a program has to open and read many different
files.

The most efficient way to store the images would be to store tham all
in a single FITS binary table extension.  (Or the logically related
sets of images could be grouped together in a small number of binary
tables).  Each row of the table could hold one image.  The column that
stores the images would need to be defined as a vector (e.g., TFORMn  =
'100I'  to hold a 10x10 array of 16 bit integers), and the TDIMn
keyword convention would need to be used to specify the dimentionality
of the array (e.g., TDIMn = '(10,10)  ').  You could define as many
other columns in the table as desired to store other descriptive
parameters about the image, such as the coordinates, or date, or
filter.

We use this format a lot for our X-ray data files, but many reduction
packages cannot readily read these images without preprocessing the
FITS files to convert them into a standard primary array image or
IMAGE extension.

Bill Pence
NASA/GSFC

---

From: ron at argus.lpl.Arizona.EDU (Ron Watkins)
Date: 11 Jul 1994 18:44:16 GMT

I would have thought the most efficient way would be to use a 3 dimensional
primary array. This has the advantage of having a single header and there
wont be any padding of the data space for EVERY array, rather only one padding
at the end of all the arrays. If the arrays are all the same size, then a
3-D array might be the best storage mechanism.
			Ron W.
--
Ron Watkins    [ron at argus.lpl.arizona.edu]    /            /~~~~)     /
931 Gould-Simpson                            /            /____/     /
University of Arizona                       /            /          /
Tucson AZ. 85721 -- (602) 621-8606         (____ unar & / lanetary (____ ab.

---

From: tody at lepus.tuc.noao.edu (Doug Tody)
Date: Mon, 11 Jul 1994 17:39:22 GMT

> FITS files, such that they would be readily accessible by standard

Hi Bruce,

My recommendation would be to keep it simple.  A few hundred images is not
that many - the separate file approach should not be ruled out even if it
is all overhead for such a small image.  If greater efficiency is desired
I would consider either using the IMAGE extension (if each image requires
a separate header) or storing the small images as bands of a 3D image (if
all can share the same header).  The most general and efficient approach is
to use a table, but this is overkill for such a simple dataset.

	- Doug

-- 
Doug Tody	National Optical Astronomy Observatories    IRAF project
tody at noao.edu   P.O. Box 26732, Tucson, Arizona, 85726      iraf at noao.edu

---

From: kimball at stsci.edu (Timothy Kimball)
Date: 11 Jul 1994 19:56:22 GMT

In article <2vs3u0$a7e at news.CCIT.Arizona.EDU>, ron at argus.lpl.Arizona.EDU (Ron Watkins) writes:
|> I would have thought the most efficient way would be to use a 3 dimensional
|> primary array. This has the advantage of having a single header and there
|> wont be any padding of the data space for EVERY array, rather only one padding
|> at the end of all the arrays. If the arrays are all the same size, then a
|> 3-D array might be the best storage mechanism.
|> 			Ron W.

Yes, except for the keyword values that will be different for each image.
You would need to handle this with: 1) a whole lot of individual
keywords in the common header, or 2) a table extension listing keyword
values for each plane of the primary array.  So it's not as strightforward as
just putting another axis in the image, unless the keyword values are common
to all the images.

-- tdk