Next: Data Format for the NOAO Mosaic
Previous: The Data Handling System for the NOAO Mosaic
Up: Instrument-Specific Software
Table of Contents - Index - PS reprint

Astronomical Data Analysis Software and Systems VI
ASP Conference Series, Vol. 125, 1997
Editors: Gareth Hunt and H. E. Payne

IRAF Data Reduction Software for the NOAO Mosaic

Francisco Valdes
IRAF Group, NOAO, PO Box 26732, Tucson, AZ 85726



NOAO is building a large format (8K×8K pixels) camera using a mosaic of eight 2048×4096 CCDs with eight amplifier readouts initially, and eventually up to 16. This paper describes the data reduction architecture and software being developed for this instrument. The real time display and data capture are described by Tody (1997) and the data format for recording data is described in separate paper by Valdes (1997).

[1]National Optical Astronomy Observatories, operated by the Association of Universities for Research in Astronomy, Inc. (AURA) under cooperative agreement with the National Science Foundation.


Figure 1 illustrates the components and data flow of the NOAO CCD Mosaic Software System. The data reduction components are highlighted here.

Figure: The NOAO CCD Mosaic Software System. Original PostScript figure (11kB).

The data acquisition system (DAS) sends pixel and descriptive information through a message bus to subscribing clients. Two major clients are the real time display (RTD) and the data capture agent (DCA). The RTD displays the image data they are being received and interacts with other processes such as the IRAF task IMEXAMINE. The DCA writes observation information to a FITS Multiextension file (FITS-ME) in the Mosaic data format, and sends observation file status messages to the data reduction agent. The message bus, RTD, DCA, and Mosaic data format are described elsewhere (see Tody 1997; Valdes 1997).


The data reduction agent (DRA) operates on the observation files. It is a high level task, with a sophisticated graphical user interface (GUI), to perform pipeline calibrations, reductions, data quality assessment, archiving and possibly other functions, by communicating with IRAF tasks which access the observation files through the IRAF FITS Image Kernel (Zarate & Greenfield 1995).

The DRA is a continuously running, event-driven process, triggered when the data capture agent finishes writing an observation to disk and when the user initiates an action via the GUI. The first case provides automatic processing and archiving. The second case allows manual calibrations or recalibrations of the automatically processed data when additional or improved calibration data become available. For example, automatic processing can proceed using calibration data from the start of the night, and recalibration can be done after calibrations at the end of the night are obtained.

The DRA is controlled by a GUI. This interface provides a browsing tool for the observations, processing status information, quality assessment information, a tool to view and manipulate the calibration data base, the ability to delete or exclude data, control of the automatic processing, selection of pipeline calibration, reduction, and assessment recipes, control of parameters, recall of raw data, and initiation of recalibrations.

The pipeline calibration, reduction, and quality assessment are defined by recipes selected from a list of recipes. A recipe is basically a macro or script that is executed on a specified disk file or set of disk files. Pipeline calibration consists of the standard CCD calibration operations, setting the WCS, and combining sequences of calibration exposures with scaling and bad pixel rejection. The output includes uncertainties, an exposure map, and pixel masks.


Basic CCD calibration processing is performed by the IRAF task CCDPROC. It provides the standard CCD calibrations for each of the amplifier/CCD readouts of the Mosaic. These include pre/overscan calibration, trimming of pre/overscan and bad edge regions, bad pixel and saturated pixel masking and replacement, zero level (also called bias) calibration, dark count calibration, flat field calibration, and propagation of uncertainties from the detector readout characteristics and the calibration data. This task may also combine calibrated amplifier images into a single image for the CCD. The output Mosaic format then consists of multiple extensions for the CCDs.

The Mosaic version of CCDPROC is actually a relatively simple task that understands the details of the Mosaic data format. It extracts the individual amplifier images and associated data, such as pixel masks and uncertainties, and passes them to a lower level task to do the actual processing. It then takes the calibrated data and updated associated data and puts them back into the Mosaic data format. The lower level task is written to process individual images and associated pixel masks and uncertainties from an input to an output and has no knowledge of the details of the Mosaic data format.

COMBINE (Calibration Images):

Calibration observations, such as flat fields, generally include many exposures to minimize noise. The exposure sequences are identified by the DRA which uses COMBINE to create master calibrations. The DRA keeps track of the master calibrations and applies the appropriate one to new science exposures.

COMBINE combines the individual elements of the Mosaic matched by amplifier or CCD identification. The combining is done pixel-by-pixel within each amplifier/CCD image. It also propagates combined bad pixel masks, variance images, and exposure maps. The Mosaic version of COMBINE is a relatively simple task that understands the details of the Mosaic data format. It extracts the individual amplifier/CCD images and associated data, such as pixel masks and uncertainties, and passes them to a lower level task to do the actual combining. The combined data and updated associated data are then put back into the Mosaic data format.

COMBINE (Dithered or Rastered Science Images):

The combining of calibration exposures is straightforward in the sense that there does not need to be any interpolation, shifting, and coordinate manipulation. The combining of dithered or rastered science exposures is more complex, particularly with regard to coordinate systems. Such data are first resampled into a single image in a celestial coordinate system that can be shifted by integer amounts along both image axes before combining. This is done by MSCIMAGE. COMBINE uses the coordinate system produced by MSCIMAGE to shift and then combine dithered or rastered observations.


The Mosaic World Coordinate System (WCS) maps the image pixels to celestial coordinates on the sky. The mapping is stored in the headers for each amplifier/CCD image. The WCS is defined in two stages. The first stage applies a predetermined calibration and the second stage adjusts this calibration based either on a catalog of sources in the field of the exposure or registers the WCS in multiple overlapping exposures based on common objects in the images.

The WCS calibration file consists of plate solutions for each amplifier/CCD determined from calibration exposures. The plate solution is then applied to observations by adding the telescope pointing and, possibly, instrument position angle. In other words, the WCS is determined once at some telescope pointing reported by the telescope control system. This WCS is used for other telescope pointings with a zero point offset set by the difference in reported telescope coordinates between the calibration and the observation. If the detector may be rotated then the calibration also includes a rotation axis origin determination and uses the difference in instrument position angles to adjust the WCS.

The first stage of setting the WCS for an observation using a calibration file and the telescope pointing is a basic calibration operation performed by MSCWCS. Note that if the WCS is set at an earlier stage by the data acquisition system or the data capture agent then this option of MSCWCS might not be needed.

The WCS set by the first stage is likely to be off by a small amount due to errors in the telescope pointing and instrument flexure. The second stage is to use objects in the image to adjust the WCS. This second stage may use many objects and a full astrometric catalog to make a new calibration. However it is more likely that there are only a few objects and possibly no source catalog. In that case the few objects can be used to make small zero point and rotation adjustments in either an absolute sense if the objects have known celestial coordinates or a relative sense if common objects in multiple exposures are used to register the exposures.

The adjustment of the WCS using a catalog of sources in the field of observation is also performed by the task MSCWCS. It assumes that the existing WCS is fairly close. It takes each source in an input source catalog and searches near the expected position in the image for an object. The object position is determined using a centering algorithm. Once a set of measured pixel positions and catalog celestial coordinates is determined the WCS can be adjusted for an offset and rotation or possibly a new plate solution can be computed.

MSCWCS can be run automatically given a good first WCS and a catalog of sources. If the observer supplies the source catalog or the data reduction agent can automatically obtain a catalog (say by using the telescope coordinates and a catalog server) then this second stage WCS calibration performed by MSCWCS can be part of the basic calibration performed by the DRA.


The task MSCREGISTER uses objects in a set of Mosaic observations to adjust the world coordinate system (WCS) for each observation to best register the objects. This means that overlapping objects will have nearly the same coordinates subject to the limitations set by the form of the WCS description. The set of objects need not appear in all observations but there must be some reasonable overlap so that each observation has common objects with one other observation and all the observations form a single contiguous region.

Several algorithms are required. The objects in each amplifier/CCD image must be cataloged. Then common objects between the many catalogs must be identified. Finally the set of WCS must be registered in some ``best" way.


In the basic processing data from the individual amplifiers and CCDs are kept separate except that data from multiple amplifiers from a single CCD may be merged together into a single unit for the CCD. The observer may analyze the calibrated exposures keeping the readouts separate. This avoids any resampling of the pixel data. However, the observer may wish to resample the elements of the Mosaic into a single large image. MSCIMAGE uses the WCS to resample the pixels into a uniform grid on the sky. This corrects alignment errors in the detector and optical distortions.

Basically, a uniform sky grid of equal sized pixels about some point in the sky is defined and the observed pixels are interpolated to this grid. By using the same grid for dithered or rastered sets of observations, the images can then be combined using only integer pixel shifts in the two image axes. The goal is to require only a single interpolation of the data.


Tody, D. 1997, this volume

Valdes, F. 1997, this volume

Zarate, N., & Greenfield, P. 1996, in Astronomical Data Analysis Software and Systems V, ASP Conf. Ser., Vol. 101, eds. G. H. Jacoby and J. Barnes (San Francisco, ASP), 331

© Copyright 1997 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA

Next: Data Format for the NOAO Mosaic
Previous: The Data Handling System for the NOAO Mosaic
Up: Instrument-Specific Software
Table of Contents - Index - PS reprint