The table system is the principle data structure for long term storage in aips++. Within this system, a convention known as the MeasurementSet has been adopted to describe all astronomical data. This convention uses an agreed upon set of standard table column names along with an agreed organization of a main table and several subtables to hold astronomical data. It is necessarily a complex structure because, in general, astronomical data is complex. Within aips++, users will typically never interact directly with the MeasurementSet. Instead, specific, limited views of the MeasurementSet that are tailored for certain analysis goals are constructed as needed. Users interact with the MeasurementSet through a specific view.
In the synthesis imaging work in aips++, a VisibilityIterator and a VisSet have been developed. The VisibilityIterator is the view of the MeasurementSet. It determines how the user steps (iterates) through through the MeasurementSet. Specific values common to synthesis imaging needs are available as function calls to the iterator (rather than as columns in the MeasurementSet). The view may in fact present derived quantities not found in the MeasurementSet. The VisSet is an interface to the MeasurementSet that manipulates reference copies of the MeasurementSet. The VisSet contains three VisibilityIterators: observedCoherence, modelCoherence, and calibratedCoherence. Values are placed back into a MeasurementSet through an iterator. In a general sense, these concepts will be used within SDCalc.
SDCalc is designed around an atom called an SDRecord. An SDIterator is the view of the MeasurementSet that SDCalc will use. An SDIterator serves up individual SDRecords or collections of SDRecords to the rest of SDCalc. It may ultimately be desirable to have the single dish equivalent of the VisSet (an SDSet). This might include, for example, processedData and observedData to associate these two iterators more closely.
SDCalc will manipulate and display SDRecords. There will be an ``active'' SDRecord that SDCalc will display automatically after each operation if the user chooses to do so. Operations will place ther results into that active location when appropriate. A simple ``undo'' is then possible by keeping a copy of the active record before each operation and swapping that copy when ``undo'' is requested. This only allows for a single level of undo. More complicated undo schemes will eventually be considered if necessary.
Within Glish, an SDRecord is a Glish record consisting of the data, a header holding standard parameters associated with the data (either at the time of the observations or during the processing within SDCalc), a non-standard header holding all other ancilliary information (e.g. telescope specific information), and a history of the SDCalc operations that have been done to the data. Each of these parts, data, header, non-standard header and history, is in turn a Glish record.
The data record will consist of a two-dimensional array and the necessary fields to describe each axis of the array. The first axis can be any linear axis in whatever coordinate system the user chooses. The second axis indicates the polarization or stokes values. Usually there will be either 1, 2, or 4 pixels along this axis. The first axis will be referred to as the x-axis throughout the remainder of this document. The SDRecord is flexible enough to hold all one-dimensional data as long as the x-axis coordinate is linear with pixel number.
The basic data flow within SDCalc is from some data source (a MeasurementSet, an aips++ image, a single dish FITS file) through an SDIterator and into an SDRecord. Operations are done on this SDRecord, possibly combining with other SDRecords and leaving an SDRecord as a result. For efficiency purposes it appears that it will be necessary to support operations on SDIterator (e.g. average all SDRecords contained in a specific SDIterator to produce a single SDRecord, fit baselines to all SDRecords in an SDIterator to produce a new SDIterator, etc.). SDCalc will provide display tools to view a single SDRecord or ``browse'' an SDIterator (which is mearly a set of SDRecords).
Data selection is done by generating a new SDIterator from the original SDIterator with the selection criteria applied. If the underlying data source uses the Table system (see comments on IO issues later in this document), then such a selection would produce a new SDIterator which was mearly a reference to the selected rows of the original SDIterator (i.e. no data would need to be copied). It must eventually be possible to select on any field in an SDRecord. Operations on SDIterators (either simply retrieving or storing an SDRecord or operations on whole SDIterators) will be identical when using the original data or selected data.
At some stage, the user will want to store an SDRecord. The SDRecord is sufficiently flexible that certain processing may make it inefficient or even impossible for a general SDRecord to be stored in the current design of the MeasurementSet. Other storage options must be provided. Specifically, it should be possible to store an SDRecord into a table of SDRecords. It may become true that a MeasurementSet will be sufficiently flexible that this is no longer required although we suspect that users will value this ability in any case. Whatever the underlying storage mechanism it should be easy to write an SDRecord or set of SDRecords to a single dish FITS file.