From dishfits-request@fits.CX.NRAO.EDU Wed Oct 4 13:26:17 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02422; Wed, 4 Oct 89 13:26:15 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA12945; Wed, 4 Oct 89 13:27:18 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02418; Wed, 4 Oct 89 13:18:59 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891004130929.0000214A0B1@cvax.CV.NRAO.EDU> Status: RO From: HLISZT@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Draft agenda and travel info for single-dish meeting in Green Bank Date: Wed, 4 Oct 89 13:09:29 EDT Single-Dish FITS Standards Workshop Green Bank W. Va, 31 October-1 November 1989 Dear Colleague: Thank you for the interest you have shown in establishing a standard data exchange format for single-dish radioastronomy. With interest and enthusiasm, we will create one! This message contains travel information and a reply form which you can return via post or EMail. We are also sending by conventional mail a package which is similar to this one but with more complete travel information (and a map). The last portion of this mail message contains a draft agenda for the meeting. You will notice that this agenda begins with a formal schedule of presentations concerning various aspects of FITS, but soon becomes only an outline of steps which we might take to design and implement the new standard. Any suggestions you might have regarding the format of the meeting are welcome. If you have (even preliminary) ideas concerning data exchange, please let us hear from you; the floor will be open in what we hope will be a free and spirited exchange of such ideas. If you wish to make a formal presentation, "we" would be glad to place you on the program. On the second day of the meeting, we have placed on the agenda a rather general item concerning single-dish software. If the previous discussions fill the available time, this item will be sacrificed. However, if time is available, it would be informative for the group to hear about recent developments at your telescope. ================================================================================ INFORMATION REGARDING TRAVEL TO GREEN BANK (AND/OR CHARLOTTESVILLE) Transportation will largely be the responsibility of the individual participants. The nearest major airports to Green Bank are in Washington, DC [Dulles (IAD) and National (DCA)] and Pittsburgh, PA. Direct travel to GB from either of these metropolitan areas involves a 4-5 hour drive through the picturesque and twisty mountains of West Virginia and western VA. Otherwise, one might wish to fly into Charlottesville, VA where transport between NRAO Headquarters and Green Bank will be arranged. Green Bank is about 75 miles due West of Charlottesville, or about 120 miles (some 3 hours) by road. About one hour of this trip is fairly adventurous driving. A map giving driving directions to Green Bank is included in the non-electronic version of this mailing. We regret that there can be no financial subsidies from the NRAO. However, once in Green Bank, lodging in the NRAO dormitories and on-site houses (this may be either double or single occupancy) will be provided without charge. The NRAO will also provide meals and frequent liquid refreshment. For those specifically desiring single rooms, lodging in nearby motels can be arranged (by the NRAO) at the participant's expense. Carpooling or van pickup for the sessions will be organized. The weather at the end of October will be fairly cold with crisp nights (Green Bank is at 2700 ft altitude). Visitors are advised to bring warm clothes and to be prepared for the possibility of rain or snow. ================================================================================ REGARDING YOUR TRAVEL PLANS Please return by post to H. Liszt/Single-Dish Fits Meeting NRAO Edgemont Road Charlottesville, VA USA 22903-2475, or EMail via BITNET to HLISZT@NRAO, Internet to HLISZT@NRAO.edu, SPAN to 6654::HLISZT or phone Harvey Liszt 804-296-0344 (an answering machine will pick up the phone after 3 1/2 rings) -------------------------------------------------------------------------------- Name:............................... Phone/E-Mail:............................ I will arrive in (city)....................... at (date/time)................... And will depart from.......................... at (date/time)................... Before the meeting, I will need accomodations in Charlottesville on (dates)............................... After the meeting, I will need accomodations in Charlottesville on (dates)............................... Please arrange transport for me from Charlottesville to Green Bank on 30 Oct. yes....... no....... Please arrange transport for me from Green Bank to Charlottesville on 1 Nov. yes....... no....... I will share a room with...................................... I really strongly prefer a single room yes....... no....... I will be accompanied by (non-participants only)........................ ================================================================================ Single-Dish FITS Standards Workshop--(DRAFT) AGENDA -------------------- day 1 -- TUESDAY, OCTOBER 31, 1989 ------------------------ ==>Introduction: (8:45 am) Welcome and Introductory Remarks.......................H. Liszt History, Background, and Current Status of FITS Origins and Evolution of FITS..................D. Wells Remarks from the AAS WGAS......................R. Hanisch The IAU FITS Standards Committee...............P. Grosbol The NASA FITS Standards Office.................B. Schlesinger Use of FITS in High-Energy Astrophysics................S. Murray (Coffee) ==>Steps Toward a Single-Dish FITS Format: The 1986 Format........................................R. Maddelena Alternative: A Compact Binary Table Format.............D. Wells Other alternatives?....................................(Anyone) General Discussion of Design and Methodology............EVERYONE! (Lunch) ==>Specifying A Standard For Single Dish Data Exchange: Methodology/Architecture/Design Use of tables, random groups, other FITS mechanisms Vocabulary--Variable types; Usage of keywords/table headers Semantics--Meaning of keywords, definition of units World Coordinates--describing the velocity (or frequency) axis (Coffee) ==>Charting the Future, Part 1: For the immediate future Defining the crucial details Defining a working group to settle these details Construction of first-generation FITS readers and writers (Cocktails and Dinner) ================================================================================ Single-Dish FITS Standards Workshop--(DRAFT) AGENDA ------------------- day 2 -- WEDNESDAY, NOVEMBER 1, 1989------------------------ ==>Charting the Future, Part 2: (8:45 am) For the long-term Disseminating the products of the working group General implementation of the new standard Assuring general acceptance of the standard Assuring adherence to the new standard (coffee) ==>General Concerns Affecting the Software Performance of Single Dishes Presentations from various instruments.................? ==>Wrap-up And General Summary of the Workshop ================================================================================ From dishfits-request@fits.CX.NRAO.EDU Thu Oct 5 18:02:59 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA00824; Thu, 5 Oct 89 18:02:57 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA16314; Thu, 5 Oct 89 18:03:23 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA00743; Thu, 5 Oct 89 17:55:48 EDT Return-Path: Message-Id: <8910052150.AA13865@RA.STSCI.EDU> Status: RO From: (Bob Hanisch) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@nrao.edu Subject: single dish data and FITS format Date: Thu, 5 Oct 89 16:50:51 EST Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): From dishfits-request@fits.CX.NRAO.EDU Mon Oct 9 17:01:11 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05146; Mon, 9 Oct 89 17:01:08 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA26636; Mon, 9 Oct 89 17:02:06 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05123; Mon, 9 Oct 89 16:55:43 EDT Return-Path: X-Vms-Mail-To: DISHFITS Message-Id: <891009165133.00004A45081@cvax.CV.NRAO.EDU> Status: RO From: HLISZT@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Comments by R. Hanisch (St Sci) Date: Mon, 9 Oct 89 16:51:33 EDT ****************************************************************** The following was sent to NRAO by R. Hanisch (hanisch@stsci.edu or hanisch@scivax.span) ****************************************************************** From: STSCIC::HANISCH 5-OCT-1989 17:50:34.74 To: STOBIE CC: Subj: current version of FITS memo Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): From dwells@fits.CX.NRAO.EDU Wed Oct 11 17:56:50 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08854; Wed, 11 Oct 89 17:56:48 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA02170; Wed, 11 Oct 89 17:56:20 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA08809; Wed, 11 Oct 89 17:38:09 EDT Return-Path: Message-Id: <8910112138.AA08803@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Re: single dish data and FITS format Date: Wed, 11 Oct 89 17:38:05 EDT From hanisch@stsci.edu Fri Oct 6 11:24:52 1989 Subject: my memo seems to have been chopped off for some people Don, My memo on hierarchical keywords seems to have been truncated for some recipients. Bill Cotton reported to me that his copy had been chopped, and I just checked mine and it also is chopped, at the same place. Could I ask you to check on the mail exploder and see if it might be the culprit? Here's the text. Please send it out to the exploder, if you would. Have to head off to the airport in the next few minutes. Thanks, Bob. [see technical explanation at end -- DWells] Some Comments About Single-Dish FITS Format and Related FITS Issues Bob Hanisch, ST ScI October 1989 In preparation for the meeting in Green Bank later this month, I have written down some ideas concerning how to handle the special case of single dish radio astronomy data within the FITS format. This is being distributed in advance of the meeting so that attendees can think about the issues prior to arrival. Harvey Liszt distributed an e-mail message to all of us which contained a careful analysis of the header structures proposed by Stobie and Morgan. Unfortunately I think this analysis does not really address the general problem, i.e., how to manage the keyword space unique to a certain class of instruments, detectors, or a particular project. From the FITS point of view this is the critical issue, and most of the items discussed in the memo are the details that need to be worked out once the more general problem has been addressed. What Stobie and Morgan described in their original memo is a simple hierarchical keyword space. The presence of the hierarchy is denoted by the use of the SINGLDSH keyword, followed by a secondary hierarchy (GENERAL, NRAO-12M, etc.) which further defines the scope of the keyword definitions. The AIPS system has already implemented a hierarchical scheme of sorts, in that AIPS uses the FITS COMMENT and HISTORY records for storing information unique to AIPS. Non-AIPS FITS readers have no obligation to parse the contents of these cards beyond simply recording them with the other header information, or even ignoring them if they wish. The point here is to have a keyword space which can be hidden from FITS reading programs that do not understand or have no interest in some specialized set of keywords. The advantages of such a scheme include the fact that we need not have any community-wide consensus on the definitions of ALL FITS keywords. Rather, subgroups such as the single-dish radio astronomy community can agree upon their own conventions and be largely independent of other subdisciplines. If we agree that having some sort of hierarchical keyword naming scheme is worthwhile (and I think that this is NOT a foregone conclusion -- more comments on this later), the first thing to do is determine the best syntax for its implementation. I see several difficulties with the proposal of Stobie and Margon (and implicitly with the AIPS technique of hiding similar information behind COMMENT and HISTORY records): o The proposal is limited to a two-level hierarchy, a primary domain (SINGLDSH) and a secondary domain (GENERAL, NRAO-12M, etc.). This seems unnecessarily restrictive, although it is clearly simple to generalize to several more levels. o The keyword hierarchy takes up a lot of space on the header line, leaving little space for values or comments. This situation is exacerbated if deeper hierarchies are to be permitted. o FITS readers require special code to parse the header and retrieve the data values. o Everytime a new domain is introduced, FITS readers everywhere will have to be updated to recognize (and most often ignore) the new domain name. Some time ago I proposed implementing hierarchical keywords by REQUIRING that each hierarchical declaration actually begin with the keyword HIERARCH (this was during a FITS planning meeting held in Charlottesville in Jan. 1988, as I recall). The reasoning was that this is such an ugly keyword that no one else in their right mind would use it, thus it would serve as a flag to indicate that a special parser would have to go into action in order to retrieve the actual keyword name and value buried further to the right on the card. In retrospect, this plan suffers the same disadvantages mentioned above EXCEPT for the last one; by reserving a special keyword to introduce a hierarchical keyword domain, the problem of the arbitrary introduction of new domains is circumvented. However, the other problems, especially the lack of space on the header card, are not resolved. An alternative scheme has come to mind (based on an idea proposed by Doug Tody at the same meeting) which looks much more like simple nested do-loops in Fortran, or structure definition statements in any modern computer language. It would be based on begin/end statements for various keyword domains, and could be nested arbitrarily deep, accommodating things like SINGLDSH/GENERAL/NRAO/12METER/... or whatever. I propose to use keywords of the form KHBEGIN and KHEND to signify a keyword domain (KH = keyword hierarchy; I don't really care what the particular names are so long as we can agree on something unique and representative). For example, the syntax might be something like this (paying little attention to the actual column spacing): . . (standard FITS header) . . KHBEGIN = 'SINGLDSH' / begin a keyword hierarchy called SINGLDSH STATION = 'NRAO-12M' / one or more keywords unique to this UNIQKW2 = 1.2345678 / hierarachy . . KHBEGIN = 'GENERAL' / second level of hierarchy UNIQKW3 = 9.8765432 / one or more keywords unique to this . / second level of the hierarchy . . . (other KHBEGIN keywords define additional levels of the hierarchy, . as needed) KHEND = 'GENERAL' / close out the GENERAL keyword list KHEND = 'SINGLDSH' / close out the SINGLDSH keyword list . . Now, there are some things that I haven't thought through that carefully yet, like whether or not the KHBEGIN and KHEND keywords should have a numeral attached (minimizing the number of keywords with the same name and perhaps making the parsing of the hierarchy a bit more fool-proof). Also, there's the question of whether you would want to allow the full flexibility of going in and out of a particular domain. That is, should this sort of thing be permitted:? KHBEGIN = 'SINGLDSH' / begin a keyword hierarchy called SINGLDSH . . KHBEGIN = 'GENERAL' / second level of hierarchy . . KHEND = 'GENERAL' / close out the GENERAL keyword list . . KHBEGIN = 'NRAO-12M' / another second level hierarchy . . KHEND = 'NRAO-12M' . . KHBEGIN = 'GENERAL' / reopen the GENERAL list . . KHEND = 'GENERAL' KHEND = 'SINGLDSH' / close out the SINGLDSH keyword list This is not a big deal, and maybe it would simply be a matter of style whether or not this would be allowed or just discouraged. However, there are a couple of major advantages of this approach to managing the keyword space: o The keywords within the hierarchy can all be parsed with standard FITS readers. o There is no wasted space on the individual keyword cards, allowing more complete use of the comment field. If there are enough `private' keywords to warrant setting up a hierarchy, the overhead of the BEGIN and END cards is small. o The logic for skipping over the `private' keyword definitions is straightforward: upon encountering a KHBEGIN card, the reader would be free to skip parsing other cards until the corresponding KHEND card was found. In order to simplify this somewhat, we may wish to limit the depth of the hierarchy, say to 9 levels (that should be enough for just about everyone, and would allow the formation of numbered hierarchies, KHBEGIN1, KHBEGIN2, etc., as described above). There is clearly redundant information in using both the numbering scheme AND the value field of the BEGIN/END cards. o The definition of new keyword hierarchies (SINGLDSH, INTERF, AIPS, CCD, HST, etc.) does not require FITS readers to be updated. A one-time update is all that is necessary, and even this is not essential for existing FITS readers to work since they could simply ignore all of this stuff, with no harm done. Thus, if the need for private keyword definition spaces is genuine, a scheme such as that proposed here might be a more viable one for implementation than that previously proposed. However, the question remains as to whether such private keyword domains are even required. Just to review this situation a bit, the prime motivation for having a hierarchical keyword space of some sort is so that various projects, disciplines, etc., can define any set of new keywords without having to worry about name collisions with other such groups. But is this really a concern? Does it really matter if a CCD data set and a single-dish radio data set have inconsistent definitions of BIAS? or REFPT? or DELTAX? I guess it depends on what level of interoperability we expect to achieve with FITS. In a practical sense I think we all recognize that the idea of establishing some sort of global dictionary for FITS keywords is impossible. The hierarchical scheme says, ok, we can't do it so let's hide all of our private definitions. The alternative says, ok, we can't do it but it doesn't really matter because the types of data we're transferring are so heterogeneous. (By the way, I wish to point out something about the practice of defining new FITS keywords. There seems to have arisen some idea that it is forbidden to define keywords other than those mentioned in the series of FITS standards papers. Nonsense. Reread Paper I, p. 366, middle of the 2nd column: "Note that the two long keywords, INSTRUMEN and TELESCOP are names which have been truncated after 8 characters. We suggest this as a good convention for the creation of new keywords." Whether they intended it or not, Don, Eric, and Ron left the door wide open for the creation of new keywords and even recommended how it should be done.) What does matter, I think, is to have a small set of reserved keywords that all FITS readers might reasonably be expected to understand, or at least not interfere with. FITS Paper I defines the fundamental set (SIMPLE, BITPIX, NAXIS, NAXISi (i=1,NAXIS), and END) and an optional set (BSCALE, BZERO, BUNIT, CRVALi, etc.). One solution to this problem would be for the various FITS committees to review Paper I and agree upon such a set of reserved keywords, and then let various projects use the rest of the keyword space to their hearts' desires. This is, after all, the situation we have now and maybe it's not so bad. Anyway, wait until you all see HST data! In any case, I think that a clarification and standardization of some small set of keywords is desirable regardless of whether we adopt a hierarchical naming convention project or discipline dependent keywords. Well, those are my thoughts at the moment. I've gone on a bit longer than I thought, but I hope that those of you who've stuck with me to this point have got some food for thought and that we can have a lively discussion about this in Green Bank. See you all there. ======================= non-FITS technical discussion below: ============= Why did Bob's message on 05 October get truncated? The SMTP (Simple Mail Transfer Protocol) of the Internet terminates a message by a line that has *only* one period (.) on it. This rule allows the mail protocol to work the same way on top of a variety of lower protocols, character sets and OSes. So, what does an SMTP mailer do if a message contains a text line with only a period? Why, double it, of course! And, of course, a mail daemon which receives such a message will change the double period back to a single period before delivery to the customer mailbox. Messages containing single-period lines are rare. Bob's message at 1755EDT last Thursday 05 October contains the first such lines I have seen in several years. His message was truncated *exactly* at the first occurrence of such a line, and so we know that the problem may be that some mail daemon somewhere failed to double the period on transmission. At present I am unable to say with certainty which of three different computers, one at STScI and two at NRAO, was the guilty party. I have added tabs before the periods in this copy to defeat the RFC822 rule. -Don From dishfits-request@fits.CX.NRAO.EDU Fri Oct 13 20:34:52 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02034; Fri, 13 Oct 89 20:34:47 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA02650; Fri, 13 Oct 89 20:34:18 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA02026; Fri, 13 Oct 89 20:08:14 EDT Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891013195606.00000B24031@cvax.CV.NRAO.EDU> Status: RO From: "GUILLOTE@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Single-dish FITS Date: Fri, 13 Oct 89 19:56:06 EDT Date sent: 13-OCT-1989 13:52:50.84 To: DISHFITS@NRAO Single-Dish FITS Format Stephane Guilloteau (1) and Thierry Forveille (2) (1) Institut de Radio Astronomie Millimetrique (IRAM) 300 Rue de la Piscine F-38406 Saint Martin d'Here FRANCE E-Mail "GUILLOTE@FRGAG51.BITNET" (2) Groupe d'Astrophysique de Grenoble (GAG) Observatoire de Grenoble BP 53 X F-38041 Grenoble Cedex FRANCE E-Mail "FORVEILL@FRGAG51.BITNET" Dear collegues, In the previous mails delivered through the DISHFITS exploder, the goal of using FITS for single-dish data was not clearly exposed. We present here an (possibly obvious, and certainly not exhaustive) analysis of the basic needs and goals, leading to a proposal. FITS is designed for "transport" of images or other data. For single-dish data, we may consider several cases of "transport" 0) Transport between the same analysis program, running on identical computers, on different sites. FITS is not required in this case 1) Transport between the same analysis program, running on different computers. FITS may be required, since the binary data format can be different on the two computers. Using FITS instead of dedicated conversion programs avoids the need of dedicated programs on each type of computer 2) Transport between different analysis programs, probably running on different computers. 3) As a subset of 2), possible transport between dedicated single-dish analysis programs and "general" image processing systems (AIPS or others). 4) Archiving. FITS may be desirable in this case, as it guarantees the readability on any computer later on. Each case impose some constraint on the design of a single-dish (or image) FITS format. Case 1) and 4) require that ALL the information be written on FITS tape. This can be handled using dedicated keywords, or, as done for AIPS, substructures in the COMMENT and HISTORY cards. Whatever the choice of implementation, keywords are required matching EXACTLY the data format, with a simple provision for future extensions. IRAM and other observatories are currently using the CLASS program for single-dish data analysis. The data structure for this program is segmented in "Sections" describing some aspects of the data. The currently used sections is described in Appendix in this MAIL. Case 2), considered here as exchange between two single-dish data analysis programs, requires some agreement upon a reasonable subset of keywords in order to preserve most (if not all) information in the transport. Some loss of information seems unavoidable, since not all analysis programs have a one to one correspondence in header parameters. Case 3) is even more tricky. It is likely that a lot of information will be lost in this case. However, a minimal requirement should be that the data can be correctly transferred. Following the original FITS rule, this would imply putting the data in the standard data area, NOT in a table extension. The minimal subset of keywords to be defined would then be NAXIS, NAXISi and BITPIX. However, most image processing are (or should be) able to process coordinates properly, including frequency information, and it would be highly desirable to transfer in the most general way this information too. Looking a little bit closer to what is a single-dish data set, we may consider two cases : - Spectra - Continuum drifts Spectra can be considered as 1-dimensional subsets of 3-D data cubes. AIPS-style FITS headers could be readily used to described completely REDUCED single-dish spectra. As an example, consider the following FITS header, which has been produced by the CLASS to FITS conversion program : ---------------------------------------------------------------------- SIMPLE = T / BITPIX = 16 / NAXIS = 4 / NAXIS1 = 512 / NAXIS2 = 1 / NAXIS3 = 1 / NAXIS4 = 1 / BLANK = 32767 / Blanking value BSCALE = 0.1595546564204E-05 / BZERO = 0.4077937081456E-01 / DATAMIN = -0.1150350086391E-01 / DATAMAX = 0.9305904805660E-01 / BLOCKED = T / BUNIT = 'K ' / CTYPE1 = 'FREQ ' / CRVAL1 = 0.0000000000000E+00 / Offset frequency CDELT1 = 0.1220703125000E+05 / Frequency resolution CRPIX1 = 0.2565000000000E+03 / CTYPE2 = 'RA---GLS' / EQUINOX = 0.1950000000000E+04 / CRVAL2 = 0.5150368630886E+02 / CDELT2 = 0.8333333046443E-01 / CRPIX2 = 0.0000000000000E+00 / CTYPE3 = 'DEC--GLS' / CRVAL3 = 0.3108135491610E+02 / CDELT3 = 0.8333333046443E-01 / CRPIX3 = 0.0000000000000E+00 / CTYPE4 = 'STOKES ' / CRVAL4 = 1.0000000000000 / CDELT4 = 0.0000000000000 / CRPIX4 = 0.0000000000000 / TELESCOP= '100-M ' / OBJECT = 'NGC1333 ' / LINE = 'NH3(1,1) ' / Line name RESTFREQ= 0.2369449499989E+11 / Rest frequency VLSR = 0.6500000000000E+04 / Velocity of reference channel ORIGIN = 'CLASS-Grenoble ' / DATE = '13/10/89' / Date written DATE-OBS= ' 1/ 9/84' / Date observed DATE-RED= '27/ 6/89' / Date reduced END ---------------------------------------------------------------------- Note that CRVAL2 and CRVAL3 are coordinates at the point of tangency, as should be, and CDELT2 CDELT3 represent here the offsets of the observed point with respect to this reference direction. Keywords CD00i00i may be used instead. The frequency axis is defined as an offset to a rest frequency given in keyword RESTFREQ, and the corresponding source velocity is given in keyword VLSR. (It may be better to specify the frequency axis as effective observing frequency, and use a keyword VELO-LSR for the source velocity). This header is very close to a standard AIPS FITS header produced for a 3-D VLM cube for example, and no "vital" information has been lost if the data is already reduced. We are not expert in "image" processing, but presumably such a FITS header could be understood by any FITS reading program without major loss of information (the only two non standard keywords are RESTFREQ and VLSR). Continuum drifts are handled in a similar way: the frequency axis has dimension 1, and one sky coordinate axis has dimension equal to the number of data points. It may be interesting to define a very special "projection code" for drifts in horizontal coordinates, which are frequently used to check the telescope pointing, e.g. something like -HOR. Turning back to points 1) and 4), which imply in fact essentially the same constraints, we believe that a hierarchical structure as described by Bob Hanisch essentially fulfills all the requirements. As an example, here is a (quasi) FITS header part which corresponds to the data structure used in CLASS (See Appendix). Reference to notes are numbered (*i). We did not bother editing a real FITS header, but only the keywords are mentionned. KHBEGIN = 'CLASS' / (*1) VERSION = 'SEP-1989' / Date of version KHBEGIN = 'POINTER' / (*2) GENERAL = 20 / POSITION= 11 / SPECTRO = 15 / Spectroscopic data FSWITCH = 5 / for NPHASE = 1 DRIFT = 0 / Not present BEAM = 0 CALIBRAT= 19 / May be shorter SKYDIP = 0 GAUSS = 9 / for NLINE = 1 HFS = 0 SHELL = 0 CONTINUU= 0 HISTORY = 9 / Sequence = 4 PLOT = 4 BASE = 8 / For 2 windows XCOORD = 0 / (*5) KHEND COMMENT * OBS.GENERAL : General parameters, always present KHBEGIN ='GENERAL' ! (mgen=9,mgen2=11) NUMBER = ! rnum ! Observation number VERSION = ! rver ! Version number TELESCOP= ! rteles(3) ! Telescope name DATE_OBS= ! rdobs ! Date of observation DATE_RED= ! rdred ! Date of reduction SYSTEM = ! rtypec ! Type of coordinates KIND = ! rkind ! Type of data QUALITY = ! rqual ! Quality of data SCAN = ! rscan ! Scan number UT_TIME = ! rut ! UT of observation LST_TIME= ! rst ! LST of observation AZIMUTH = ! raz ! Azimuth ELEVATIO= ! rel ! Elevation TAU = ! rtau ! Opacity TSYS = ! rtsys ! System temperature INTEGRAT= ! rtime ! Integration time KHEND COMMENT * OBS.POSITION : Position information. KHBEGIN ='POSITION' ! (mpos=11) SOURCE = ! rsourc(3) ! Source name EPOCH = ! repoch ! Epoch of coordinates LAMBDA = ! rlam ! Lambda BETA = ! rbet ! Beta LAMBDA_O= ! rlamof ! Offset in Lambda BETA_OFF= ! rbetof ! Offset in Beta PROJECTI= ! rproj ! Projection system KHEND COMMENT * OBS.SPECTRO : Spectroscopic information (for spectra) KHBEGIN ='SPECTRO' ! (mspec=15) LINE = ! rline(3) ! Line name REST = ! rrestf ! Rest frequency NCHAN = ! rnchan ! Number of channels REFERENC= ! rrchan ! Reference channel FRES = ! rfres ! Frequency resolution FOFF = ! rfoff ! Frequency offset VRES = ! rvres ! Velocity resolution VOFF = ! rvoff ! Velocity at reference channel BAD = ! rbad ! Blanking value IMAGE = ! rimage ! Image frequency VELOCITY= ! rvtype ! Type of velocity KHEND COMMENT * OBS.BASE : Baseline information (for spectra of drifts) KHBEGIN ='BASE' ! (mwind=20,mbase=4+2*mwind) DEGREE = ! rdeg ! Degree of last baseline SIGMA = ! rsigfi ! Sigma AREA = ! raire ! Area under windows NWINDOW = ! rnwind ! Number of line windows WIND1(NWINDOW) ! rw1(mwind)! Lower limits of windows (*3) WIND2(NWINDOW) ! rw2(mwind)! Upper limits of windows KHEND COMMENT * OBS.HISTORY : Scan numbers of initial observations KHBEGIN ='HISTORY' ! (mseq=100,morig=2*mseq+1) SEQUENCE= ! rnseq ! Number of sequences START(SEQUENCE) ! rstart(mseq) ! Start can number of seq. (*4) REND(SEQUENCE) ! rend(mseq) ! End scan number of seq. KHEND COMMENT * OBS.PLOT : Default plotting limits. KHBEGIN = 'PLOT' ! (mplot=4) YMIN = ! ramin ! Min Y value plotted YMAX = ! ramax ! Max Y value plotted XMIN = ! rvmin ! Min X value plotted XMAX = ! rvmax ! Max X value plotted KHEND COMMENT * OBS.CALIBRATION : Calibration parameters KHBEGIN = 'CALIBRATION' ! (mcalib=19) BEAM_EFF= ! rbeeff ! Beam efficiency FORW_EFF= ! rfoeff ! Forward efficiency GAIN_IM = ! rgaini ! Image/Signal gain ratio H2OMM = ! rh2omm ! MM of water vapor PAMB = ! rpamb ! Ambient pressure (hPa) TAMB = ! rtamb ! Ambient temperature (K) TATMSIG = ! rtatms ! Atmosphere temp. signal band TCHOP = ! rtchop ! Chopper temperature TCOLD = ! rtcold ! Cold load temperature TAUSIG = ! rtaus ! Opacity signal band TAUIMA = ! rtaui ! Opacity image band TATMIMA = ! rtatmi ! Atmosphere temp. image band TREC = ! rtrec ! Receiver temperature MODE = ! rcmode ! Calibration mode FACTOR = ! ratfac ! Applied calibration factor ALTITUDE= ! ralti ! Site elevation COUNTS(3) ! rcount(3) ! Power of Atm., Chopp., Cold KHEND COMMENT * OBS.GAUSS : Gauss fit results (for spectra or drifts) KHBEGIN = 'GAUSS' ! (MXGAUS=5,mfit=3+6*MXGAUS) NLINE = ! rnline ! Number of components SIGMA_BA= ! rsigba ! Sigma on base SIGMA_RA= ! rsigra ! Sigma on line FIT(3*NLINE) ! rnfit(3*MXGAUS) ! Fit results ERR(3*NLINE) ! rnerr(3*MXGAUS) ! Errors KHEND KHEND = 'CLASS' / End CLASS hierarchy ---------------------------------------------------------------------- (*1) Define a CLASS hierarchy (*2) This structure is not required since the parameters can be derived from a sequential reading of the following ones. (*3) Here appears a rather general problem : the definition of arrays into FITS. Common practice has been to use an index as in NAXISi for example. But this is limited to small arrays in practice (size < 10 probably). In the list of original scan numbers (*4), the size may be larger. This problem is fairly general and here is an opportunity to address it. A possibility would be to use a single (non-indexed) keyword. If the count has been previously defined, this solution is unambigous: auto increment should simply be used by the reading program. The two following examples would be valid, and correspond to the following list of original scans added to produced the current one : 100 to 110, 112 to 120, 345 to 360, 363 and 364, 400. KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100 REND = 110 START = 112 REND = 120 START = 345 REND = 360 START = 363 REND = 364 START = 400 REND = 400 KHEND KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100 START = 112 START = 345 START = 363 START = 400 REND = 110 REND = 120 REND = 360 REND = 364 REND = 400 KHEND However this solution is very space consuming for long arrays. Should table-like data be used for those array keywords? If so, how could it be logically placed if hierarchies are implemented as suggested by Bob Hanisch ? Could we use something more compact such as KHBEGIN ='HISTORY' SEQUENCE= 5 START = 100, 112, 345, 363, 400 REND = 110, 120, 360, 364, 400 SEQUENCE= 1 START = 500 REND = 510 KHEND This is non standard FITS in terms of alignment of formatted variables, but can be easily read on any computer using a list directed I/O Fortran format. The number of values in each line is defined by keyword SEQUENCE here, so that the total number must be computed by adding all values of the SEQUENCE keyword (note that the name SEQUENCE is specific to the example. Other names would apply to other cases). (*5) XCOORD is used in CLASS for data (spectra or drifts) which are not regularly sampled. It has been used for example to store in CLASS format the IRAS LRS survey. The corresponding section contains a unit for the array coordinates (integer code), followed by the array of coordinates of the samples (here "coordinates" may be frequency of course). This problem of irregularly sampled data may be important in single-dish data as shown by the IRAS LRS example. A logical solution to handle that in FITS format is the use of standard extension tables. In so far, we have not discussed the fate of multi-backend and/or multi-beam observations. For multi-backend, the use of AIPS-style FITS header seems to imply to separate each backend data set in a different FITS file. An a priori alternative would be to declare the frequency axis as randomly sampled, but we then turn into several difficulties: 1) for reduced data, considering regularly sampled data as random ones is obviously a lack of simplicity, and probably most processing systems are not even able to do so (3_D VLM data cubes in AIPS with random velociy axis?) 2) for raw data, it is likely that the calibration parameters depend upon the backend. A specific structure would be required for each backend. A different FITS file for each backend is much simpler. The only disadvantage is the loss of storage space, but the cost (and volume) of storage is decreasing every day. For multi-beam, the problem may be different, because the relative beam positions are intrinsically random parameters, while the calibration parameters are mostly identical for all beams. A suggestion would be the use of GROUP data structure like in UVFITS. A few parameters must be precised for each beam however (beam efficiency and beam sizes, and possibly a scaling factor representing the average gain of the backend part used for this beam). Using GROUP data structure may also be possible for reduced (calibrated) data, allowing to store a complete "random" map (regularly sampled maps should use the normal axis coordinates). For raw data, this probably does not make sense, as many calibration parameters will be valid only for a few observations. ---------------------------------------------------------------------- In summary, following those guidelines, our proposal is (in priority order) 1) Always put the data in the FITS data area, NOT in an extension. 2) Put all important informations in a "standard" FITS header, based on the one used in AIPS for example. This "standard" header must be general enough to be understood by ALL FITS reading programs. 3) Adopt the Bob Hanisch proposal for hierarchy implementation and add a list of hierarchic keywords to describe completely the other informations. For simplicity reasons, this list may duplicate informations already present in the "standard" header. 4) Use table extensions for the coordinates of non regularly sampled data. 5) Use GROUP data structure for multi-beam observations, and possibly also for complete reduced "random" maps. 6) Find a solution for "array" keywords; we only suggested possible alternatives. 7) If possible, agree upon names for the hierarchical keywords, in order to ease data exchange between different single-dish reduction programs. (*) This does not preclude use of specific keywords for each analysis program, specially for data archive. Comments please... (*) This is the lowest priority for us, because our FITS reading program has the capability to define equivalent names for a keyword. This capability is easy to implement in any program, and already saved us a lot of additional programming time. ---------------------------------------------------------------------- Turning to the 6 questions asked by W.R. Burns in his original letter, our answers are : 1) CLASS (Continuum and Line Analysis Simple System) for single-dish data, CLIC (Continuum and Line Interferometer Calibration) for IRAM interferometer data, with a data format which is an extension of the CLASS format GILDAS (Grenoble Image and Line Data Analysis Software) for image processing. and AIPS as a reference in interferometry. 2) CLASS CLIC and GILDAS are developped in collaboration between IRAM and GAG, and the software is now used on many sites. 3) External support, limited to bug repairs by lack of manpower. Enhancement done in-house on a "urgent need" basis. In house software works only under VMS up to now. 4) Hardware : fast workstations, probably UNIX Software : UNIX conversion, may be X-windows, upgrade of image processing to handle more specifically IRAM Plateau de Bure data, and more homogenously 30-m and Plateau de Bure data. 5) We use AIPS, and distribute our software. Sharing code with some standardization would seem unpractical given our limited manpower (however if anybody is interested by our small code to get equivalent names, it can be done of course!). 6) Of course, single dish data could make use of other packages. This is precisely the goal of our proposal points 1 and 2. PS: Stephane Guilloteau is unable to attend the meeting, and Thierry Forveille will be the IRAM representative. ---------------------------------------------------------------------- Appendix : data structure (in VAX-VMS FORTRAN syntax) used for single-dish data in CLASS format, with corresponding standard FORTRAN variable names and physical meaning: * STRUCTURE /OBSERVATION_DESCR/ * * OBS.POINTER : Length of sections. If 0, section not present. STRUCTURE /POINTER_DESCR/ POINTER INTEGER GENERAL INTEGER POSITION INTEGER SPECTRO INTEGER FSWITCH INTEGER DRIFT INTEGER BEAM INTEGER CALIBRATION INTEGER SKYDIP INTEGER GAUSS INTEGER HFS INTEGER SHELL INTEGER CONTINUUM INTEGER HISTORY INTEGER PLOT INTEGER BASE INTEGER XCOORD END STRUCTURE * * Section -2 /CRPAR/ * OBS.GENERAL : General parameters, always present STRUCTURE /GENERAL_DESCR/ GENERAL ! (mgen=9,mgen2=11) INTEGER NUMBER ! rnum ! Observation number INTEGER VERSION ! rver ! Version number CHARACTER*12 TELESCOPE ! rteles(3) ! Telescope name INTEGER DATE_OBS ! rdobs ! Date of observatio n INTEGER DATE_RED ! rdred ! Date of reduction INTEGER SYSTEM ! rtypec ! Type of coordinate s INTEGER KIND ! rkind ! Type of data INTEGER QUALITY ! rqual ! Quality of data INTEGER SCAN ! rscan ! Scan number REAL*8 UT_TIME ! rut ! UT of observation REAL*8 LST_TIME ! rst ! LST of observation REAL*4 AZIMUTH ! raz ! Azimuth REAL*4 ELEVATION ! rel ! Elevation REAL*4 TAU ! rtau ! Opacity REAL*4 TSYS ! rtsys ! System temperature REAL*4 INTEGRATION ! rtime ! Integration time END STRUCTURE * * Section -3 * OBS.POSITION : Position information. STRUCTURE /POSITION_DESCR/ POSITION ! (mpos=11) CHARACTER*12 SOURCE ! rsourc(3) ! Source name REAL*4 EPOCH ! repoch ! Epoch of coordinat es REAL*8 LAMBDA ! rlam ! Lambda REAL*8 BETA ! rbet ! Beta REAL*4 LAMBDA_OFF ! rlamof ! Offset in Lambda REAL*4 BETA_OFF ! rbetof ! Offset in Beta INTEGER PROJECTION ! rproj ! Projection system END STRUCTURE * * Section -4 * OBS.SPECTRO : Spectroscopic information (for spectra) STRUCTURE /SPECTRO_DESCR/ SPECTRO ! (mspec=15) CHARACTER*12 LINE ! rline(3) ! Line name REAL*8 REST ! rrestf ! Rest frequency INTEGER NCHAN ! rnchan ! Number of channels REAL*4 REFERENCE ! rrchan ! Reference channels REAL*4 FRES ! rfres ! Frequency resoluti on REAL*4 FOFF ! rfoff ! Frequency offset REAL*4 VRES ! rvres ! Velocity resolutio n REAL*4 VOFF ! rvoff ! Velocity at refere nce channel REAL*4 BAD ! rbad ! Blanking value REAL*8 IMAGE ! rimage ! Image frequency INTEGER VELOCITY ! rvtype ! Type of velocity END STRUCTURE * * Section -5 * OBS.BASE : Baseline information (for spectra of drifts) STRUCTURE /BASE_DESCR/ BASE ! (mwind=20,mbase=4+2*mwind) INTEGER DEGREE ! rdeg ! Degree of last bas eline REAL*4 SIGMA ! rsigfi ! Sigma REAL*4 AREA ! raire ! Area under windows INTEGER NWINDOW ! rnwind ! Number of line win dows REAL*4 WIND1(MWIND) ! rw1(mwind)! Lower limits of wi ndows REAL*4 WIND2(MWIND) ! rw2(mwind)! Upper limits of wi ndows END STRUCTURE * * Section -6 * OBS.HISTORY : Scan numbers of initial observations STRUCTURE /HISTORY_DESCR/ HISTORY ! (mseq=100,morig=2*mseq+1) INTEGER SEQUENCE ! rnseq ! Number of sequence s INTEGER START(MSEQ) ! rstart(mseq) ! Start can numbe r of seq. INTEGER END(MSEQ) ! rend(mseq) ! End scan number of seq. END STRUCTURE * * Section -7 * OBS.PLOT : Default plotting limits. STRUCTURE /PLOT_DESCR/ PLOT ! (mplot=4) REAL*4 YMIN ! ramin ! Min Y value plotte d REAL*4 YMAX ! ramax ! Max Y value plotte d REAL*4 XMIN ! rvmin ! Min X value plotte d REAL*4 XMAX ! rvmax ! Max X value plotte d END STRUCTURE * * Section -8 * OBS.FSWITCH : Frequency switching information (for spectra) STRUCTURE /FSWITCH_DESCR/ FSWITCH ! (mxphas=5,mfsw=1+4*mxphas) INTEGER NPHASE ! rnphas ! Number of phases REAL*8 DECALAGE(MXPHAS) ! rdecal(mxphas) ! Frequency off sets REAL*4 DUREE(MXPHAS) ! rduree(mxphas) ! Time per phas e REAL*4 POIDS(MXPHAS) ! rpoids(mxphas) ! Weight of eac h phase END STRUCTURE * * Section -14 * OBS.CALIBRATION : Calibration parameters STRUCTURE /CALIBRATION_DESCR/ CALIBRATION ! (mcalib=19) REAL*4 BEAM_EFF ! rbeeff ! Beam efficiency REAL*4 FORW_EFF ! rfoeff ! Forward efficiency REAL*4 GAIN_IM ! rgaini ! Image/Signal gain ratio REAL*4 H2OMM ! rh2omm ! MM of water vapor REAL*4 PAMB ! rpamb ! Ambient pressure ( hPa) REAL*4 TAMB ! rtamb ! Ambient temperatur e (K) REAL*4 TATMSIG ! rtatms ! Atmosphere temp. s ignal band REAL*4 TCHOP ! rtchop ! Chopper temperatur e REAL*4 TCOLD ! rtcold ! Cold load temperat ure REAL*4 TAUSIG ! rtaus ! Opacity signal ban d REAL*4 TAUIMA ! rtaui ! Opacity image band REAL*4 TATMIMA ! rtatmi ! Atmosphere temp. i mage band REAL*4 TREC ! rtrec ! Receiver temperatu re INTEGER MODE ! rcmode ! Calibration mode REAL*4 FACTOR ! ratfac ! Applied calibratio n factor REAL*4 ALTITUDE ! ralti ! Site elevation REAL*4 COUNTS(3) ! rcount(3) ! Power of Atm., Cho pp., Cold END STRUCTURE * * Section -16 * OBS.SKYDIP : For Skydips observations. No associated data. STRUCTURE /SKYDIP_DESCR/ SKYDIP ! (msky=10,mskydip=10+4*msky) CHARACTER*12 LINE ! rsline(3) ! Line name REAL*8 REST ! rsrest ! Rest frequency REAL*8 IMAGE ! rsimag ! Image frequency INTEGER NSKY ! rnsky ! Number of points o n sky INTEGER NCHOP ! rnchop ! - - - - chopper INTEGER NCOLD ! rncold ! - - - - cold load REAL*4 ELEVATION(MSKY) ! relev(msky) ! Elevations REAL*4 EMISSION (MSKY) ! remiss(msky) ! Power on sky REAL*4 CHOPPER (MSKY) ! rchopp(msky) ! Power on choppe r REAL*4 COLD (MSKY) ! rcold(msky) ! Power on cold l oad END STRUCTURE * * Section -9 /CRGAUS/ * OBS.GAUSS : Gauss fit results (for spectra or drifts) STRUCTURE /GAUSS_DESCR/ GAUSS ! (MXGAUS=5,mfit=3+6*MXGAUS) INTEGER NLINE ! rnline ! Number of componen ts REAL*4 SIGMA_BASE ! rsigba ! Sigma on base REAL*4 SIGMA_RAIE ! rsigra ! Sigma on line REAL*4 FIT(3*MXGAUS) ! rnfit(3*MXGAUS) ! Fit results REAL*4 ERR(3*MXGAUS) ! rnerr(3*MXGAUS) ! Errors END STRUCTURE * * Section -12 /CRSHEL/ * OBS.SHELL : "Stellar shell" profile fit results (for spectra) STRUCTURE /SHELL_DESCR/ SHELL ! (mshell=43) INTEGER NLINE ! rlshel ! Number of componen ts REAL*4 SIGMA_BASE ! rbshel ! Sigma on base REAL*4 SIGMA_RAIE ! rrshel ! Sigma on line REAL*4 FIT(20) ! rnshel(20)! Fit results REAL*4 ERR(20) ! reshel(20)! Errors END STRUCTURE * * Section -13 /CRNH3/ * OBS.HFS : "Hyperfine Structure" profile fit results (for spectra) * (e.g. NH3, HCN) STRUCTURE /HFS_DESCR/ HFS ! (mnh3=27) INTEGER NLINE ! rlnh3 ! Number of componen ts REAL*4 SIGMA_BASE ! rbnh3 ! Sigma on base REAL*4 SIGMA_RAIE ! rrnh3 ! Sigma on line REAL*4 FIT(12) ! rnnh3(12) ! Fit results REAL*4 ERR(12) ! renh3(12) ! Errors END STRUCTURE * * Section -10 /CRCONT/ * OBS.DRIFT : Continuum drift description (for drifts) STRUCTURE /DRIFT_DESCR/ DRIFT ! (mcont=16) REAL*8 REST ! rfreq ! Rest frequency REAL*4 WIDTH ! rwidth ! Bandwidth INTEGER NPOINT ! rnpoin ! Number of data poi nts REAL*4 REFERENCE ! rrpoin ! Reference point REAL*4 TIMOFF ! rtref ! Time at reference REAL*4 ANGOFF ! raref ! Angular offset at ref. REAL*4 PA ! rapos ! Position angle of drift REAL*4 TIMRES ! rtres ! Time resolution REAL*4 ANGRES ! rares ! Angular resolution REAL*4 BAD ! rcbad ! Blanking value INTEGER SYSTEM ! rctype ! Type of offsets REAL*8 IMAGE ! rcimag ! Image frequency REAL*4 COLLA ! rcolla ! Collimation error Az REAL*4 COLLE ! rcolle ! Collimation error El END STRUCTURE * * Section -11 * OBS.BEAM : Beam-switching parameters (for spectra or drifts) STRUCTURE /BEAM_DESCR/ BEAM ! (mbeam=5) REAL*4 AZIMUTH ! rcazim ! Azimuth of observa tion REAL*4 ELEVATION ! rcelev ! Elevation of obser vation REAL*4 THROW ! rspace ! Beam spacing REAL*4 PA ! rbpos ! Position angle of beams INTEGER SYSTEM ! rbtype ! System for angle END STRUCTURE * * Section -15 /CRPOINT/ * OBS.CONTINUUM : Double gaussian and baseline fit results (for drifts) STRUCTURE /CONT_DESCR/ CONTINUUM ! (mfcont=19) INTEGER NLINE ! rlcont ! Number of componen ts REAL*4 SIGMA_BASE ! rbcont ! Sigma on base REAL*4 SIGMA_RAIE ! rrcont ! Sigma on line REAL*4 FIT(8) ! rncont(8) ! Results REAL*4 ERR(8) ! recont(8) ! Errors END STRUCTURE END STRUCTURE * From dwells@fits.CX.NRAO.EDU Mon Oct 16 10:03:50 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05546; Mon, 16 Oct 89 10:03:48 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA09306; Mon, 16 Oct 89 10:03:40 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA05491; Mon, 16 Oct 89 09:55:57 EDT Return-Path: Message-Id: <8910161355.AA05485@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Cc: dwells@fits.CX.NRAO.EDU Subject: Re: Single-dish FITS Date: Mon, 16 Oct 89 09:55:54 EDT Dear friends, The Guilloteau and Forveille paper is a perceptive and sophisticated discussion of FITS issues for single-dish astronomy. It is a valuable contribution in preparation for the Green Bank workshop. Here are some comments, corrections, differences of opinion: 1. re matrices versus tables: Stephane and Thierry stress that data transfer from single-dish systems to "general" IP systems should be by means of images (n-dimensional matrices) in classical FITS form. I agree that *if* the data are an n-dimensional matrix then the Basic FITS format should be used. However, I am optimistic that we will soon have effective interchange capability for other types of data structures, particularly tables. I contend that we must make the effort to build and demonstrate this capability in many systems so that astronomical data interchange and archiving will no longer be confined to matrices. I would like to see the single-dish community take a leadership role in this effort. 2. re list-directed-READ for FITS value fields: In March 1979 Eric Greisen and I agreed that FITS keyword value fields would use the notation of FORTRAN list-directed READs. The list-directed READ design allows for lists of values, and Eric and I understood what we were proposing. My memory is that when the design went to the review panel in April-May 1979 there were strong protests that many computer systems were not sophisticated enough to cope with completely free-field headers, that our design was too sophisticated and that it ran the risk of being ignored by many groups in astronomy. We compromised. We stopped talking about multi-valued keywords and we even gave up the free-field concept --- we specified that the required keywords of a FITS header must be in a rigid format so that unsophisticated readers would have an easier job. The multiple-value problem is fundamental. I do not have a nice, neat answer for it. In headers we have another problem: Eric and I decided that all standard header lines would fit into one "card". This limits how many values a keyword can have. I believe now that the best solutions will use tables, with separate tuples for the values. 3. re irregularly sampled data: The classical answer to this problem is the random-groups format. Indeed, since 1980 the leading FITS designers have all been somewhat sad that we did not think of random-groups until after the Basic FITS Agreement had already been implemented in production systems. We wish that Basic FITS had been the random-groups format, so that simple matricies would be a special case. In 1984 we made the Generalized Extensions Agreement have this form. Jim Condon of NRAO and his collaborators have made several surveys of the northern sky with the 300-foot in a rapid scanning mode using a seven-beam feed. The randomly sampled data were formatted as random-groups FITS. AIPS is able to read such data (see relevant sections of Going AIPS for details) and there are tasks which can grid the samples into maps of the sky. Indeed, the tasks are variations on the aperture synthesis task UVMAP (the Fourier transform step is skipped). The random samples are sorted using task UVSRT. I.e., FROM THE POINT OF VIEW OF A PROGRAMMER, RANDOM SAMPLES OF FLUX ON THE SKY ARE STRUCTURALLY ALMOST IDENTICAL TO THE RANDOM SAMPLES OF THE UV PLANE ACQUIRED IN APERTURE SYNTHESIS, AND SIMILAR SOFTWARE APPROACHES ARE FEASIBLE. Note that scanning infrared satellite data (IRAS) and high-energy photon event data fall in this same class. Although the 300-foot surveys were continuum data only I believe that the AIPS tasks are more general, that they can sort and grid spectral-line surveys. Bill Cotton's "3-D Binary Tables" design, which he devised for VLBA calibration, is yet another approach to the irregularly sampled data problem. It is not as elegant mathematically, but the bit stream that it produces is essentially *identical* to that produced by random groups (so that there is no difference in efficiency) and he can transmit data types that random-groups cannot handle. 4. re hierarchical keyword syntax: Bob Hanisch has raised two issues: (1) "Do we really need a hierarchical namespace?" and (2) "What syntax should we use?". I still believe that the FITS community should endeavour to transmit *MEANING* (semantics) as well as mere *DATA* (syntax), and I believe that it will be necessary to organize our name-space in order to achieve this goal reliably. However I am grateful to Bob for having the courage to go against the common wisdom in this area, to make us think about the issue. Regarding syntax, there are two proposals. Either we devise rules for what I call the "push-down-stack" syntax (the concept discussed by Hanisch and endorsed by Guilloteau and Forveille), or we adapt the existing "domain-name-system" rules for single keyword=value header cards. I think that either approach can probably be made towork in the end, but I have more confidence in the second approach. I have a paper of my own which has been in preparation for several weeks, and I expect to broadcast it soon. Regards, Don Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2] National Radio Astronomy Observatory | SPAN: NRAO::DWELLS [6654::] Edgemont Road | BITnet: DWELLS@NRAO Charlottesville, VA 22903-2475 USA | UUCP: ...!uunet!nrao.edu!dwells +1-804-296-0277 (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278 From dwells@fits.CX.NRAO.EDU Mon Oct 16 17:23:36 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA06260; Mon, 16 Oct 89 17:23:28 EDT Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA10347; Mon, 16 Oct 89 17:22:44 EDT Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA06207; Mon, 16 Oct 89 17:03:31 EDT Return-Path: Message-Id: <8910162103.AA06200@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Binary Table Structure for Spectra Date: Mon, 16 Oct 89 17:03:26 EDT Dear Friends-of-FITS: The paper below is a proposal to the single-dish radio community, which could adopt it even if the rest of astronomy doesn't want to do so. I, of course, hope that it may be possible to forge an alliance across all of astronomical spectroscopy. My actual proposal is only the first 9 pages. The remaining 23 pages are appendices containing the text of the Floating Point Agreement, the Grosbol/Wells Hierarchical Keyword draft proposal and a detailed description of AIPS support for single-dish random-scan data and Cotton's 3-D Binary Table extension format. Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2] National Radio Astronomy Observatory | SPAN: NRAO::DWELLS [6654::] Edgemont Road | BITnet: DWELLS@NRAO Charlottesville, VA 22903-2475 USA | UUCP: ...!uunet!nrao.edu!dwells +1-804-296-0277 (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278 ------------------- LaTeX, 32 pages, cut here ------------------------- \documentstyle[11pt,twoside]{article} \pagestyle{headings} \raggedbottom % \begin{document} \title{A Binary Table Structure for Spectra} \author{Donald C. Wells\\ {\small \raisebox{0.1cm}{\rule{0cm}{0.3cm}}NRAO, Charlottesville, VA}\\ {\small 804-296-0277, {\tt dwells@nrao.edu}}} \date{{\em DRAFT} \today} \maketitle \begin{abstract} The 3-D binary table FITS extension is proposed as an efficient format to transmit groups of astronomical spectra and associated parameters. Syntactic conventions for keywords and table column labels are suggested. \end{abstract} \tableofcontents \clearpage \section{The Problem} The basic FITS (Flexible Image Transport System) Agreement of 1979 defined a syntax for transmitting N-dimensional matricies. It was (and still is) perfectly capable of transmitting spectra, for which N=1, as vectors of binary data described by ASCII headers using the keyword=value notation. However, such usage is inefficient: in most cases more bytes are needed for the header than for the binary vector! Spectroscopy observations generally need more ancillary parameters, more keyword=value pairs, than do imaging observations and this only exaggerates the disproportion between header and data for spectroscopy. The basic FITS format is simply not efficient for spectroscopy applications. Spectroscopy observations often involve large numbers of short integrations, which implies that large numbers of ASCII headers must be written, transmitted, archived and processed. Although many people have wished that the vectors as well as the headers could be transmitted in ASCII form so that {\em everything} would be human-readable, this is simply not a practical option for our new data systems in which more and more feeds are analyzed by spectrometers with higher and higher resolution. Binary notation is essential for the spectra, and we must strive to reduce the overhead of the headers. This means that even the headers must be in binary, and redundancy must be minimized. \section{The Solution} We need binary data structures which include the ancillary parameters in binary form, bound to the binary vectors. These structures must be described in FITS style in order to make them {\em flexible.} The ASCII description will require almost as many bytes as would a basic FITS header. To minimize this overhead we need to be able to transmit {\em arrays} of the structures. In this concept only one copy of the ASCII description is needed for an arbitrarily large number of binary spectra plus parameters, all grouped together in one FITS file. There are two existing FITS formats that have the desired property: the random groups format\footnote{Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 371, "An Extension of FITS for Groups of Small Arrays of Data".} and the 3-D binary tables format\footnote{Cotton, circa 1986, coded in AIPS; see Appendix~\ref{app:C3DBTF} and Section~\ref{sect:C3DBTF}.}. There is no question that random groups are a feasible solution, but binary 3-D tables are preferable, for two reasons. First, experience shows that many people have difficulty understanding the random groups concept but everyone claims that they understand tables intuitively. Second, the random groups format requires that the same data type be used for both spectra and parameters but the 3-D binary tables allow mixed data types, a great convenience and valuable notational power. I propose that Cotton's 3-D binary table format be adopted for astronomical spectroscopy data interchange. \section{Conventions} The basic concept that I am proposing is that a set of spectroscopic observations be regarded as a {\em table}, with the spectral vectors appearing as one of the columns of the table and the ancillary parameters appearing in other columns. The following subsections propose certain conventions for the organization of the table. \subsection{Minimizing Redundancy}\label{sect:redundant} Many of the parameters associated with a set of spectroscopic observations are constant for the whole set. If each such parameter must appear in a separate column of the table the same bytes will appear over and over. I propose that such parameters be moved to the extension header and encoded as keyword=value, and that they be regarded as a {\em virtual column.} \subsection{Multiple Feeds} Many spectroscopic instruments today observe several positions on the sky simultaneously. There are two possible encoding conventions. First, we could encode such observations as one structure containing several binary vectors, with many parameters shared between the spectra and many other parameters appearing once for each of the spectra. Alternatively, we could encode the several spectra as separate structures, with the parameters which are identical being duplicated but without the notational complexity of the multiple instances of spectra and parameters in one structure. I believe that the latter case, the separate structure scheme, is easier to design and to implement, and so I propose that it be adopted. Some spectrometers measure more than one band simultaneously. This case is analogous to multiple feeds and the separate structure approach handles it with no new notation. In some cases datasets contain spectra with different numbers of channels. I propose that the number of channels be specified in a column. If all spectra have the same length the column need not be carried with the spectra in the structures --- it can be given once in the extension header (see section \ref{sect:redundant}). \subsection{Hierarchical Table Labels} Table column labels pose the same problems of potential name conflict as do keywords. It appears that the only feasible solution will be some kind of hierarchical classification scheme. The two draft proposals available currently are the Hierarchical Keyword Agreement (the ``domain-name-system'' proposal) of Grosb{\o}l and Wells, whose text is given below in Appendix~\ref{app:HKA}, and the ``push-down-stack'' proposal of Tody and Hanisch, as discussed in Hanisch's DISHFITS-exploder message of 05~October. In the remainder of this paper I will discuss the Grosb{\o}l/Wells notation because it leads to a natural generalization for table labels. I have not yet decided whether the Tody/Hanisch proposal offers an equally suitable solution for this application. The FITS ASCII Tables standard\footnote{Harten, R.H., Grosb{\o}l, P., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 365, "The FITS Tables Extension".} does not specify any limit on the length of column label strings. This is yet another example of the continuing need for interpretation of the rules of the standards. Let us assume that the string-value rule of the Basic FITS Agreement\footnote{Wells, D.C., Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 363, "FITS: A Flexible Image Transport System". See p.~364-365.} applies: string values may be of any length which fits on one card (80 characters), but should be unique/useful if truncated to 8 characters. I therefore propose that hierarchical classification codes be used to obtain uniqueness in the labels, {\em but that the order be reversed so that the last substring, the keyword, will be in the first 8 characters.} If the same code string is used as a keyword in the extension header, as proposed in Section~\ref{sect:redundant} above, it will appear in normal order, with the {\tt HIERARCH} code in the first 8 columns and the keyword at the end of the string, just before the equals sign. \subsection{Units} I strongly recommend that only SI units be used in data interchange in astronomy, as specified in the Basic FITS Agreement. \subsection{Spectral Format Table Labels} I propose that column names and parameter values should be the keyword and axis-type names used previously in FITS designs, wherever possible. By this I mean that in situations where in other proposals keywords are used in FITS headers that those same names be used in table column labels for tables of spectra, and that where various conventions are specified for the value fields of those keywords that those same conventions be used for the value fields in those columns. Thus, I am proposing that the parameter columns be regarded as {\em virtual header cards}. \section{Status of Relevant Draft Standards} \subsection{Generalized Extensions, ASCII Tables} The two FITS standards were published in Astron.~Astrophys.~Suppl. in 1988, and were approved by IAU Commission~5 at the Baltimore meeting in August 1988.\footnote{IAU Resolution B1, p.~10, IAU Information Bulletin 61, January 1989.} \subsection{The Floating Point Agreement} The FPA has been discussed for at least four years in committees. The prototype implementations were done circa 1985-6. The final text was adopted 28~February~1989, and is given in Appendix~\ref{app:FPA}. The FPA was endorsed by the European Working Group for Coordination of Astronomical Software in May 1989, and by the AAS Working Group for Astronomical Software in June 1989. The Agreement is now under consideration by the IAU FITS Working Group, which is awaiting the final interchange and interoperability proof for the 64-bit data type. Activation is anticipated for 1990. \subsection{Binary 3-D Table Format}\label{sect:C3DBTF} Bill Cotton designed his table format as an extension of the standard ASCII table format in order to represent certain complicated calibration table structures needed for the VLBA analysis software. The only description currently available is from section 14.5.3 of ``Going AIPS'', the AIPS programmer's reference manual. This text has been extracted from the AIPS directories and is given below in Appendix~\ref{app:C3DBTF}. This extension was implemented in AIPS circa 1986, and code supporting it has been operational on several hundred computers for several years now. The extension was implemented in MIDAS circa 1987, and AIPS-MIDAS interoperability trials have been made. As of 1989 the extension is being implemented as a format for photon lists in high energy astronomy, first for the ROSAT data analysis software. The 3-D Binary Table design is generally believed by the FITS standards committees to be the draft proposal ``most likely to succeed''. Only one or two trivial changes are anticipated when it finally passes the IAU sometime in the future. \subsection{Hierarchical Keyword Agreement} The original idea was implemented by Eric Greisen in the first FITS software for the NRAO-Charlottesville IBM synthesis mapping package in 1979. A more recent prototype circa 1986-7 is the SNGLDSH format at NRAO. The concept has been discussed in the FITS committees since 1987. A current draft proposal by Grosb{\o}l and Wells dates from 28~February~1989; it is reproduced below in Appendix~\ref{app:HKA}. Alternative concepts are still being offered. At present no clear consensus exists favoring any one proposal. \section{An Example Showing Packing Efficiency} The preceding discussion has proposed Cotton's FITS extension as a solution appropriate for all spectroscopic applications in astronomy. At present the main astronomy discipline considering this problem is single-dish radio astronomy. There has been much preliminary work involving keywords, syntax and even semantics for this application, and it need not be repeated here. Indeed, even the header syntax for the 3-D extension is well documented in an appendix to this paper, and so I choose not to give a full example. The important point to be addressed is the {\em packing efficiency} which is possible for the proposed design.. Consider a radio spectrometer with 1024 channels in floating point form. A data file will start with a basic FITS header which has no data matrix ({\tt NAXIS=0}). This will be followed by the extension header, which will specify {\tt XTENSION='A3DTABLE'} (once the IAU approves this extension the value will probably be {\tt '3DTABLE'}). Each spectrum will be encoded as a binary data structure which is described with the notational machinery of the 3-D tables. A grossly simplified data structure which has an example of each data type might be: \begin{center} \begin{tabular}{|l|l|l|l|l|l|l|} \hline label: &{\tt SCAN} &{\tt SOURCE} &{\tt BASEFREQ} &{\tt DELTFREQ} &{\tt NCHAN} &{\tt SPECTRUM} \\ \hline format: &{\tt 1J} &{\tt 20A} &{\tt 1D} &{\tt 1F} &{\tt 1I} &{\tt 1024F} \\ \hline bytes: &4 & 20 & 8 & 4 & 2 & 4096 \\ \hline \end{tabular} \end{center} The {\tt SCAN} number is a 32-bit integer, the {\tt SOURCE} name is a string, the frequency of the reference channel (in Hertz, of course) is a double precision number and the frequency increment is single precision. The spectrum itself is single precision floating point and the number of channels {\tt NCHAN} is specified by a 16-bit integer (note that the {\em maximum} number of channels is given by the format of the {\tt SPECTRUM} field; {\tt NCHAN} might be less than this number). {\em The column labels used here are not to be regarded as a part of my proposal; rather, if my 3-D table concept is accepted for astronomical spectra an ad hoc task force should negotiate these details.} The binary data types can be flexible: one site might encode {\tt SCAN} numbers as 16-bit integers while another used 32-bits. Likewise, some implementations may choose to encode spectra using 16-bit integers rather than 32-bit reals in order to save bytes in the files (the FITS machinery for integer scaling permits this to work even for inherently non-integer values). For this example there will be 2880 bytes in the basic header, 5760 bytes in the extension header, 4134 bytes per spectrum and one tapemark. It is true that a real spectral observation will probably need about 50 parameters, not 5, but even at 4 bytes per parameter this will add only 200 more bytes per spectrum. Therefore, it will still be true that for one spectrum the header overhead will dominate. Indeed, 50 more columns in the table will require several more logical records of extension header to document, making the imbalance even worse! However, additional spectra will add no more overhead. If the table contains as many as 20 spectra the efficiency will exceed 90\%. \section{Unresolved Issues, Other Ideas} \begin{itemize} \item{\bf Hierarchical Keywords:} There is no consensus yet on the syntax, or even on the necessity of a hierarchical data space. \item{\bf Vocabulary, Semantics, Units:} This proposal is only concerned with the syntax for transmitting spectra. Each discipline of astronomy must agree on the vocabulary and on the semantics and units for its special language. \item{\bf World Coordinates:} We must agree on conventions for describing the mapping between pixel number and frequency in the spectra. I recommend that we utilize notations similar to those devised by Eric Greisen for imagery. This means that we need to identify how many types of mappings occur in astronomical instruments and assign names to them. We must analyze their mathematical forms to determine how many parameters each has and attempt to devise a uniform convention for the notation. An important question will be whether all of astronomy can agree to use frequency as the independent variable for spectroscopic data interchange; if not, we will have both wavelength and frequency (wavenumbers are just frequency by another name, of course). All of these world coordinates issues are beyond the scope of this paper. \item{\bf Higher-Level Organization of Datasets:} Multiple named table extensions may be grouped together in one file --- this provides a higher level organizational tool. The Generalized Extensions standard also specifies keywords and conventions for a hierarchical structure of extensions, which is yet another higher level organizational tool. \item{\bf Auxilliary Tables:} The FITS table extensions could be used to transmit calibrations, source lists, etc., in addition to the spectra --- this could permit the elimination of even more redundancy in the sense of ``normalizing'' a relational DBMS schema. \end{itemize} \appendix % ================================================================== \clearpage \section{Floating Point Proposal for FITS}\label{app:FPA} \begin{center}Don Wells, Preben Grosb{\o}l\end{center} \begin{center}28 February 1989\footnote{This text is very slightly revised from the version that was endorsed by the European FITS Committee in April 1989 and by the AAS Working Group for Astronomical Software in June 1989. It will now be reviewed by the IAU FITS Working Group for adoption as a part of the FITS standards, and will probably be activated in 1990.}\end{center} \subsection{The Proposal} The Basic FITS, Random Groups and Generalized Extensions Agreements are revised to add IEEE-754 32- and 64-bit floating point numbers to the original set of FITS data types. BITPIX=-32 and BITPIX=-64 signify 32- and 64-bit IEEE floating point numbers; the absolute value of BITPIX is used for computing the sizes of data structures. The full IEEE set of number forms are allowed for FITS interchange, including all special values (e.g., the 'not-a-number' cases). The order of the bytes will be sign and exponent first, followed by the mantissa bytes in order of decreasing significance. The BLANK keyword will be ignored by FITS readers when BITPIX=-32 or -64. \subsection{Discussion} The dynamic ranges of astronomical data have increased steadily in the years since the original FITS Agreement (March 1979) which defined 8-, 16- and 32-bit integers as the only data formats. All integer formats are limited by an absolute error, whereas floating point formats can span a much wider numerical range with only a relative error. Floating point formats are desirable for the applications that need increased dynamic range. Many astronomical data systems now use floating point in data processing, and conversion to and from FITS integer formats sacrifice accuracy and dynamic range and consume significant computer time. Almost all of the new computers designed since about 1981 use the IEEE format (see references below). We are proposing that FITS be revised to allow transmission of 32- and 64-bit floating point data within the FITS format using the IEEE standard. This Floating Point Agreement will also apply to the Random Groups Extension and to those Generalized Extensions for which BITPIX is not explicitly restricted (e.g., BITPIX=8 for XTENSION='TABLE'). In order to transmit floating point data in the main data matrix we must add some new convention to the main FITS header. We propose to add two new allowed values for the basic keyword BITPIX. The set of values specified in the original FITS Agreement was 8, 16 and 32; we propose to add -32 and -64 to indicate IEEE single- and double-precision floating point data. Thus, the number of bits per pixel would be the absolute value of BITPIX. This implies that the number of 2880-byte logical records used by the main data matrix must be computed using the absolute value of BITPIX (this rule was specified in the Generalized Extensions Agreement (Grosb{\o}l etal. 1988)). The IEEE standard specifies a variety of special exponent and mantissa values in order to support the concepts of plus and minus infinity, plus and minus zero, 'denormalized' numbers and 'not-a-number' (NaN). We propose that all of these special cases should be fully accepted for FITS interchange. We propose that the order of the bytes should be sign and exponent first, followed by the mantissa bytes in order of decreasing significance (i.e., the standard non-byte-swapped order). The BLANK keyword of the original FITS Agreement should be ignored by FITS readers when BITPIX=-32 or -64 (the NaNs of the IEEE format will act as the blank). FITS writers should not write the BLANK keyword if BITPIX=-32 or -64. The BSCALE and BZERO values should be applied by FITS readers if they differ from 1.0 and 0.0. However, such usage be regarded as bad practice, and strongly discouraged, because of the risk of generating overflows and underflows in FITS readers. \subsection{Bibliography} "An Implementation Guide to a Proposed Standard for Floating-Point Arithmetic", COMPUTER magazine, Vol. 13, No. 1, pp.68-79, January 1980. COMPUTER, Vol. 14, No. 3, March 1981. ANSI/IEEE 754-1985, "IEEE Standard for Binary Floating Point Arithmetic". ISDN-xxxxx? Wells, D.C., Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 363, "FITS: A Flexible Image Transport System". Greisen, E.W., Harten, R.H.: 1981, Astron. Astrophys. Suppl. Ser. 44, 371, "An Extension of FITS for Groups of Small Arrays of Data". Grosb{\o}l, P., Harten, R.H., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 359-364, "Generalized Extensions and Blocking Factors for FITS". Harten, R.H., Grosb{\o}l, P., Greisen, E.W., Wells, D.C.: 1988, Astron. Astrophys. Suppl. Ser. 73, 365, "The FITS Tables Extension". \subsection{Implementation Notes} Note-1: Many data analysis systems today use floating point internally and so conversion costs for FITS input and output will be reduced by this proposal. For machines with IEEE hardware the conversion cost will be nearly zero, especially when BZERO=0 and BSCALE=1 and the reader has two DO-loops (one with and one without scaling). Note-2: The interpretation of the IEEE sign, exponent and mantissa bit fields is precisely specified by the five rules of Section 3.2.1 of ANSI/IEEE Standard 754-1985. Algorithms for transformation between IEEE and non-IEEE formats can be derived from these rules. The FITS Committees will provide a central archive for conversion software for various architectures. FITS readers on non-IEEE systems should test the IEEE exponents for the special values 0 and 255/2047 (for 32/64-bit numbers). Exponent zero should be mapped to the local value for floating zero (exponent zero is used by IEEE for 'denormalized' numbers as well as for both plus and minus zero); exponent 255/2047 should be mapped to the local value used for blank pixels (all IEEE infinity and NaN cases have exponent 255/2047). The IEEE exponent field can be selected from the 32/64-bit values by masking with hexadecimal byte string 7F,80,00,00/7F,F0,00,00,00,00,00,00 (i.e., the constant {\tt 7F800000}/{\tt 7FF0000000000000} hexadecimal on computers with the normal word ordering; for VAXes use constant {\tt 00007F80}/{\tt 0000000000007FF0} instead). The two tests are made by comparing the result against zero and against the mask. FITS writers executing on non-IEEE systems should map their local value for blank pixels to an IEEE NaN value; we suggest that all bits set (minus one in 32- and 64-bit two-s complement integers) is a good choice. The FITS writer should also map local floating point absolute values which exceed the largest value of IEEE (approximately 10**38 for 32-bit, 10**304 for 64-bit) to IEEE-Infinity. We recommend that absolute values smaller than the smallest normalized IEEE value (approximately 10**-38 or 10**-304) be mapped to zero. These two tests are particularly important on machines which use 'hexadecimal' normalization, with an exponent range of approximately 10**75 in single-precision. The DEC VAX architecture is an important special case. VAX systems use the DEC-proprietary 'F' format for 32-bit floating data and their byte order is 'swapped'. Once the swap has been corrected the F-format differs from the IEEE 32-bit format only in that the exponent bias differs by one and the (hidden) binary point is shifted by one bit. The two differences add, and the total effect is that the two number systems differ by a factor of 4.0 (VAXF = 4.0 * IEEE32, IEEE32 = 0.25 * VAXF). This floating point multiplication operation is little, if any, more expensive than the conversion operation between integer and floating point which is required by the original FITS Agreement. VAX systems should test the exponent field on input to detect the infinity and NaN cases and map them to the local blank code using the test specified previously. The minus zero and denormalized cases may also need to be trapped. Initial prototype implementations of this Agreement have disclosed that some IEEE hardware produces -0.0 results and that reserved operand errors occur in VAXes when these values are used as operands. The 32-bit operand should be compared against 32768 to detect this case, and the result 0.0 be given rather than multiplication by 4.0. The DEC VAX architecture supports two different double precision formats, 'D' and 'G'. The 'G' format has the same relationship to IEEE 64-bit format that the DEC 'F' format has to IEEE 32-bit (VAXG = 4.0 * IEEE64). The 'D' format, however, requires a complicated transformation to convert to/from IEEE 64-bit; programmers should consult archives of FITS code to obtain suitable algorithms and implementations for this case. % =================================================================== \clearpage \section{Hierarchical Keywords in FITS Headers}\label{app:HKA} \begin{center}P.~Grosb{\o}l and D.~Wells\end{center} \begin{center}28 February 1989 (Draft-1)\end{center} \subsection{The Proposal} A new standard keyword HIERARCH\footnote{This name was derived by the traditional FITS rule that names longer than 8 characters should be truncated. It is, in fact, an English word in its own right: the {\bf Oxford English Dictionary} says (definition B.2) that a {\em hierarch} is the ``...commander of the celestial hierarchy''!} is introduced for all FITS headers, both Basic FITS headers and Generalized Extension headers. The remaining 72-characters of an 80-character HIERARCH line will contain one or more keyword classification codes followed a single keyword=value pair. Code and keyword names will be 1-8 characters (uppercase alphanumeric plus the minus sign), just as for standard keywords. Column 9 will be blank; the first code will begin in column 10 and additional codes and the keyword will be delimited by one or more blanks. Value fields will be decoded according to the rules of Basic FITS. The intent of this convention is to provide a hierarchical classification of keywords in order to reduce the possibility of conflicts of keyword names, especially 'proprietary' keywords. The set of approved top-level classification codes will be controlled by the FITS Working Group of IAU Commission 5. \subsection{Example} \begin{verbatim} 1 2 3 4 5 6 123456789012345678901234567890123456789012345678901234567890 HIERARCH topcode level2 code3 keyword = value \end{verbatim} \subsection{Discussion} The IAU FITS Working Group will need to decide whether top-level classification codes should be assigned for individual institutions and projects, or for disciplines of astronomy. Name-spaces assigned to institutions can be managed privately by those institutions. Name-spaces assigned to disciplines will require the creation of international committees. The discipline groups could assign institutional codes under their top-level codes as well as agreeing on discipline-specific generic codes and keywords. The latter approach can lead to a greater degree of standardization among special interest groups in astronomy, without burdening the IAU Working Group. This is, in principal, very desirable. However, it is unclear whether or not such interest groups can be organized in practice. \subsection{Notes} \begin{enumerate} \item The Basic-FITS rule that equal signs must be followed by one or more blanks is relaxed for HIERARCH lines because the rules for value-field syntax prevent ambiguities. \item The maximum number of classification codes is naturally limited by the finite length of a HIERARCH line; however, implementors should probably limit HIERARCH codes to no more than five levels. \item Note that Basic FITS does not specify whether keywords are limited to the set of uppercase alphanumeric characters plus the minus sign, but those are the only characters occurring in the examples given in the 1981 paper. \item Lines 2/4 and 2/5 of Figure 1 on page 370 of the Basic FITS paper (Astron. Astrophys. Suppl. Ser. 44, 363-370, 1981) show an example of a hierarchical keyword notation which is similar to the one proposed here. It is regretable that this notation was not recommended more clearly in the Basic FITS paper, as much confusion would have been prevented over the years. \end{enumerate} % =================================================================== \clearpage \section{3-D Binary Table Extension Format}\label{app:C3DBTF} % document translated from DEC RUNOFF to LaTeX format % by program RNOTOTEX version CVF02B at 13-APR-1989 10:52:05.03 % Source file: CHAP14.RNO %\chapter{FITS Tapes} {\bf Note:} The text of this appendix is taken from Chapter~14 of {\bf Going AIPS}, the AIPS programmer's reference manual. The chapter describes how AIPS uses the FITS agreements. It gives an overview of FITS from the AIPS perspective, and then proceeds to discuss each of the FITS formats used by AIPS. Section~\ref{app:C3DBTF2} below (14.5.3 in {\bf Going AIPS}) describes the 3-D binary table format, which Bill Cotton designed circa 1986 to meet the special requirements of the calibration databases for the VLBA. Subsections of the {\bf Going AIPS} chapter which are titled ``Image Files'', ``Random Group (UV data) Files'', ``Older AIPS Tables'', ``AIPS FITS INCLUDEs'', and ``AIPS FITS Parsing Routines'' have been deleted because they are not relevant to this paper. The subsection titled ``Single Dish Data'' has been retained because of its special interest for single-dish radio astronomers; it describes the support provided by AIPS for forming N-dimensional images out of signals (continuum or multi-spectral) sampled at {\em random} locations by various scanning instruments. This section shows the random-groups approach to the spectroscopic format problem. It is important to realize that any data structure which can be realized within the random groups concept can also be realized within the 3-D binary tables concept. Indeed, at the bit stream level the two formats are essentially identical. The binary tables design does support several data types which are not possible in random groups (the most important case is character strings); this is a fundamental advantage for the table approach. \subsection{Overview } The principal route for getting data and images into and out of AIPS is by FITS (Flexible Image Transport System) format tape files. FITS is an internationally adopted medium of exchange of astronomical data and allows easy interchange of data between observatories and image processing systems. FITS also has the advantages that it is a self-defining format and that the actual bit pattern on the tape is independent of the machine on which the tape was written. The purpose of this chapter is to describe the general features of FITS and the details of the AIPS implementation. This chapter is not intended to be a rigorous description of the FITS standards. See the chapter on devices for information on read and writing tapes in AIPS. The fundamental definition of the FITS system\index{FITS format} is given in Wells, Greisen, and Harten (1981), with an extension described in Greisen and Harten (1981). A proposed further extension is given in Harten, Grosb{\o}l, Tritton, Greisen and Wells, (1984). FITS has been adopted as the recommended medium of exchange of astronomical data by the IAU, the Working Group on Astronomical Software (WGAS) of the AAS, and comparable working groups in Europe. AIPS now also supports the proposal of these working groups for the writing of blocked FITS tapes.\index{blocked tapes} Because of the great flexibility of the FITS system, many of its features have been adopted for the internal data storage format in AIPS. See the chapter on the catalog header for more details on the AIPS internal storage format. There are three main portions of a FITS file (1) the main header, (2) the main data, and (3) any number of records containing auxiliary information. In addition, an extension of the original definition of the FITS structure allows storage of ungridded visibility data. Each of these is discussed in detail in the following sections. \subsection{Philosophy } FITS is a philosophy as much as a data format. The underlying philosophy is to provide a standardized, simple, and flexible means to transport data between computers or image processing systems. FITS is standardized in the sense that any FITS reader should be able to read any FITS image, at least to the degree that the array read is of the correct dimensions and the pixel values have at least the correct relative scaling. In addition, any FITS reader should be able to cope with any FITS format tape and, at least, skip over portions, or ignore keywords, that it doesn't understand. \index{FITS format} The requirement of simplicity means that the implementation of FITS reading and writing should be fairly straightforward on any computer used for astronomical image processing. Simple also implies that the structure of the file should be self-defining and, to a large degree, self-documenting. The main advantage of FITS is its flexibility. Due to the self-defining nature of the files, a large range of data transport needs are fulfilled. The introduction of new keywords gives the ability to add new pieces of information as needed, and the use of generalized extension files allows almost unlimited flexibility in the type of information to be stored. Thus, FITS can grow with the needs of the Astronomical community. The great flexibility of FITS is a potential weakness as well as a strength. There is a great temptation to proliferate keywords and new extension file types. This should be done with great caution. Since FITS is a worldwide medium of data exchange, there needs to be coordination of keywords and extension files to prevent duplication and inconsistencies in usage. The most fundamental philosophical ideal of FITS is that no change in the system should render old tapes illegal or unreadable. This philosophy is reflected in the AIPS implementation of FITS in that all obsolete implementations (e.g., old CLEAN component or antenna extension files) are trapped and processed in the most accurate manner possible. \subsection{Single Dish Data } Observations made with filled aperature instruments are frequently made at essentially random positions on the sky, possibly using a number of offset feeds or detectors. This type of data may be convienently described using the random groups (UV FITS) format. The FITS form of this data is the same as visibility data except that the number and meaning of the random parameters are different. The celestial coordinates may be either Right Ascension and declination or projected coordinates about a specified tangent point. A logical record consists of all data recorded from a given beam on the sky at a given time. A dummy AN table is optional. \subsubsection{Single dish random parameters } The single dish random parameter types (PTYPEn) are described in the following: \begin{enumerate} % list nest 1 \item 'RA' and 'DEC': These random parameters are the Right Ascension and Declination of the observation in degrees. If the coordinates have been projected onto the tangent plane then the RA and Declination types become 'RA--xxxx' and 'DEC-xxxx' where xxxx is the projection code. See the chapter on AIPS catalogue headers and/or AIPS memoes 27 and 46 for details of the projection codes. These random parameter these are required but the order is arbitrary. \item 'DATE': The time tags for the data are kept in the form of Julian date in days. This random parameter is required but the order is optional. \item 'BEAM': This random parameter gives the beam number + 256. This random parameter is optional. The beam offset makes the data look more like uv data and more of the the AIPS uv data tasks will work for this data. \item 'SCAN': This random parameter gives the scan number. This random parameter is optional. \item 'SAMPLE': This random parameter gives the sample number in the scan. This random parameter is optional. \end{enumerate} % - list nest 1 \subsubsection{Single dish regular axis coordinates } The units of the regular axis coordinates are defined by convention; the conventions used by AIPS for the regular axis types (CTYPEn) are the following: \index{uv FITS format} \begin{enumerate} % list nest 1 \item 'COMPLEX': the complex axis consists of the real, imaginary and (optional) weight. Magic value blanking is supported. The imaginary part may be used to carry any baseline values which have been subtracted. This axis is required. \item 'STOKES': this axis is used to describe which Stokes' parameters are given; the conventions are the same as used internally in AIPS. These conventions are discussed in the chapter on disk I/O. This axis is required. \item 'FREQ': the frequency axis coordinates are in Hz. This axis is required. \item 'IF': The IF axis is a construct which allows irregularly spaced groups of frequency channels. The IF number specifies an entry in the ('AIPS CH') table which must follow the data if this axis is present. This table gives the offsets from the reference frequency specified by the FREQ axis. This axis is optional but if it is present then a CH table must be present. \item 'RA' and 'DEC': the celestial coordinates are given in degrees. The values associated with these axes are irrelevant (although they should be present) for unprojected data. For data with projected coordinates the coordinate values of these axes should be the tangent point, i.e. the position on the sky at which the plane onto which the coordinates are projected is tangent to the celestial sphere and these axes should become 'RA---ccc' and 'DEC--ccc' where ccc is the projection code. These axes are required. \end{enumerate} % - list nest 1 Weights and flagging are handled the same as for visibility data. Sort order is the same as for visibility data except that the sort codes for sorting by u and v become: \begin{verbatim} U => ordered by RA V => ordered by Declination X => descending ABS (RA) Y => descending ABS (Declination) Z => ascending ABS (RA) M => ascending ABS (Declination) \end{verbatim} \subsubsection{Example Single Dish Data File Header } The following is a FITS header for a single dish data file containing 16 frequency channels and a single IF and using unprojected coordinated. The first two lines are not part of the FITS file. \index{single dish FITS format} \begin{verbatim} 000000000111111111122222222223333333333444444444455555555556666666666 123456789012345678901234567890123456789012345678901234567890123456789 SIMPLE = T / BITPIX = 16 / NAXIS = 6 / NAXIS1 = 0 /NO STANDARD IMAGE JUST GROUP NAXIS2 = 3 / NAXIS3 = 1 / NAXIS4 = 16 / NAXIS5 = 1 / NAXIS6 = 1 / EXTEND = T /TABLES FOLLOWING MAIN IMAGE BLOCKED = T /TAPE MAY BE BLOCKED OBJECT = 'ALL SKY ' /SOURCE NAME TELESCOP= '300 FT ' / INSTRUME= '6 CM ' / OBSERVER= 'PHANTOM ' / DATE-OBS= '02/12/86' /OBSERVATION START DATE DD/MM/YY DATE-MAP= '09/11/87' /DATE OF LAST PROCESSING DD/MM/YY BSCALE = 4.98494155534E-05 /REAL = TAPE * BSCALE + BZERO BZERO = 0.00000000000E+00 / BUNIT = 'JY ' /UNITS OF FLUX EPOCH = 1.950000000E+03 /EPOCH OF RA DEC BLANK = -32768 /TAPE VALUE OF BLANK PIXEL CTYPE2 = 'COMPLEX ' / CRVAL2 = 1.00000000000E+00 / CDELT2 = 1.000000000E+00 / CRPIX2 = 1.000000000E+00 / CROTA2 = 0.000000000E+00 / CTYPE3 = 'STOKES ' / CRVAL3 = -1.00000000000E+00 / CDELT3 = -1.000000000E+00 / CRPIX3 = 1.000000000E+00 / CROTA3 = 0.000000000E+00 / CTYPE4 = 'FREQ ' / CRVAL4 = 1.66499989984E+09 / CDELT4 = 5.000000000E+07 / CRPIX4 = 1.000000000E+00 / CROTA4 = 0.000000000E+00 / CTYPE5 = 'RA ' / CRVAL5 = 0.00000000000E+00 / CDELT5 = 1.000000000E+00 / CRPIX5 = 1.000000000E+00 / CROTA5 = 0.000000000E+00 / CTYPE6 = 'DEC ' / CRVAL6 = 0.00000000000E+00 / CDELT6 = 1.000000000E+00 / CRPIX6 = 1.000000000E+00 / CROTA6 = 0.000000000E+00 / GROUPS = T / GCOUNT = 14655. / PCOUNT = 5 / PTYPE1 = 'RA ' / RA in degrees PSCAL1 = 1.09890000000E-02 / PZERO1 = 0.00000000000E+00 / PTYPE2 = 'DEC ' / Declination in degrees PSCAL1 = 1.09890000000E-02 / PZERO2 = 0.00000000000E+00 / PTYPE3 = 'BEAM ' / Beam number PSCAL3 = 3.66412235782E-10 / PZERO3 = 0.00000000000E+00 / PTYPE4 = 'DATE ' / Date + time as Julian date (days) PSCAL4 = 2.50000000000E-01 / PZERO4 = 2.44603650000E+06 / PTYPE5 = 'DATE ' / PSCAL5 = 1.52587890600E-05 / PZERO5 = 0.00000000000E+00 / ORIGIN = 'AIPSNRAO CVAX 15APR87' / DATE = '13/11/87 ' / FILE WRITTEN ON DD/MM/YY HISTORY AIPS WTSCAL = 1.83759533423E+00 / CMPLX WTS=WTSCAL* (TAPE*BSCALE+BZERO) END \end{verbatim} \subsection{Extension Files } There is frequently auxiliary information associated with an image or data set which needs to be saved in the same tape file. Examples of this in AIPS are the Antenna files and CLEAN component files. There is currently a draft proposal to the IAU (Harten {\it et. al.} 1984) defining a standard format for the invention of extension files to be written after the main data records (if any) and defining a ``Tables'' type extension file. The Tables extension files will be able to carry information which can be expressed in the form of a table. AIPS also makes use of a 3-D Table extension which is similar to tables but allows arrays as table entries. The following section will describe the proposed standards which are being incorporated into AIPS. \index{FITS format} \index{FITS tables} \subsection{Standard Extension } The standard, generalized extension file is not a true tape file in the sense that it is separated by tape EOF marks, but is a number of records inside a FITS tape file which contains information of relevance to the file. Each standard extension ``file'' will have a header which is very similar to the main FITS header. This header consists of one or more 2880 8-bit byte ``logical'' records each containing 36 80-byte ``card images'' in the form: \begin{verbatim} keyword = value / comment \end{verbatim} The extension file header begins in the first record following the last record of main data (if any) or following the last record of the previous extension file. The format of the generalized extension ``file'' header is such that a given FITS reader can decide if it wants (or understands) a given extension file type and can skip over the extension file if the reader decides it doesn't. Most of the standards concerning data types and bit orders for the main FITS data records also apply to extension files. One difference is that 8-bit pixel values can be used to indicate ASCII code. The use of the generalized extension ``files'' requires the use of a single additional keyword in the main header: \begin{enumerate} % list nest 1 \item EXTEND (logical) if true (T) indicates that there {\it may} be extension files following the data records and, if there are, that they conform to the generalized extension file header standards. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The required keywords in an extension file header record are, in order: \begin{enumerate} % list nest 1 \item XTENSION (character) indicates the type of extension file, this must be the first keyword in the header. \item BITPIX (integer) gives the number of bits per ``pixel'' value. The types defined for the main data records plus 8-bit ASCII are allowed. \item NAXIS (integer) gives the number of ``axes''; a value of zero is allowed which indicates that no data records follow the header. \item NAXIS1 (integer) is the number of ``pixels'' along the first axis (if any). \item NAXISn (integer) is the number of ``pixels'' along the last axis. \item PCOUNT (integer) is the number of ``random'' parameters before each group. This is similar to the definition of random group main data records. The value may be zero. \item GCOUNT (integer) is the number of groups of data defined as for the random group main data records. If an image-like file (e.g., a table file) is being written this will be 1. \item END is always the last keyword in a header. The remainder of the record following the END keyword is blank filled. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} There are three optional standard keywords for extension file header records. The order, between the required keywords and the END keyword, is arbitrary. \begin{enumerate} % list nest 1 \item EXTNAME (character) can be used to give a name to the extension file to distinguish it from other similar files. The name may have a hierarchal structure giving its relation to other files (e.g., ``map1.cleancomp'') \item EXTVER (integer) is a version number which can be used with EXTNAME to identify a file. \item EXTLEVEL (integer) specifies the level of the extension file in a hierarchal structure. The default value for EXTLEVEL should be 1. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The number of bits in an extension file (excluding the header) should be given by the formula: \begin{verbatim} NBITS = BITPIX * GCOUNT * (PCOUNT + NAXIS1 * NAXIS2 * ... * NAXISn) \end{verbatim} The number of data records following the header record are then given by: \begin{verbatim} NRECORDS = INT ((NBITS + 23039) / 23040) \end{verbatim} It is important that the above formulas accurately predict the number of data records in an extension ``file'' so that readers can skip over these ``files''. The data begins in the first record following the last record of the header. Extreme caution must be exercised when inventing new types of extension files. In particular, duplication of types, or several types with the same function, must be avoided. This means that when a new extension file type is invented, it should be as general as possible so that it may be used for other similar problems. \subsection{Tables Extension } A very common type of extension file is one containing data that can be expressed in the form of a table. That is, a number of entries which are all identical in form. A general, self-defining table extension file type is proposed by Harten {\it et. al.} (1984). The following sections describe the proposed format. \index{FITS format} \index{FITS tables} The table extension file uses ASCII records to carry the tabular information. Each table entry will contain a fixed number of entries (although the number can vary between different extension files). For each entry is given (1) a label (optional), (2) the beginning column, (3) an undefined value (optional), (4) a Fortran format to decode the entry, (5) scaling and offset information (optional), and (6) the units (optional). \subsubsection{Tables Header Record } The keywords for tables extension file headers are given in the following: \begin{enumerate} % list nest 1 \item XTENSION (character) is required to be the first keyword and has a value 'TABLE ' for table extension files. \item BITPIX (integer) is a required keyword which must have a value of 8 indicating printable ASCII characters. \item NAXIS (integer) is a required keyword which must have a value of 2 for tables extension files. \item NAXIS1 (integer) is a required keyword which gives the number of characters in a table entry (i.e., ``row''). \item NAXIS2 (integer) is a required keyword which gives the number of entries in the table (i.e., number of rows). A value of 1 is allowed. \item PCOUNT (integer) is a required keyword which must have the value of 0 for tables extension files. \item GCOUNT (integer) is a required keyword which must have the value of 1 for tables extension files. \item TFIELDS (integer) is a required keyword which must follow the GCOUNT keyword. TFIELDS gives the number of fields in each table entry. \item EXTNAME (character) is the name of the table. \item EXTVER (integer) is the version number of the table. \item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is recommended. (optional) \item TBCOLnnn (integer) the pixel number of the first character in the nnn'th field. \item TFORMnnn (character) the Fortran format of field nnn (I,A,E,D) \item TTYPEnnnn (character) the label for field nnn. (optional, order arbitrary) \item TUNITnnn (character) the physical units of field nnn. (optional, order arbitrary) \item TSCALnnn (floating) the scale factor for field nnn. True\_value = tape\_value $\ast$ TSCAL + TZERO. Note: TSCALnnn and TZEROnnn are not relevant to A-format fields. Default value is 1.0 (optional, order arbitrary) \item TZEROnnn (floating) the offset for field nnn. (See TSCALnnn.) Default value is 0.0 (optional, order arbitrary) \item TNULLnnn (character) the (tape) value of an undefined value. Note: an exact left-justified match to the field width as specified by TFORMnnn is required. (optional, order arbitrary) \item AUTHOR (character) the name of the author or creator of the table. (optional, order arbitrary) \item REFERENC (character) the reference for the table. (optional, order arbitrary) \item END must always be the last keyword and the remainder of the record must be blank filled. \end{enumerate} % - list nest 1 \index{FITS format} \index{FITS tables} The TFORMnnn keywords should specify the width of the field and are of the form Iww, Aww, Eww.dd, or Dww.dd (integers, characters, single precision and double precision). If -0 is ever to be distinguished from +0 (e.g., degrees of declination), the sign field should be declared to be a separate character field. \subsubsection{Table Data Records } The table file data records begin with the next record following the last header record and each contains 2880 ASCII characters in the order defined by the header. Table entries do not necessarily begin at the beginning of a new record. The last record should be blank filled past the end of the valid data. \subsubsection{Example Table Header and Data } The first two lines of numbers are only present to show card columns and are not part of the extension file. \index{FITS format} \index{FITS tables} \begin{verbatim} 1 2 3 4 5 6 7 12345678901234567890123456789012345678901234567890123456789012345678901 XTENSION= 'TABLE ' / EXTENSION TYPE BITPIX = 8 / PRINTABLE ASCII CODES NAXIS = 2 / TABLE IS A MATRIX NAXIS1 = 60 / WIDTH OF TABLE IN CHARACTERS NAXIS2 = 449 / NUMBER OF ENTRIES IN TABLE PCOUNT = 0 / RANDOM PARAMETER COUNT GCOUNT = 1 / GROUP COUNT TFIELDS = 3 / NUMBER OF FIELDS IN EACH ROW EXTNAME = 'AIPS CC ' / AIPS CLEAN COMPONENTS EXTVER = 1 / VERSION NUMBER OF TABLE TBCOL1 = 1 / STARTING CHAR. POS. OF FIELD N TFORM1 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE1 = 'FLUX ' / TYPE (HEADING) OF FIELD N TUNIT1 = 'JY ' / PHYSICAL UNITS OF FIELD N TSCAL1 = 1.0 / SCALE FACTOR FOR FIELD N TZERO1 = 0.0 / ZERO POINT FOR FIELD N TBCOL2 = 17 / STARTING CHAR. POS. OF FIELD N TFORM2 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE2 = 'DELTAX ' / TYPE (HEADING) OF FIELD N TUNIT2 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N TSCAL2 = 1.0 / SCALE FACTOR FOR FIELD N TZERO2 = 0.0 / ZERO POINT FOR FIELD N TBCOL3 = 33 / STARTING CHAR. POS. OF FIELD N TFORM3 = 'E15.6 ' / FORTRAN FORMAT OF FIELD N TTYPE3 = 'DELTAY ' / TYPE (HEADING) OF FIELD N TUNIT3 = 'DEGREES ' / PHYSICAL UNITS OF FIELD N TSCAL3 = 1.0 / SCALE FACTOR FOR FIELD N TZERO3 = 0.0 / ZERO POINT FOR FIELD N END \end{verbatim} The rest of the header block is blank filled. The data cards start on the next block boundary. \begin{verbatim} 0.183387E+00 -0.138889E-03 0.694444E-04 0.146710E+00 -0.138889E-03 0.694444E-04 0.117368E+00 -0.138889E-03 0.694444E-04 0.938941E-01 -0.138889E-03 0.694444E-04 0.183387E+00 -0.138889E-03 0.694444E-04 . . . \end{verbatim} \subsection{3-D Tables Extension }\label{app:C3DBTF2} There are several types of extension files in AIPS which do not fit in a natural way into the FITS Tables format (e.g., gain and calibration tables). These files tend to be large, and conversion to and from ASCII can be expensive. A ``3-D'' extension to the FITS tables format, in which an entry can be a 1-dimensional array, gives the needed flexibility. Use of binary data, including IEEE floating point formats, allows efficient implementation of readers and writers. \index{FITS format} \index{FITS tables} Each row in a 3-D table contains a fixed number of entries (although the number can vary between different extension files). The header is a standard FITS extension header; for each table entry is given (1) the size and data type of the entry, (2) a label (optional), (3) the units (optional), and (4) an undefined value (optional). An entry may be omitted from the table, but still defined in the header, by using a zero repeat count in the TFORMn entry. \subsubsection{3-D Tables Header Record } The keywords for 3-D tables extension file headers are given in the following: \begin{enumerate} % list nest 1 \item XTENSION (character) is required to be the first keyword and has a value 'A3DTABLE' for table extension files. \item BITPIX (integer) is a required keyword which must have a value of 8. \item NAXIS (integer) is a required keyword which must have a value of 2 for 3-D Tables extension files. \item NAXIS1 (integer) is a required keyword which gives the number of 8-bit bytes in a table row. \item NAXIS2 (integer) is a required keyword which gives the number of rows in the table. A value of 1 is allowed. \item PCOUNT (integer) is a required keyword which must have the value of 0 for tables extension files. \item GCOUNT (integer) is a required keyword which must have the value of 1 for tables extension files. \item TFIELDS (integer) is a required keyword which must follow the GCOUNT keyword. TFIELDS gives the number of fields in each table entry. \item EXTNAME (character) is the name of the table. \item EXTVER (integer) is the version number of the table. \item EXTLEVEL (integer) is the hierarchal level number of the table, 1 is recommended. (optional) \item TFORMnnn (character) is the size and data type of field nnn. If repeat count is absent, it is assumed to be 1. A value of zero is allowed. (required, order arbitrary) \item TTYPEnnnn (character) the label for field nnn. (optional, order arbitrary) \item TUNITnnn (character) the physical units of field nnn. (optional, order arbitrary) \item TNULLnnn (character) the value of an undefined value. These are only for integer data types; for floating data types, the IEEE nan (not a number) values are used. \item AUTHOR (character) the name of the author or creator of the table. (optional, order arbitrary) \item REFERENC (character) the reference for the table. (optional, order arbitrary) \item END must always be the last keyword and the remainder of