From dishfits-request@fits.CX.NRAO.EDU Thu Nov 9 16:13:47 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA07816; Thu, 9 Nov 89 16:13:44 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA16547; Thu, 9 Nov 89 16:13:19 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA07775; Thu, 9 Nov 89 16:01:41 EST Return-Path: X-Vms-Mail-To: DISHFITS,BANANAS Message-Id: <891109155525.0000083CFF1@cvax.CV.NRAO.EDU> Status: RO From: EGREISEN@NRAO.EDU Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU, bananas@cv3.CX.NRAO.EDU Subject: Use of FITS for real-time systems Date: Thu, 9 Nov 89 15:55:25 EDT A SUGGESTION FOR USE OF FITS FOR REAL-TIME SYSTEMS Eric W. Greisen, NRAO November 9, 1989 The fundamental FITS formats have often been considered for use in real-time systems, but have generally been rejected. In some cases, FITS-like formats have been used, which incorporated some of the strengths of the FITS conventions, but which undermined the general interoperability of this IAU-supported data interchange format. During the meetings in Green Bank, convened to discuss radio single dish use of FITS, a scheme to support real-time systems with a standard FITS format became evident to me and I described it briefly. I am expanding that idea here to provide a document for discussion and wider distribution. FITS has many advantages for real-time systems. It is reasonably self documenting, which gives users of the output data who are divorced from the telescope itself by reasons of politics, distance, or time a reasonable chance to determine what is on the tape and what may be done to read and translate it. FITS allows for the association of diverse information through its character headers and through the use of special records, which may be of standard types or, if absolutely necessary, of types of the telescope programmers' own devising. The association of these collections of data is done by the position of tape end-of-file marks; all data between EOFs are associated. A new Fits-based real-time tape format may be read immediately or with only a few modifications by one or more of the generalized FITS readers which are now widely distributed around the globe. Blocked FITS tapes make very efficient use of the recording medium. Unfortunately, the original FITS formats have a "fatal" flaw. Both the basic format and the random groups format have the requirement that the main data matrix or collection of matrices must follow the main header immediately. Any special records, such as tables describing the source and instrumental parameters, must follow the main data. Further, the size of the main data must be fully specified in the character header. Thus, one is vulnerable to tape parity errors, telescope program and power failures, and the like. If these altogether too common problems arise during the writing of the main data, then the essential control tables are not written and the data that were recorded are essentially useless. Furthermore, the size of the main data is not necessarily known a priori. The exact number of samples may be determined by accidents of timing, on-line data editing, or the users' decision to terminate or extend the observation. Again, the problem of placing the critical control tables after the data causes serious difficulties. Therefore, I suggest that real-time systems use the binary tables extension to FITS suggested by Cotton, not only for the control tables, but also for the main data themselves. The description and structure of these real-time data tables should be determined by the ideas put forward by Wells and adopted for use in the radio single-dish case. The main header of a tape file would probably be fairly terse and be used mainly to specify that the main data matrix is null (NAXIS = 0). It could also provide a variety of documentation of the format conventions and general history of the particular observations. Then, all control tables would be written following the standard conventions for FITS extension tables. Presumably the binary tables version would be selected by the efficiency-conscious programmers of real-time systems. Other extension files would then be written, also following the rules of the generalized extensions of FITS. Finally, one would write the main data as a binary table. Each row would contain one sample of the data array together with any conformal arrays needed to describe those data (such as a matching weighting array) and any other time-varying parameters needed to describe those data (such as time, source ID, observation coordinates, system temperatures, flagging, or whatever). The use of the binary tables format allows all of these parameters to be in natural forms (i.e., character, integer, floating, double precision, bit array). On a computer that uses standard byte order and IEEE floating-point, the FITS writer would only need to copy the data from internal data arrays into the n times 2880-byte tape IO buffers. Less standard computers would have to do some translation of the bits, but that should not be too expensive. As given so far, this suggestion solves the problem of losing control information when the data-writing is interrupted. It also improves the flexibility of the binary format over that allowed in the basic and random-groups FITS formats, since in those all data must be of a single numeric binary type. However, the binary tables format still requires the specification of the file size in advance. Two conventions --- not rules --- need to be agreed upon to resolve the remaining difficulties. The first is the understanding that no important files are to be found after the main data table; that normally only the tape end-of-file will follow that table. Then, the corrective response to a premature end-of-file is obvious. One simply keeps all the data previously read, discarding (or marking as uncertain) only those tables rows which came from the last tape record prior to the unexpected EOF. The second understanding is that real-time systems will normally write data tables of modest size, so that the observation would occupy more than one table under many circumstances. Then, most data tables would be completely filled to the predicted size and the cost of filling the last such table with blank rows, rather than a premature EOF, should be modest. In this way, no "error" would be made in reading them. I suppose that the only argument against writing many of these data tables in a single tape file is the problem of parity errors. Since there is no accord between the internal logical structure of a data table and the physical blocking in FITS, all data following a parity error within the current table may be lost because of that parity error. There are recovery mechanisms that will work under many circumstances. Following a parity error, the FITS reader can check the first eight bytes of each logical record for the keyword XTENSION signalling the start of a new table. Systems which require still greater certainty of data integrity could write their data tables with synchronization columns at the beginning of each row, namely a column with characters such as the value assigned EXTNAME and a column with the row number. Following a parity error, the reader would search each logical record for this pattern and recover its reading of the data table knowing how many rows had been lost. In general, some compromise between writing the headers and control files only once and the danger of parity or other device failures should dictate the size of the individual tables and the number written in a single tape file. Advances in recording technology, such as the new magnetic and optical cartridge devices, will probably reduce this problem to a minimum. I note that random-group FITS has seldom been used outside of the radio-interferometric community, and that the ideas above provide a better solution for that community than the random-groups format. I suggest that we switch to that format in radio interferometry both for real-time and post-observation systems. Needless to say, AIPS would have to support both forms forever including being able to write random groups as well as tables. Furthermore, I suspect that AIPS will still treat the uv data as random groups internally, since the differentiation in the internal forms would be small. From dwells@fits.CX.NRAO.EDU Tue Nov 14 16:29:39 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA16479; Tue, 14 Nov 89 16:29:36 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA00237; Tue, 14 Nov 89 16:28:57 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA16364; Tue, 14 Nov 89 15:50:55 EST Return-Path: Message-Id: <8911142050.AA16355@fits.cx.nrao.edu> Status: RO From: dwells@fits.CX.NRAO.EDU (Don Wells) Sender: dishfits-request@fits.CX.NRAO.EDU To: rww@hoh-2.att.com Cc: astro@stsci.edu, bananas@nrao.edu, dishfits@fits.CX.NRAO.EDU Subject: Re: 3-D Binary Tables Date: Tue, 14 Nov 89 15:50:26 EST Dear Friends of FITS, Bob Wilson of Bell Labs, who did not attend the recent Green Bank FITS workshop (30 Oct - 01 Nov), wrote to me this morning asking what we had agreed to, because he is ready to re-write their data acquisition program. Here is my answer to him. Other people who did not attend may also find this review of the meeting to be of some use, so I am broadcasting it. Please note that this is my own very personal view of what happened; it is surely biassed and/or incomplete. Other participants are free to broadcast their versions of course. Because this workshop, which was convened to agree on conventions for single dish radio astronomy, in fact proposed a dramatic new set of architectural conventions for all of FITS, conventions of great generality, I have decided to broadcast this summary to the "ASTRO" and "BANANAS" exploders, as well as to the "DISHFITS" exploder. Many people are on more than one of these distribution lists and will get multiple copies. Sorry about that... = = = Dear Bob, The big appendix of my draft paper of a month or so ago, 65 KB long, contains the text from chapter 14 of "Going AIPS", the AIPS programmer's reference manual, which includes the description of Bill Cotton's "3-D Binary Tables" extension. Probably you have a copy of Going AIPS around, and can copy it. Or else you probably got my big paper (if not I can re-send it) and can print the appendix from it. This is currently the only description of 3-D binary tables. It is also completely sufficient as a formal description. Because everyone has endorsed Bill's design we know that he will revise and extend that Going AIPS text into a formal paper which will be the description that the FITS committees are now guaranteed to endorse. Also, we know that nothing in that description will be changed in any significant way. One or two new data types will probably be added; the candidates discussed so far are 8-bit unsigned integers and perhaps single precision complex. Its name will be changed from 'A3DTABLE' to XTENSION='3DTABLE'. The principal complaint against Bill's design was that it only allowed 1-D vectors in the fields; that hard fixed limit made many of the designers uneasy. My draft paper said that the spectrum dimension should be in a separate column of the table. When Bob Hanisch challenged me on this point my answer on DISHFITS was that I wanted columns called NAXIS and NAXIS1 to define the dimension and "trusted that the generalization was obvious". I did this because I wasn't sure how ready the spectroscopy community was to accept the idea that I really wanted to propose a design that was far more general than just 1-D spectroscopy. By the time of the meeting I judged that I could get away with it, and so in my verbal presentation I exposed the underlying concept, which was that all of the architecture of Basic FITS N-dimensional matrix technology could be mapped onto rows of binary tables. I.e., that each row could be a virtual Basic FITS file (required keywords plus optional keywords plus N-dimensional binary matrix). This idea was accepted fully by the workshop. Note that this is just a convention about how to use Bill's primitive binary design and so his paper and the paper that will ultimately document the convention are independent. Also, the optional keywords and value conventions peculiar to radio spectroscopy must be agreed to by the discipline specific group and appear as yet another independent paper. Eric's recent DISHFITS/BANANAS discussion of real-time FITS via binary tables is also just a convention which is independent of the three papers mentioned above. Aside from any technical merit that my proposal may have (the 10-year success history of Basic FITS suggests plenty of merit) the real reason I proposed this architecture was that it eliminates almost all options on conventions for keywords and values, because it means that the proposed architecture automatically inherits all of the prior FITS agreements and R-and-D work *intact*. Thus it vastly simplified the negotiations, which were easily completed by the end of the second day (except for a number of details which the editors of the drafts will handle). It also means that implementors have almost nothing new to learn, that they already have an understood language for describing any new implementations that use the convention. I considered these advantages to be compelling and apparently others agreed. A major objection to my proposal was that real data acquisition systems often need auxiliary arrays to accompany the main array, and many designers wanted these to be in the same row. Sure, Bill's binary tables can handle this problem trivially, but that is just saying that anarchy is always an implementable design. Negotiations were in difficulty on this point for some hours until Eric Greisen suddenly proposed a simple compromise which saved my proposal. Eric agreed that we could only afford to have one set of columns to document the matrix structure (Preben Grosbol's alternative proposal for multi-valued NAXISn was considered briefly by the workshop, even though Preben himself labelled it "disgusting" and said he wouldn't vote for it; I myself thought it was an ingenious negotiating ploy that stimulated the compromise.) Eric proposed that multiple matricies be allowed with the understanding that they have the same dimensionality as the primary matrix even if not necessarily the same data type (the BITPIX keyword of Basic FITS is redundant in my proposal because the 3-D binary table syntax does the job). This proposal was accepted. It appears that only one new keyword is needed to label the matricies. A recent DISHFITS message from Jodrell proposed the obvious convention that the first matrix is the primary one. An example of an auxiliary matrix would be IR detectors that average multiple short integrations and have an RMS matrix. Clearly any such row is equivalent to two or more Basic FITS files with almost identical headers. Therefore it is still consistent with the architectural concept that I proposed. It is clear that the architecture implies that a generalized filter could be built that would combine a set of Basic FITS files into a table extension or would decompose the table extension back into the files. The net effect of all this is that we have agreed on a set of conventions built on top of the new 3-d binary table extension type which allows us to write sets of astronomical observations *much* more efficiently than before because the repeated header data is now in binary form. This is even more true becuase of my suggestion that keyword=value pairs that are constant through a set of data (i.e., constant value in a column of the table) should be pulled out into the header for the table as keyword=value (i.e., a virtual column convention). The total result is a nearly optimum design, better than I and others had anticipated when we began this design exercise. By combining the ideas of a whole team of people this new architecture merges most of the prior history of Basic FITS, Random Groups and Tables into one comprehensive design that can do spectroscopy plus almost every other astronomy application that we currently know about. Eric recently proposed that we consider using this design instead of Random Groups for interferometer data. Before the GB meeting any such suggestion would have been regarded as heresy, as anathema, by all real FITS people. Now it seems reasonable. The new design is particularly important for spectroscopic applications, for which FITS has always had an efficiency problem. Although sets of ordinary 2-D images can be encoded in the new design it would be a *serious* mistake for everyone to rush to do so, at least until that packing/unpacking filter is written by someone! The reason, of course, is that for ten years we have had effective interchange of binary imagery in astronomy using Basic FITS, and it would be irresponsible to undermine this solid success. Likewise, although ASCII tables can be encoded several times more efficiently in binary form, it would be a mistake to do this for catalogs of observations in which the efficiency is more than compensated by the greater degree of interoperability given by XTENSION='TABLE'. (Obviously another generalized filter could be built to convert ASCII tables to binary and vice versa; some student should take this on as an exercise, hint, hint...) = = = The Grosbol/Wells hierarchical name convention proposed in another appendix of my DISHFITS paper (and which was discussed at the 1989 FITS meetings) is currently favored, but some opposition still exists. The FITS committee people agreed to resolve this disagreement within the next month or so. There was a consensus at the Green Bank Workshop that astronomy really should use hierarchical keywords even though they are not absolutely required. Note that the new data architecture does not depend in any critical way on the hierarchical convention, if any, that is ultimately endorsed. Drafts of papers will soon appear on DISHFITS etc. with more details, especially radio spectroscopy details, and will surely circulate widely within the astronomical community. The high energy astronomy community appear now to have a consensus favoring the binary tables for interchange of lists of descriptions of photon events. Thus the 3-D binary extension proposal will have the backing of the Very Long Baseline Array (for which it was originally developed), the radio single dish (spectroscopy) community and the high energy community. The FITS committee chairmen intend to recommend 3-D binary tables to the spring 1990 European and North American FITS meetings for endorsement because of the consensus that now exists favoring this design. = = = You ask: "Is there any chance that NRAO will publish a 'suggested' single dish FITS reader in code, pseudo-code, or flow chart form to guide the naive implementer?" This must be number one on the all-time list of frequently-asked-questions regarding FITS. The standard answer is "maybe, but don't hold your breath while waiting for it." The reason is that the problem is not well-posed. The purpose of a reader is to translate from the language of FITS to the language of a *PARTICULAR* data analysis system. Your question is like asking "is there any chance of devising a translator from English to other languages?" If you ask for an English-to-French translator you at least have a well-posed problem. Likewise, if you ask for a single-dish FITS reader for AIPS or MIDAS or IRAF or ... you will have a well-posed problem -- in each specific case. For AIPS and MIDAS the existing FITS readers can already read 3-D binary tables into their tables systems. Currently only AIPS supports vectors in the tables, but MIDAS will add them during 1990. Because of the interest among the high energy community it seems plausible that IRAF will acquire basic binary table capability in its reader and writer during 1990, but I don't know the details of those plans. Once the data are in a generalized table system on disk their interpretation according to the agreed conventions can be coded into application programs that read the fields of the tables. The virtual columns will be stored as auxiliary files of keyword=value pairs in all of the large data analysis systems, and those files can be searched by library routines called by the spectroscopic application programs. A new answer to your question is that NASA has issued a contract to produce a FITS "User's Guide" (an Implementors Handbook of Recommended Practices). The work is being done by Barry Geldzahler. The first draft of this document is well advanced and will probably be circulated during 1990. I don't expect that it will contain the sort of pseudo-program that you asked for, but it will contain datatype translation routines for specific architectures, a variety of hints, do-s and don't-s, etc. The new FITS Support Office at GSFC is working on yet another relevant project, the codification of the FITS papers into a single canonical "FITS Standard" document. Barry Schlesinger is doing this work, which will also circulate in draft form during 1990. As for writing FITS, it has always been true that it is easy to produce a program to write almost any dataset out into FITS. Generalized readers are the hard problem. = = = A Sign-Of-The-Times: Many of the discussions during the Workshop were conducted in the language of relational database systems. Relational theory arguments were invoked by several participants as reasons for or against various proposed design features. The proposed architecture of Basic-FITS-files-in-Binary-Table-rows, even the binary vectors, maps directly onto many of the current state-of-the-art RDBMS systems. It is not inconceivable that in a few years one could decide to produce a generalized tool to read the table extensions into relations in a database and then code application programs to manipulate the relations. This may be a good path to a new generation of data analysis systems for astronomy. We shall see... Best regards, Don Donald C. Wells, Associate Scientist | NSFnet: dwells@nrao.edu [192.33.115.2] National Radio Astronomy Observatory | SPAN: NRAO::DWELLS [6654::] Edgemont Road | BITnet: DWELLS@NRAO Charlottesville, VA 22903-2475 USA | UUCP: ...!uunet!nrao.edu!dwells +1-804-296-0277 (38:02.2N/78:31.1W) | TWX=510-587-5482, Fax=+1-804-296-0278 From dishfits-request@fits.CX.NRAO.EDU Fri Nov 10 13:30:41 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA09346; Fri, 10 Nov 89 13:30:39 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA19018; Fri, 10 Nov 89 13:30:50 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA09343; Fri, 10 Nov 89 13:25:37 EST Return-Path: <"19463::JBVAD::PAH"@cvax.CV.NRAO.EDU> X-Vms-Mail-To: STADAT::NRAO::DISHFITS Message-Id: <891110130546.00001583031@cvax.CV.NRAO.EDU> Status: RO From: Paul Harrison <"19463::JBVAD::PAH"@NRAO.EDU> Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Some quick suggestions... Date: Fri, 10 Nov 89 13:05:46 EDT As a result of trying to make a header for a single dish FITS format file I have the following suggestions 1. Make AZIM, ELEV, ZENI the standard names for azimuth elevation and zenith angle respectively. We decided on AZIMUTH and ELEVATIO in the meeting but these names do not allow axes to be labeled in the standard fashion including a projection type. 2. Have the convention that the first matrix is the `main' data matrix when there is more than one matrix. This is obviously easy to decide if the second data matrix is ERRORS or QUALITY, or some such type, and if we keep to the convention that different backends/frontends etc. go on different rows it should always be easy. Incidentally it might be a good idea to formalize the names ERRORS and QUALITY. 3. The question of irregularly sampled axes was `finessed' at the workshop. But after a little thought I have a solution that seems reasonably easy to implement within the 3d binary tables concept. There is nothing in the format that stops CRVALn from being an array of dimension MAXISn, in which case the array can contain the explicit coordinate value of each pixel. This case could be flagged by setting CDELTn to 0, and CRPIXn having a mandatory value of 1. That is all for now, although I think I will probably come up with some more before I am through! Paul Harrison (Jodrell Bank) From dishfits-request@fits.CX.NRAO.EDU Tue Nov 7 16:01:37 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA03669; Tue, 7 Nov 89 16:01:34 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA10507; Tue, 7 Nov 89 16:01:32 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA03644; Tue, 7 Nov 89 15:48:52 EST Return-Path: <"TUCVAX::PMURPHY"@cvax.CV.NRAO.EDU> X-Vms-Mail-To: DISHFITS,PMURPHY Message-Id: <891107154644.00000C21061@cvax.CV.NRAO.EDU> Status: RO From: Pat Murphy <"TUCVAX::PMURPHY"@NRAO.EDU> Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: IEEE-VAX floating conversion Date: Tue, 7 Nov 89 15:46:44 EDT With the near-certain imminent approval of the IEEE floating point format for binary FITS data by the IAU, many observatories will want to implement FITS writers and/or readers on VAX/VMS or VAX/UNIX machines that can read IEEE floating point numbers and convert them to native VAX format, and vice versa. While it is possible to just multiply/divide by 4.0, this practice is NOT general enough to handle all cases. In particular, the IEEE format includes extensions with no counterpart in the VAX Format such as plus or minus infinity and not-a-number (NaN). Even worse, it allows negative zero which causes a fatal reserved operand fault on a VAX. I have written a pair of routines that will convert one or more IEEE floating point numbers to native VAX format and vice versa. The IEEE-to-VAX routine detect all the above cases and produce a sensible result without causing any fatal errors. They have been tested for normal data, plus or minus "not-a-number", and plus or minus infinity. They currently write an indefinite value (defined in the source as 1.6E38) in place of these IEEE exceptional numbers. The source code (in C) for these functions is appended to this message; details of calling sequences are given in the precursor comments. The modules have been tested in both VAX C (V2.4) and GNU C (V1.22) and appear to work in either, although I could not get the INDEF constant in Gnu C to set correctly (this can be fixed by using a regular variable instead). One nice feature of these modules is that they incorporate word and/or byte swapping, invoked by flags set in the calling sequence. Depending on how you got your data onto a VAX, you may have to swap bytes and/or (16-bit) words. See the comments for details, but if in doubt, write some test data and try the various combinations before committing yourself. I can't promise to support this routine or vouch that it's error-free; however, my testing indicates that it seems to work fine for my test data. If you do find a geniune error, however, please let me know. - Pat Murphy ________________ ==================================| 12-m Radio |============================== | Patrick P. Murphy / Telescope on | Internet: PMurphy@nrao.edu | | Scientific Programming Analyst | Kitt Peak, AZ | Bitnet: PMurphy@NRAO | | Nat'l Radio Astronomy Obsvty. \ | Span/Hep: 6654::PMurphy | | 949 N. Cherry, Campus Bldg. 65 | | UU: uunet!nrao.edu!pmurphy | | Tucson, AZ 85721-0655 / Tucson | Phone: (602) 882-8250 | =================================/ * |============================== |______ ^12m/KP | \--------- 8<8<8<8<8<8<8<8<----------- cut here ------------>8>8>8>8>8>8>8>8 /* IEEEFLT -- convert IEEE floating-point format to or from DEC/VAX format * * =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= * Copyright ) 1989 by Patrick P. Murphy and the National Radio Astronomy * Observatory. This software is freely distributable and may be copied * with only the following restrictions: * * 1. This copyright notice must appear EXACTLY as it appears here * in all copies; * 2. The software CANNOT be sold for commercial gain, and the only * charge made in distributing is restricted to a reasonable fee * (if any) to cover handling and materials; * 3. If object code (e.g. object files or object modules in a library) * based on any of these modules is copied or distributed, the * source code with this notice must accompany this object code. * * This software is made available AS IS with NO EXPRESS OR IMPLIED WARRANTY * of any kind. While Patrick P. Murphy and the National Radio Astronomy * Observatory cannot provide any support for this software, we would * like to hear of any errors in the code. If you find any errors, please * report them to: Patrick P. Murphy, NRAO, 949 N. Cherry Avenue, Campus * Building 65, Tucson, AZ 85721-0655, USA; or send e-mail to any of these * addresses: pmurphy@nrao.edu, pmurphy@nrao.bitnet, nrao::pmurphy (on * SPAN or HEPNET, nrao=6654), or ...uunet!nrao.edu!pmurphy. * =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= * * There are two functions in this module: ieee2vax and vax2ieee for single * precision. They are capable of converting a 32-bit ANSI/IEEE-754 floating- * point number or numbers to standard VAX 32-bit F-floating format, and vice * versa. * * These routines are intended for use on a VAX that has imported IEEE data. * It is NOT yet general enough for use anywhere. In particular, on big- * endian machines (like Suns), the order of the mantissa, exponent, * and sign in the union structure should be reversed. * * The calling sequence is: * * (void) ieee2vax (data, &nreals, &wswap, &bswap) * float *data; * int *nreals, *wswap, *bswap; * * or, from fortran: (*** always use variables, NOT constants or parameters!) * * CALL IEEE2VAX (DATA, NREALS, WSWAP, BSWAP) * REAL DATA(*) * INTEGER NREALS, WSWAP, BSWAP * * and the calling sequence for vax2ieee is identical. The conversion is * done IN PLACE. * * This module includes optional byte/word swapping. A VAX word looks like: * * --------------------------------------------- * | s | exponent | mantissa | :A (byte address) * |-------------------------------------------| * | mantissa | :A+2 * --------------------------------------------- * * and values read in, e.g. from a National Instruments GPIB bus will in * general need to be word-swapped. However, data read from a byte stream, * e.g. data read in image mode across a TCP/IP network FTP link, will not * have to be word swapped but will need byte-swapped. Same for tape data. * * I have not tested the conversion of denormalized numbers. *---------------------------------------------------------------------- * Created May 1989 by Pat Murphy, NRAO/Tucson * Modified Oct 1989 by Pat Murphy: do it right, check for NaN, Inf, etc. * Modified Oct 1989 by Pat Murphy: Add VAX2IEEE as well. * Modified Nov 1989 by Pat Murphy: prevent reserved operand fault (use * integer instead of real) */ #define INDEF 1.6e38 /* or whatever you want it to be */ union fstruc { unsigned long int longword; short int words[2]; /* for swapping */ char bytes[4]; /* ditto */ float flt; /* raw number */ struct { /* disassemble floating # */ unsigned int mant1:7; /* top part of fraction */ unsigned int exponent:8; /* Exponent with bias */ unsigned int sign:1; /* sign bit */ unsigned int mant2:16; /* other part of fraction */ } f; } ; void ieee2vax (buf, nreals, wswap, bswap) unsigned long int *buf; int *nreals, *wswap, *bswap; /* pointers so Fortran calling possible */ { float temp; int i; short int itemp; /* for swapping words */ char ctemp; /* for swapping bytes */ union fstruc ieee; /* ================================================ */ if (*nreals < 1) return; for (i=0; i<*nreals; i++) { ieee.longword = buf[i]; /* Use int (flt can cause fault) */ if (*bswap == 1) { ctemp = ieee.bytes[0]; /* byte swap (4 bytes) */ ieee.bytes[0] = ieee.bytes[1]; ieee.bytes[1] = ctemp; ctemp = ieee.bytes[2]; ieee.bytes[2] = ieee.bytes[3]; ieee.bytes[3] = ctemp; } if (*wswap == 1) { itemp = ieee.words[0]; /* word swap */ ieee.words[0] = ieee.words[1]; ieee.words[1] = itemp; } /* let's check for NaN, Infinity, negative zero... */ if (ieee.f.exponent == 255) /* NaN or Infinity. */ ieee.flt = INDEF; /* don't care about sign */ else if (ieee.f.exponent == 0) /* Zap negative zero, causes */ if ((ieee.f.mant1 == 0) && /* reserved operand fault on */ (ieee.f.mant2 == 0)) /* a VAX (i.e. disregard */ ieee.flt = 0.0; /* sign on +/- zero) */ else /* Denormalized IEEE number */ ieee.flt = ieee.flt * 2; /* has exponent of -126 (??) */ else { /* Regular number */ if (ieee.f.exponent > 254) /* make sure exponent won't */ ieee.flt = INDEF; /* overflow */ else { /* add 2 to exponent */ ieee.flt = ieee.flt * 4; /* (VAX has different bias) */ } } buf[i] = ieee.longword; /* put VAX value back */ } } void vax2ieee(buf, nreals, wswap, bswap) float *buf; int *nreals, *wswap, *bswap; /* pointers so Fortran calling possible */ { float temp; int i; short int itemp; /* for swapping words */ char ctemp; /* for swapping bytes */ union fstruc vaxf; /* ================================================ */ if (*nreals < 1) return; for (i=0; i<*nreals; i++) { vaxf.flt = buf[i]; if (vaxf.f.exponent < 3) /* prevent underflow */ temp = 0.0; else temp = (vaxf.flt / 4.0); /* shift exponent by 2 */ if (*bswap == 1) { /* byte swap needed */ ctemp = vaxf.bytes[0]; vaxf.bytes[0] = vaxf.bytes[1]; vaxf.bytes[1] = ctemp; ctemp = vaxf.bytes[2]; vaxf.bytes[2] = vaxf.bytes[3]; vaxf.bytes[3] = ctemp; } if (*wswap == 1) { /* word swap needed */ itemp = vaxf.words[0]; vaxf.words[0] = vaxf.words[1]; vaxf.words[1] = itemp; } buf[i] = temp; /* put IEEE value back */ } } From dishfits-request@fits.CX.NRAO.EDU Tue Nov 28 06:25:58 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA10881; Tue, 28 Nov 89 06:25:56 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA06723; Tue, 28 Nov 89 06:25:58 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA10871; Tue, 28 Nov 89 06:18:41 EST Return-Path: Message-Id: <8911280845.AA06714@oso.chalmers.se> Status: RO From: Michael Olberg Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@nrao.edu Subject: hierarchical keywords Date: Tue, 28 Nov 89 09:45:13 -0100 28-Nov-89 I'd like to comment on Bob Hanish's summary of the meeting at ST ScI where he reports on the critical attitude of the attendees towards a hierarchical keyword scheme in FITS which would hide all but the globally agreed upon keywords from a 'classical' FITS reader program. At the Greenbank meeting I was myself in favour of this scheme compared to the Bob's BEGIN/END proposal which would leave all low hierarchy keywords accessible to the present FITS parsers, although they may not be able to deduce the full meaning of a keyword without the help of a human interpreter. The implementation of a keyword hierarchy by simple BEGIN/END grouping can in my opinion in fact be implemented in FITS without changing anything, i.e. without the BEGIN/ENDing of keywords. It might be a quick solution to the hierarchy problem while we are waiting for the gurus to come up with some- thing more final and sophisticated. The whole idea was triggered by a comment from Preben at the Greenbank meeting, where he pointed out that there is in general no meaning attached to the sequence in which FITS keywords appear on tape (except for a very few). I propose that we agree on such a meaning, namely that certain keywords define the domain of lower hierarchy keywords that come afterwards(!). The keyword TELESCOP for instance would have to appear before all keywords that we thought should be site specific. Equally, the keyword INSTRUME could trigger some extra meaning for the instrument specific keywords following after. It is conceivable that we find on the same tape from a given site two scans which were taken with two different spectrometers, where the spectrometer charac- teristics are given with the same set of keywords, although the interpretation of them should be slightly different, because they were preceeded by different INSTRUME cards. Of course the scope of keywords like TELESCOP and INSTRUME would always be until the apearance of another keyword like that or the END card. If one introduced a keyword like DISCIPLN or BRANCH or whatever you like, you would already have a scheme which was at least as powerful as the Morgan/Stobie idea of SNGLDISH GENERAL and SNGLDISH NRAO-12M etc. Michael Olberg Onsala Space Observatory From dishfits-request@fits.CX.NRAO.EDU Tue Nov 28 10:14:59 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11204; Tue, 28 Nov 89 10:14:57 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA07110; Tue, 28 Nov 89 10:10:30 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA11194; Tue, 28 Nov 89 10:04:52 EST Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891128095325.00000940031@cvax.CV.NRAO.EDU> Status: RO From: "FORVEILL@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Prototype implementation Date: Tue, 28 Nov 89 09:53:25 EDT Date sent: 28-NOV-1989 13:26:54.63 To: DISHFITS@NRAO Dear colleagues Althought I have been silent on this list since the Green Bank meeting, I have not been inactive, and I now have a working prototype implementation of our agreement. The present version has limited functionalities, in that the writer only writes as table columns the CDELTi (which as discussed a the meeting can be used to code position offsets) and (of course) the data matrix. This is already sufficient to store spectra obtained at random offsets around a reference position. The reader is a bit more general and supports all matrix description keywords (CRPIXi, CRVALi, CROTAi, MAXISi, AXTYPi). Extensions are of course trivial, and I may add a few other important keywords one day. I will now probably defer any major work till I receive sample files from Harvey Lizst for a cross check. In the mean time, I would like to discuss my experience with the new extension, make a few comments on the draft agreement and ask for opinions on a few questions. My dominant feeling is that the design is sound and can be implemented reasonably easily. Program complexity is obviously increased relative to the basic FITS agreement, but the large efficiency gain is worth the additionnal effort. I would like EXTNAME to become a required keyword for 3D tables. 3D tables are general enough that I would like to check, before I even attempt to, that I am not trying to decode a Clean Component table, or ROSAT photon lists. We should probably require MAXIS to be in the extension header and not in a column. If some people really want to store matrix with variable number of axes they can define extra dummy axes. That would be poor style anyway. This limitation eases implementation, since axis work space can then be allocated while reading the header (or checked for overflow in static implementations). My impression is that the TUNITi keywords are redundant with the requirement that all quantities in a FITS file be in SI units. There is the risk that they encourage the use of non standard units, and I would therefore advocate their exclusion from the 3D table proposal. The 3D draft specifically requests that GCOUNT=1. One could imagine allowing larger values of GCOUNT, thus providing an additional hierarchy in the file. Should that be allowed, explicitely forbidden, or reserved for future use? Some comments on the message of Paul Harrison 1) I have no objections to AZIM and ELEV. Note however that these keywords don't refer to matrix axes. They are basically atmospheric (+pointing) parameters and I see no reason why they would be projected. Horizontal coordinates are of course perfectly legal for the matrix, but they would then be stored as CTYPEi (with a value of 'AZIM-GLS', or anything like that). 3) I strongly oppose using CRVALn arrays to store non regular coordinates. CRVALi has a well defined meaning in the basic FITS paper and this is NOT an array. Althought the proposed modification only affects 3D tables, it breaks the correspondence principle and general unpackers become unpractical. Thierry From dishfits-request@fits.CX.NRAO.EDU Thu Nov 30 10:39:36 1989 Received: from NRAO.EDU (cv3.cv.nrao.EDU) by fits.cx.nrao.edu (4.0/SMI-DDN) id AA15634; Thu, 30 Nov 89 10:39:34 EST Received: from fits.cx.nrao.edu by NRAO.EDU (3.2/DCW-2n ) id AA12722; Thu, 30 Nov 89 10:39:46 EST Received: by fits.cx.nrao.edu (4.0/SMI-DDN) id AA15627; Thu, 30 Nov 89 10:32:38 EST Return-Path: X-Vms-Mail-To: DISHFITS AT NRAO Message-Id: <891130101718.00002132031@cvax.CV.NRAO.EDU> Status: RO From: "FORVEILL@FRGAG51" Sender: dishfits-request@fits.CX.NRAO.EDU To: dishfits@fits.CX.NRAO.EDU Subject: Structured keywords Date: Thu, 30 Nov 89 10:17:18 EDT Date sent: 30-NOV-1989 15:51:14.09 To: DISHFITS@NRAO A few comments on the message of Bob Hanisch I think we do need structured keywords soon. I agree with Bob that the need is not urgent for transport between different reduction systems. The main reason is that the core keywords (as defined in the original paper) are more or less sufficient for reduced data, and that nothing else is really interpreted in most such cases. However, FITS is no longer used only for transport. We are porting CLASS to Unix and intend to use FITS to convert the data between incompatible hardwares. AIPS uses FITS to transport their internal data structures between major versions of the data format. FITS is used to constitute archives that will remain readable forever. Some people plan to use it as an internal format, which makes good sense on IEEE hardware. In all these cases, one wants to transmit more than a mere data matrix, and structured keywords are then necessary. If no general agreement can be reached in the short term (say within 6 monthes), we should aim for an agreement within the radio community, where there seems to exist a consensus for some form of keyword hierarchy. If we cannot agree, even within a limited community, we will proceed on our own and implement the HIERARCH scheme. I am fully aware of the importance of universaly accepted standards, but endless discussions are not acceptable when the need is getting urgent. The consequences for non conformant readers will be minimal: they will ignore this intrinsically private information. AIPS has been doing that for a decade using HISTORY and nobody ever complained about it. Thierry