From dwells@azalea.cx.nrao.edu Sun Mar 29 00:53:28 1992 Newsgroups: sci.astro.fits From: dwells@azalea.cx.nrao.edu (Don Wells) Subject: Happy Birthday, FITS! [LONG] Organization: National Radio Astronomy Observatory, Charlottesville, VA. Distribution: sci Date: Sun, 29 Mar 1992 05:48:47 GMT Happy Birthday, FITS! I consider March 28th to be the birthday for FITS, because it was on March 28, 1979, that Eric Greisen and I completed the Basic FITS Agreement. FITS is now a teenager -- today is its 13th birthday. ---*--- A file folder which I brought from Arizona to Virginia in 1981 is labelled "Image Interchange Tape Format notes, Ron Harten (Leiden) correspondence". That folder contains a number of documents relevant to the early history of FITS. I reproduce many of them below as a birthday present for FITS. --- Magic Numbers --- The folder contains a memo by me dated 27 December 1976 and titled "Word and Pixel Combinations for the proposed Universal Interchange Format". The purpose of this memo was to document my thinking soon after Ron Harten (NFRA [now at RCA Astroelectronics, Princeton, NJ]) had visited me at KPNO in November 1976, and had suggested that our community needed to have an interchange format. Ron told me that he had used a magic record size to move Westerbork data between IBM and CDC computers in the Netherlands; my memory is that it was 720 bytes. The memo contains the following tables and text: # words per integral # pixels per integral # bits prime # words # of pixels # of words per word factors per packet 8-bit 16-bit 32-bit 8-bit 16-bit 32-bit -------- --------- ---------- ----- ------ ------ ------ ------ ------ 6 2.3 240 8 2.2.2 180 12 2.2.3 120 2 4 8 3 3 3 16 2.2.2.2 90 1 1 2 2 1 1 18 2.3.3 80 4 8 16 9 9 9 24 2.2.2.3 60 1 2 4 3 3 3 32 2.2.2.2.2 45 1 1 1 4 2 1 36 2.2.3.3 40 2 4 8 9 9 9 48 2.2.2.2.3 30 1 1 2 6 3 3 60 2.2.3.5 24 2 4 8 15 15 15 "The adopted packet size is 1440 bits, which is the least common multiple of the word sizes: 2.2.2.2.2.3.3.5, or 32.9.5. An agreed maximum number of packets per physical record must be decided. One suggested number is 21 packets, which is 30240 bits, and is less than the CDC 6000/7000-series tape physical record size limit of 30720 bits. For this size the tape packing factor is about 89% for both 7 and 9-track tapes at 800 bpi, which is certainly satisfactory. At 1600 bpi the packing factor falls to about 30% but we need to make some compromises here for the convenience of small computers. Anyway 800 bpi tape should probably be the recommended standard as they are the most common density." # bits per word Existing Computers with this word size -------- -------------------------------------- 12 PDP-8 16 numerous minicomputers 18 PDP-9 24 Datacraft 6024 (now Harris/6) 32 IBM 360/370-series, Interdata 7/32, 8/32 36 DEC-10, Univac 1100-series (ones-complement!) 48 CDC 3000-series 60 CDC 6000/7000-series and Cyber 170-series [Note that VAXen are not mentioned for 32 bits -- the VAX-11/780 was not announced until October 1977, ten months after this memo] --- Two Prototypes --- Ron Harten and I both built prototype formats during 1977, and exchanged data tapes. John Dickel (UIUC) travelled back and forth between Westerbork and KPNO during 1977 and 1978 carrying radio and optical imagery of supernova remnants encoded in the two formats, in order to carry out his projects to register the imagery. My folder contains letters from Ron describing his format, and letters from me offering technical criticism, and responses from Ron point by point. Ron's prototype included support for world coordinate systems and for machine-independent wide dynamic range ancillary data (he used logarithms encoded in binary integers). The folder also contains a copy of IPPS Technical Memo No. 3, titled "Structure of the image interchange tapes generated by command WIT16 in the MAGTAPE menu of the IPPS", the memo that documented my prototype. IPPS is "Interactive Image Processing System", and WIT16 is "Write Interchange Tape 16" [bits]. WIT16 had a logical record size of 1440 bits and the headers were in binary. The typical blocking factor was 21, 30240 bits, 3780 8-bit bytes, 5040 6-bit bytes, as discussed in the memo reproduced above. The "version code" of WIT16 was the integer 19770322, which indicates when I froze the design. The format made provision for arbitrary amounts of ancillary information encoded in binary, it had a field for "number of images" in order to support 3-dimensional matricies, it had a blank pixel convention, and the technical memo gives an elaborate discussion of algorithms for dealing with word size and byte-flip variations (like FITS, WIT16 was a Big-Endian format, whereas the popular PDP-11s were Little-Endian [remember, this was still 6 months before the VAX]). --- Preparing for the Basic FITS Agreement --- The Astronomy section of the NSF (US National Science Foundation] organized a meeting whose primary purpose was to try to find a strategy for making image processing computing capability more widely available in the US astronomy community. The meeting was held at KPNO in Tucson in January 1979. During the course of the meeting I said something about the problem of image interchange formats, and about our two years of experience with the prototype formats. The chairman assigned a task force of Bob Burns (NRAO), Ed Groth (Princeton) and me (KPNO) to work on the problem. My folder contains a copy of a handwritten letter from me to Bob Burns dated 29 January 1979: "I am enclosing my little writeup of the WIT16 format [IPPS Memo #3 plus the word/pixel size memo of 1976]. If I were to do it again I would consider 1440 bytes_per_record = 18 80-character card images. I would then suggest that the header record(s) be coded card images, perhaps even with one parameter per line NPIXEL = 512 (NUMBER OF PIXELS PER ROW) NROWS = 768 (NUMBER OF ROWS) etc. so that the header would be self-documenting. Ron Harten sent out a description of his ideas and I think Eric Greisen got a copy." There is also a copy of a memo from Barry Clark (NRAO-VLA) to Bob Burns; it is undated, but my copy says I received it from Burns on 09 March: "I like the style of Don Wells' longhand remarks better than that of the formal memo [IPPS#3, i.e. WIT16 with its binary header]. A single physical block size of either 1440 or 2880 Bytes sounds to me like a reasonable record length. Shorter is inefficient use of tape, longer will encounter buffer problems in very small systems. I suggest the header information shoud correspond to some reasonable standard, with keywords being the main definition effort. I suggest as a standard for the header that keywords be limited to six characters, be followed by '=', ' = ', '= ' or ' =' and then by a single value. A string of blanks would be equivalent to a single blank. Values would be in the form for Fortran 77 list directed I/O...." Barry included a page titled "Suggested List of Keywords", five of which made it into the final Basic FITS Agreement: BSCALE, BUNIT, OBJECT, HISTOR and COMENT (note that Barry proposed six character keywords). --- The meeting at the VLA 26-29 March 1979 --- There is a memo in the file from me to Steve Strom and Geoffrey Burbidge (then Director of KPNO); the date is 8 March 1979 and the subject is "Data Tape Interchange Format Standardization Effort": "One result of the January 26th NSF Image Analysis Meeting was the conclusion that representatives of the major data acquisition and processing sites, e.g. KPNO and NRAO, should endeavor to establish a common data tape format so as to facilitate the communication of digital data, particularly imagery, among astronomers. Peter Boyce [then head of the national centers section of NSF Astronomy] appointed Bob Burns (NRAO), Ed Groth (representing the Space Telescope), and me to form a committee to begin the process. We have corresponded during the past month in order to exchange ideas, particularly regarding my provisional tape format (called WIT16) which KPNO has used to write many of the image data tapes for visiting astronomers to take back to their hime institutions since March 1977. Bob Burns has now invited Ed Groth and me to join him and the VLA programmers at the VLA on Wednesday, March 28th, to begin detailed discussions of tape formats. Interchange of digital imagery among astronomers is becoming common. I believe that it is important for KPNO to encourage standardization agreements in this area, and so I am asking for approval of this trip to Socorro on the 27th and 28th of March." A note on my copy of the memo says "approved 12 March". Another page has notes of a conversation with Bob Burns on 14 March, including this action item: "will send NBS standards 1976, NATO format". The travel voucher which I filed at KPNO [now NOAO] in April 1979 for the trip is attached to the memo. Inflation has been substantial, of course: the airfare cost only US$106. My travel expenses were $54.49, which included a fancy meal in Albuquerque for $19.71, and one of those incomparable steak dinners at the Eagle Ranch in Datil, New Mexico, for only $8.46. The negotiations began with a meeting with most of the VLA computing staff. Eric Greisen and I were charged with producing an agreement, and we retired to the VLA conference room in the cafeteria building. Luck was with us --- our personalities meshed well enough that we could negotiate -- and progress was fast. We had the great advantage of knowing the outcomes of the two prior protoyping experiments. We had plenty of good advice from our colleagues (e.g. Barry Clark's memo). Finally, Ron Harten was present in the room with us, in spirit, as I told him in a letter some weeks later: "Eric and I missed you when we met at the VLA on March 27. But we had your document from last year open on the table in front of us as we struggled with the coordinate and brightness unit specifications. We adopted your concept of the generalized coordinate scheme. Note that our block size is the same as the one you preferred last year. We think that alphanumeric keywords are much preferable to the numeric codes that you and I used in our old schemes. The use of E-notation for values eliminates the need for logarithms. We hope that you approve of our efforts..." My copy of the Basic FITS Agreement is a five-page handwritten document dated 29 March 1979, mostly in Eric Greisen's handwriting. About half of the memo consists of specifications for the reserved keywords of Basic FITS. The "special records convention", which is the basis for Random Groups and Generalized Extensions, is in my handwriting: "The set of all records associated with an image shall be concluded by a tapemark. Note that this does not preclude new record types being invented in the future, except that they must occur *after* the data array records..." There was a final meeting on March 28 in which Eric and I presented our Agreement to the VLA computing staff, and they commented on it. The principal business of that final session was to choose the name; a considerable number of people from all over the world were present, and it was agreed that "FITS" would be reasonably pronounceable and not obscene in any of the languages known to those present. From start to finish the negotiation process had taken about 36 hours. --- Test tapes, draft of the Basic FITS paper --- There is a cover letter from me dated 10 May 1979, and addressed to R.Harten (Westerbork), R.Burns (NRAO-Charlottesville), R.Hjellming (NRAO-VLA-Socorro), E.Groth (Princeton), A.Moffett (CalTech), P.Crane (ESO-Geneva [this was before ESO moved to Garching]) and R.Talbot (Rice): "I have prepared this initial draft of the FITS report, as instructed by the group at the VLA on March 28. Please examine it for misstatements,.... Eric Greisen reports that the first tapes have been exported from NRAO to a user at another site. At KPNO we are constructing a program to read the first FITS tape.. sent to us by Greisen, and we will then build a program to generate a test tape to send back to him. I will be giving a talk about FITS at the International Workshop on Image Processing in Astronomy, to be held at Trieste, 4-8 June 1979." --- FITS became the de facto standard --- The file contains a letter from me to Eric Greisen dated 9 July 1979: "You have done a wonderful job on my crude first draft. I have marked a few typos and questionable details on the copy which is enclosed. Ron Harten had a new scheme ready for Trieste, but he dropped it as soon as he received my first draft. I think it is likely that it will be supported by numerous sites. There was a considerable interest in the idea at Trieste, and I have already sent out several copies of your draft. Our FITS reading program has been operational for more than a month, but our writing program is not yet ready because the programmer [Jan Schwitters, now with the GONG project at NOAO] has gone on vacation. I hope to send you a tape by about the first week of August." Another letter dated 2nd August 1979 is from Denis Warne (Mount Stromlo and Siding Spring Observatories) to me: "...I have given the proposal some consideration and would like to add Stromlo's support to FITS as an interchange format... we have decided to modify our local format so that the astronomical description resembles your header as described in the attached memo..." This was a very important endorsement, because the Australian group had invested considerable effort in the design of their own standard format, culminating in an 11-page paper dated 29 May 1979 and titled "Standard Astronomical Data Format.." Their sophisticated proposal included the ability to encapsulate multiple data objects in a file object, a capability which was added to FITS by the Generalized Extensions Agreement of 1984. They proposed to encode ancillary information using a keyword=value mechanism, analogous to FITS. The final paragraph of their paper begins with the sentence "A subset of the format will come into operation at the Stromlo Observatory during the third quarter, 1979." My memory is that the MSSSO-format paper immediately preceded the presentation on FITS at the Trieste meeting. NRAO and NOAO had software in operation (my talk included slides of NRAO radio imagery >from the first FITS tape, photographed from KPNO's IPPS display), but MSSSO still had only a paper proposal at that moment, and so they joined us. It could easily have happened the other way around: they might have implemented before us, and the astronomy community would now be using some variation on their design. The similarities of the two proposals presented at the Trieste meeting made it obvious that interchange formats were an idea whose time had come. --- Coincidences of Historic Events --- Afficionados of historical trivia will want to know that *three* famous events happened on March 28, 1979: * Birthdate of the Basic FITS Agreement * Conservatives win British election; Margaret Thacher new Prime Minister * Nuclear power plant accident at Three Mile Island, Pennsylvania, releases radioactivity -- Donald C. Wells Associate Scientist dwells@nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N