Subject: Happy Birthday, FITS! [LONG]

From: [email-masked] (Don Wells)
Date: Sun, 29 Mar 1992 05:48:47 GMT

Happy Birthday, FITS!

I consider March 28th to be the birthday for FITS, because it was on
March 28, 1979, that Eric Greisen and I completed the Basic FITS
Agreement. FITS is now a teenager -- today is its 13th birthday. 

			---*---

A file folder which I brought from Arizona to Virginia in 1981 is
labelled "Image Interchange Tape Format notes, Ron Harten (Leiden)
correspondence". That folder contains a number of documents relevant
to the early history of FITS. I reproduce many of them below as a
birthday present for FITS.

		--- Magic Numbers ---

The folder contains a memo by me dated 27 December 1976 and titled
"Word and Pixel Combinations for the proposed Universal Interchange
Format". The purpose of this memo was to document my thinking soon
after Ron Harten (NFRA [now at RCA Astroelectronics, Princeton, NJ])
had visited me at KPNO in November 1976, and had suggested that our
community needed to have an interchange format.  Ron told me that he
had used a magic record size to move Westerbork data between IBM and
CDC computers in the Netherlands; my memory is that it was 720 bytes.
The memo contains the following tables and text:

                              # words per integral # pixels per integral
# bits   prime     # words    # of pixels          # of words
per word factors   per packet 8-bit 16-bit 32-bit  8-bit 16-bit 32-bit
-------- --------- ---------- ----- ------ ------  ------ ------ ------
   6     2.3          240
   8     2.2.2        180
  12     2.2.3        120      2     4       8      3      3      3
  16     2.2.2.2       90      1     1       2      2      1      1
  18     2.3.3         80      4     8      16      9      9      9
  24     2.2.2.3       60      1     2       4      3      3      3
  32     2.2.2.2.2     45      1     1       1      4      2      1
  36     2.2.3.3       40      2     4       8      9      9      9
  48     2.2.2.2.3     30      1     1       2      6      3      3
  60     2.2.3.5       24      2     4       8     15     15     15

"The adopted packet size is 1440 bits, which is the least common
multiple of the word sizes: 2.2.2.2.2.3.3.5, or 32.9.5.  An agreed
maximum number of packets per physical record must be decided. One
suggested number is 21 packets, which is 30240 bits, and is less than
the CDC 6000/7000-series tape physical record size limit of 30720
bits. For this size the tape packing factor is about 89% for both 7
and 9-track tapes at 800 bpi, which is certainly satisfactory.  At
1600 bpi the packing factor falls to about 30% but we need to make
some compromises here for the convenience of small computers. Anyway
800 bpi tape should probably be the recommended standard as they are
the most common density."

# bits
per word  Existing Computers with this word size
--------  --------------------------------------
  12      PDP-8
  16      numerous minicomputers
  18      PDP-9
  24      Datacraft 6024 (now Harris/6)
  32      IBM 360/370-series, Interdata 7/32, 8/32
  36      DEC-10, Univac 1100-series (ones-complement!)
  48      CDC 3000-series
  60      CDC 6000/7000-series and Cyber 170-series

[Note that VAXen are not mentioned for 32 bits -- the VAX-11/780 was
not announced until October 1977, ten months after this memo]

	--- Two Prototypes ---

Ron Harten and I both built prototype formats during 1977, and
exchanged data tapes. John Dickel (UIUC) travelled back and forth
between Westerbork and KPNO during 1977 and 1978 carrying radio and
optical imagery of supernova remnants encoded in the two formats, in
order to carry out his projects to register the imagery. My folder
contains letters from Ron describing his format, and letters from me
offering technical criticism, and responses from Ron point by point.
Ron's prototype included support for world coordinate systems and for
machine-independent wide dynamic range ancillary data (he used
logarithms encoded in binary integers).

The folder also contains a copy of IPPS Technical Memo No. 3, titled
"Structure of the image interchange tapes generated by command WIT16
in the MAGTAPE menu of the IPPS", the memo that documented my
prototype. IPPS is "Interactive Image Processing System", and WIT16 is
"Write Interchange Tape 16" [bits]. WIT16 had a logical record size of
1440 bits and the headers were in binary. The typical blocking factor
was 21, 30240 bits, 3780 8-bit bytes, 5040 6-bit bytes, as discussed
in the memo reproduced above.  The "version code" of WIT16 was the
integer 19770322, which indicates when I froze the design. The format
made provision for arbitrary amounts of ancillary information encoded
in binary, it had a field for "number of images" in order to support
3-dimensional matricies, it had a blank pixel convention, and the
technical memo gives an elaborate discussion of algorithms for dealing
with word size and byte-flip variations (like FITS, WIT16 was a
Big-Endian format, whereas the popular PDP-11s were Little-Endian
[remember, this was still 6 months before the VAX]).

	--- Preparing for the Basic FITS Agreement ---

The Astronomy section of the NSF (US National Science Foundation]
organized a meeting whose primary purpose was to try to find a
strategy for making image processing computing capability more widely
available in the US astronomy community. The meeting was held at KPNO
in Tucson in January 1979. During the course of the meeting I said
something about the problem of image interchange formats, and about
our two years of experience with the prototype formats. The chairman
assigned a task force of Bob Burns (NRAO), Ed Groth (Princeton) and me
(KPNO) to work on the problem. My folder contains a copy of a
handwritten letter from me to Bob Burns dated 29 January 1979:

"I am enclosing my little writeup of the WIT16 format [IPPS Memo #3
plus the word/pixel size memo of 1976]. If I were to do it again I
would consider 1440 bytes_per_record = 18 80-character card images. I
would then suggest that the header record(s) be coded card images,
perhaps even with one parameter per line

NPIXEL =    512 (NUMBER OF PIXELS PER ROW)
NROWS  =    768 (NUMBER OF ROWS)
  etc.

so that the header would be self-documenting. Ron Harten sent out a
description of his ideas and I think Eric Greisen got a copy."

There is also a copy of a memo from Barry Clark (NRAO-VLA) to Bob
Burns; it is undated, but my copy says I received it from Burns on 09
March:

"I like the style of Don Wells' longhand remarks better than that of
the formal memo [IPPS#3, i.e. WIT16 with its binary header]. A single
physical block size of either 1440 or 2880 Bytes sounds to me like a
reasonable record length. Shorter is inefficient use of tape, longer
will encounter buffer problems in very small systems. I suggest the
header information shoud correspond to some reasonable standard, with
keywords being the main definition effort. I suggest as a standard for
the header that keywords be limited to six characters, be followed by
'=', ' = ', '= ' or ' =' and then by a single value. A string of
blanks would be equivalent to a single blank. Values would be in the
form for Fortran 77 list directed I/O...."

Barry included a page titled "Suggested List of Keywords", five of
which made it into the final Basic FITS Agreement: BSCALE, BUNIT,
OBJECT, HISTOR and COMENT (note that Barry proposed six character
keywords).

	--- The meeting at the VLA 26-29 March 1979 ---

There is a memo in the file from me to Steve Strom and Geoffrey
Burbidge (then Director of KPNO); the date is 8 March 1979 and the
subject is "Data Tape Interchange Format Standardization Effort": 

"One result of the January 26th NSF Image Analysis Meeting was the
conclusion that representatives of the major data acquisition and
processing sites, e.g. KPNO and NRAO, should endeavor to establish a
common data tape format so as to facilitate the communication of
digital data, particularly imagery, among astronomers. Peter Boyce
[then head of the national centers section of NSF Astronomy] appointed
Bob Burns (NRAO), Ed Groth (representing the Space Telescope), and me
to form a committee to begin the process. We have corresponded during
the past month in order to exchange ideas, particularly regarding my
provisional tape format (called WIT16) which KPNO has used to write
many of the image data tapes for visiting astronomers to take back to
their hime institutions since March 1977.

Bob Burns has now invited Ed Groth and me to join him and the VLA
programmers at the VLA on Wednesday, March 28th, to begin detailed
discussions of tape formats. Interchange of digital imagery among
astronomers is becoming common. I believe that it is important for
KPNO to encourage standardization agreements in this area, and so I am
asking for approval of this trip to Socorro on the 27th and 28th of
March." 

A note on my copy of the memo says "approved 12 March". Another page
has notes of a conversation with Bob Burns on 14 March, including this
action item: "will send NBS standards 1976, NATO format".

The travel voucher which I filed at KPNO [now NOAO] in April 1979 for
the trip is attached to the memo. Inflation has been substantial, of
course: the airfare cost only US$106. My travel expenses were $54.49,
which included a fancy meal in Albuquerque for $19.71, and one of
those incomparable steak dinners at the Eagle Ranch in Datil, New
Mexico, for only $8.46.

The negotiations began with a meeting with most of the VLA computing
staff.  Eric Greisen and I were charged with producing an agreement,
and we retired to the VLA conference room in the cafeteria building.
Luck was with us --- our personalities meshed well enough that we
could negotiate -- and progress was fast. We had the great advantage
of knowing the outcomes of the two prior protoyping experiments. We
had plenty of good advice from our colleagues (e.g. Barry Clark's
memo).  Finally, Ron Harten was present in the room with us, in
spirit, as I told him in a letter some weeks later:

"Eric and I missed you when we met at the VLA on March 27. But we had
your document from last year open on the table in front of us as we
struggled with the coordinate and brightness unit specifications. We
adopted your concept of the generalized coordinate scheme. Note that
our block size is the same as the one you preferred last year. We
think that alphanumeric keywords are much preferable to the numeric
codes that you and I used in our old schemes. The use of E-notation
for values eliminates the need for logarithms. We hope that you
approve of our efforts..."

My copy of the Basic FITS Agreement is a five-page handwritten
document dated 29 March 1979, mostly in Eric Greisen's handwriting.
About half of the memo consists of specifications for the reserved
keywords of Basic FITS. The "special records convention", which is the
basis for Random Groups and Generalized Extensions, is in my
handwriting: "The set of all records associated with an image shall be
concluded by a tapemark. Note that this does not preclude new record
types being invented in the future, except that they must occur
*after* the data array records..."

There was a final meeting on March 28 in which Eric and I presented
our Agreement to the VLA computing staff, and they commented on it.
The principal business of that final session was to choose the name; a
considerable number of people from all over the world were present,
and it was agreed that "FITS" would be reasonably pronounceable and
not obscene in any of the languages known to those present. From start
to finish the negotiation process had taken about 36 hours.

	--- Test tapes, draft of the Basic FITS paper ---

There is a cover letter from me dated 10 May 1979, and addressed to
R.Harten (Westerbork), R.Burns (NRAO-Charlottesville), R.Hjellming
(NRAO-VLA-Socorro), E.Groth (Princeton), A.Moffett (CalTech), P.Crane
(ESO-Geneva [this was before ESO moved to Garching]) and R.Talbot
(Rice):

"I have prepared this initial draft of the FITS report, as instructed
by the group at the VLA on March 28. Please examine it for
misstatements,.... Eric Greisen reports that the first tapes have been
exported from NRAO to a user at another site. At KPNO we are
constructing a program to read the first FITS tape.. sent to us by
Greisen, and we will then build a program to generate a test tape to
send back to him. I will be giving a talk about FITS at the
International Workshop on Image Processing in Astronomy, to be held at
Trieste, 4-8 June 1979."

	--- FITS became the de facto standard ---

The file contains a letter from me to Eric Greisen dated 9 July 1979:

"You have done a wonderful job on my crude first draft. I have marked
a few typos and questionable details on the copy which is enclosed.
Ron Harten had a new scheme ready for Trieste, but he dropped it as
soon as he received my first draft. I think it is likely that it will
be supported by numerous sites. There was a considerable interest in
the idea at Trieste, and I have already sent out several copies of
your draft. Our FITS reading program has been operational for more
than a month, but our writing program is not yet ready because the
programmer [Jan Schwitters, now with the GONG project at NOAO] has
gone on vacation. I hope to send you a tape by about the first week of
August."

Another letter dated 2nd August 1979 is from Denis Warne (Mount
Stromlo and Siding Spring Observatories) to me: "...I have given the
proposal some consideration and would like to add Stromlo's support to
FITS as an interchange format... we have decided to modify our local
format so that the astronomical description resembles your header as
described in the attached memo..." This was a very important
endorsement, because the Australian group had invested considerable
effort in the design of their own standard format, culminating in an
11-page paper dated 29 May 1979 and titled "Standard Astronomical Data
Format.." Their sophisticated proposal included the ability to
encapsulate multiple data objects in a file object, a capability which
was added to FITS by the Generalized Extensions Agreement of 1984.
They proposed to encode ancillary information using a keyword=value
mechanism, analogous to FITS.  The final paragraph of their paper
begins with the sentence "A subset of the format will come into
operation at the Stromlo Observatory during the third quarter, 1979."

My memory is that the MSSSO-format paper immediately preceded the
presentation on FITS at the Trieste meeting. NRAO and NOAO had
software in operation (my talk included slides of NRAO radio imagery
>from the first FITS tape, photographed from KPNO's IPPS display), but
MSSSO still had only a paper proposal at that moment, and so they
joined us. It could easily have happened the other way around: they
might have implemented before us, and the astronomy community would
now be using some variation on their design.  The similarities of the
two proposals presented at the Trieste meeting made it obvious that
interchange formats were an idea whose time had come.

	--- Coincidences of Historic Events ---

Afficionados of historical trivia will want to know that *three*
famous events happened on March 28, 1979:

* Birthdate of the Basic FITS Agreement

* Conservatives win British election; Margaret Thacher new Prime
	Minister

* Nuclear power plant accident at Three Mile Island, Pennsylvania,
	releases radioactivity
--

Donald C. Wells             Associate Scientist        [email-masked]
National Radio Astronomy Observatory                   +1-804-296-0277
520 Edgemont Road                                 Fax= +1-804-296-0278
Charlottesville, Virginia 22903-2475 USA            78:31.1W, 38:02.2N