Subject: Unsigned Integers Proposal

From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Date: Wed, 23 Feb 94 17:22:57 EST

Proposal to Add Unsigned Integer Data Types to the FITS Standard
================================================================

Arnold Rots
XTE-GOF
22 February 1994

Summary
-------

As far as integer data types are concerned, FITS only recognizes
unsigned 8-bit and signed (2's complement) 16- and 32-bit integers.
Although there may have been good reasons for this selection in the
past, it is arbitrary, inconsistent, and too restrictive for present
day needs.  I propose that 16- and 32-bit unsigned integers be
recognized, as well.

History
-------

When the first draft of a FITS convention was worked out, more than
two decades ago, the context was that people needed a means to
transport radio astronomical images which usually consisted of
intensities represented by floating point numbers.  It was extremely
cumbersome, in those days, to deal with all the various floating point
representations that were in use (note, that floating point numbers
did not come to FITS until an IEEE standard had been established) and,
hence, the obvious solution was to revert to scaled integers; 16-bit
integers, that is, since not all computers could handle bigger
numbers.  The decision whether to use signed or unsigned was not
difficult: if one has to scale the number anyway, it does not matter,
while the signed integer was more familiar to the Fortran afficionados
of those days (as we all were - except for the Algol buffs).  One's or
two's complement was decided in favor of the latter; I believe that
CDC was the only significant manufacturer around at the time that
still used the former.

When 8-bit integers were admitted to FITS, the motivation was
different.  The push came, no doubt from people who had to deal with
imaging and display devices that effectively used unsigned 8-bit
integers, though characters may have provided impetus, too.
Notwithstanding the fact that the BYTE data type in most Fortran-77
extensions was a signed integer, it was recognized that that would be
a bad choice: the natural device data type was unsigned and insisting
on conformance with the other recognized integer data types would have
required endless scaling of these numbers.

However, these decisions have led to a situation where the choice of
recognized integer data types forms an arbitrary and inconsistent
collection: some signed, some unsigned, without a clear justification
based on current use.

The Current Case
----------------

Contrary to the situation of 20 years ago, many of the data
transported via FITS are now derived from photon counting instruments
that use unsigned integers as their native data type: naturally, since
one cannot count fractional photons or negative photons.  I strongly
urge the various standards organs in the community to consider the
recognition of unsigned 16- and 32-bit integers, for two reasons:

1. With many instruments generating data in these data types, it is
more efficient to avoid the necessity of data conversions, as was the
argument for the introduction of the 8-bit unsigned integers.  Twenty
years ago, one had to scale anyway, so it did not matter.  Now it
does.  Not too long ago, somebody in our community suggested that it
was quite easy to make those instruments fit the standard.  I think
this is, well, arrogant, and putting the horse behind the cart: the
FITS format is there to serve the flow of data from the instrument,
not the other way around.

2. It has been recognized by most that one should preserve the
original data from any instrument as much as possible.  So, regardless
of whether unsigned integers are easier, more natural, or more
convenient, the introduction of unsigned integers as established data
types provides the opportunity to store and transport that much more
data in raw form, with fewer chances for mistakes.

Context
-------

XTE data will be coming down largely as 8-, 16-, and 32-bit unsigned
integers.  We have no use (or, at least, very little) for negative
numbers and we intend to store those numbers in as raw a form as we
can.  Consequently, we have no intention of performing endless
unnecessary scalings from unsigned to signed, and back.

Proposed Implementation
-----------------------

Actually, I am not sure how to implement this for the primary array,
due to the somewhat kludged convention that is currently in use.  One
could either have assigned the negative numbers to the signed integers
or, say, 17 and 33 to the unsigned versions, but the former clashes
with the assignment of floating point numbers, while the latter
prevents us from calculating the data length, based on this number.

I would therefore suggest that we forget about the primary array and,
as in the case of complex and double complex, concentrate our efforts
on the binary tables.  There are two possible solutions:

1. Append the letter "U" to the data type.  A 16-bit unsigned integer
would be represented by "IU", a 32-bit one by "JU".  This has the
advantage that it does not violate the current rules.  However, it is
somewhat distasteful since it looks like another ad hoc kludge.

2. Represent 16-bit unsigned integers by the letter "U", 32-bit ones
by the letter "V".  This choice of letters is reasonably
self-explanatory since it follows the I-J convention.

  - Arnold Rots

 _---------------------Arnold H. Rots--------------------------------
  \    __  __________  Laboratory for High Energy Astrophysics / USRA
   \  / / |_   _   __| Code 668
    \/ /    | | | |_   Goddard Space Flight Center
    / /     | | |  _|  Greenbelt, MD 20771
   / /\     | | | |__
  /_/  \    |_| |____| Phone: (301) 286-1061   Building 2, rm W20F
 -------\_____________ E-mail: arots at xebec.gsfc.nasa.gov-------------

---

From: nicinski at fndaudfnal.fnal.gov (Tom Nicinski)
Date: 28 Feb 1994 15:22:54 GMT

In article <9402232222.AA08514 at xebec.gsfc.nasa.gov>, arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
|> Proposal to Add Unsigned Integer Data Types to the FITS Standard
|> ================================================================
|> 
!> [deleted material]
|> 
|> As far as integer data types are concerned, FITS only recognizes
|> unsigned 8-bit and signed (2's complement) 16- and 32-bit integers.
|> Although there may have been good reasons for this selection in the
|> past, it is arbitrary, inconsistent, and too restrictive for present
|> day needs.  I propose that 16- and 32-bit unsigned integers be
|> recognized, as well.
|> 
!> [deleted material]
!>
|> Proposed Implementation
|> -----------------------
|> 
|> Actually, I am not sure how to implement this for the primary array, ...
!>
!> [deleted material]
!>
!> ... concentrate our efforts on the binary tables.  There are two possible
!> solutions:
|> 
|> 1. Append the letter "U" to the data type.  A 16-bit unsigned integer
|> would be represented by "IU", a 32-bit one by "JU".  This has the
|> advantage that it does not violate the current rules.  However, it is
|> somewhat distasteful since it looks like another ad hoc kludge.
|> 
|> 2. Represent 16-bit unsigned integers by the letter "U", 32-bit ones
|> by the letter "V".  This choice of letters is reasonably
|> self-explanatory since it follows the I-J convention.

We have taken a different route with unsigned integers under Binary Tables.
A new logical keyword was created, TSIGNn.  If not present, the current
"signedness" of the data types is used.  If set to `T', it indicates that
the field value is signed.  If set to `F', it indicates that the field value
is unsigned.  Our FITS Table reader will only allow TSIGNn to appear for
integer data types.

This type of convention would also work for the PDU.  It's completely backwards
compatible (as is Arnold's).  We chose a new keyword, rather than a new data type,
to address PDUs (in case we ever need to do so in the future).  We needed some-
thing other than TZEROn and TSCALn, because our reader is very generic -- thus
we cannot afford to perform any data conversions where loss of information is
possible (we let the user decide how TZEROn and TSCALn are applied).

+----. .' ----+--------------------------------+----------------------------+
|   /| | |  | | Tom Nicinski / MS 234          | nicinski at fnal.fnal.gov     |
|    +-+-+\ | | Fermi National Accelerator Lab |                         _  |
| .  | | | \| | P.O. Box 500                   |                        [o] |
| `--' | |  | | Batavia, IL 60510-0500         |                        /|\ |
+---- -' -----+--------------------------------+----------------------------+

---

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Mon, 28 Feb 1994 20:26:33 +0100

I don't find the case for unsigned integers in FITS totally convincing.
The argument that one should keep the data in as raw a format as possible
is one that I usually buy, but I would make an exception here since the
transformation (just an offset, and no scaling) can be inverted exactly,
to the very last bit included. It is also simple enough that even an
industry contractor should be able to get the code right. I agree with
Arnolds that "the FITS format is there to serve the flow of data from the 
instrument, not the other way around", but the flow one should have in
mind is a flow to astrophysical interpretation, not to a tape vault.
It may be worth putting a small extra load on the data writing group
if it removes a bigger load on the readers (remember most astronomers 
still use fortran...). 

These restrictions apart, the proposal by Arnold Rots is a sound design
for unsigned support if one decides unsigneds are needed. It fullfills
the first rule for a FITS proposal: "Once FITS always FITS". 

The same is not true of the proposal by Tom Nicinski, which, contrary to
its claims is NOT backward compatible. True, a FITS reader implementing
this scheme will still read older FITS files, but the "once FITS always
FITS" rule also demands that old READERS remain valid, not just old files.
Extensions should be designed so that they can either be ignored for zero
order use of the data (e.g. the world coordinate proposal), or so that 
they cause a gracefull crash (e.g. the floating point agreement). The
TSIGNn convention is a sure recipe for producing junk whenever an innocent
FITS reader ignores the convention and reads an unsigned column as signed.

	Thierry Forveille
	Observatoire de Grenoble
	forveill at gag.observ-gr.fr