Subject: Unsigned integer support in NOST 100-1.2

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 13 Jul 1998 12:39:43 -0400

In the previous debate a few years about about adding new unsigned
integer datatypes to FITS I had argued in favor of the proposal. 
However, I have since come to appreciate that the current FITS format
already fully supports unsigned integers (16 and 32 bit at least), hence
I would now oppose any change to the Standard in this regard.  FITS is
able to store unsigned 16 and 32 bit integers in an efficient format
that is easy to read and write.  This is all that is required.  The
transformation between the internal FITS representation and the  machine
representation on a particular computer of the same unsigned integer
value is trivial: flip the value of the most significant bit (bit 32 or
16) and also swap the order of the bytes on little endian machines like
IBM PCs.  

The more important issue for programmers, I believe, is that standard
FITS software like the CFITSIO library should provide a transparent
interface for reading and writing unsigned integers in FITS files.
CFITSIO provides a whole family of routines for reading or writing data
of any supported datatype, including unsigned short, unsigned int, and
unsigned long (in the C language interface) to FITS images or tables. 
When using CFITSIO, the programmer does not need to be concerned in any
way with how the data values are internally represented in the FITS
file.  All the business with setting or reading the BZERO keyword value,
etc. is all handled internally by the interface routines.

Finally, John Davis raised the objection that the BZERO keyword =
2147483648 that is used to represent unsigned 32-bit integers in FITS is
not a valid 32-bit signed integer with some particular compilers (it is
a valid signed integer value with many other compilers, however).  This
is not a problem with the FITS format itself, but instead just
illustrates that the implementation of FITS readers and writers on any
given platform must be able to deal with the limitations of that
platform.  The similar sorts of issues have to be addressed when
reading/writing FITS files in Fortran, which doesn't even support an
unsigned integer datatype at all.

-Bill Pence

---

From: davis at space.mit.edu (John E. Davis)
Date: 13 Jul 1998 22:10:43 GMT

On 13 Jul 1998 12:47:52 -0400, William Pence <pence at tetra.gsfc.nasa.gov>
>I would now oppose any change to the Standard in this regard.  FITS is

I understand your point that a fits can store unsigned values using
the current convention.  As long as that convention is documented, I
do not care about what bits need to be flipped.  However, I think that
there is a deeper issue that keeps nagging me.  This leads me to
believe that the fits file specification does not define a
well-defined file format that is independent of the reader.

To see what I mean, consider the BZERO and BSCALE keywords.  The NOST
document states that these keywords represent floating point values,
and if the keywords are absent, then they default to 0.0 and 1.0,
respectively.  Then the value of a pixel pixel is given by:

   Physical_Value = BZERO + BSCALE * Array_Value

Does this mean that all images are floating point images?  If so, are
they 64 bit floating point values or 32 bit floating point values?
The file specification does not spell this out which means that the
interpretation is left to the fits reader.  In my opinion, the
document should state that the result of

   Physical_Value = BZERO + BSCALE * Array_Value

depends upon the range of the resulting values, and upon the data types
specified by the syntax used to specify the BZERO and BSCALE keywords.
In this respect, the default values should be the *integers*, 0 and 1,
and not floating point values.

At this point, I have concluded that a fits file simply stores any
numeric value as a 64 bit floating point value in various floating
point and signed integer representations, using the BZERO/BSCALE
values (or their equivalents) to perform the appropriate conversions.
Please try to understand that I am not implying that everything gets
written out to the fits files as 8 byte floating point double
precision numbers.

I feel that the only consistent way to view the current specification
is that all data types are implicitly converted to the largest
available data type capable of representing them, namely 64 bit
floats, and then written out in a user specified representation, e.g.,
as signed 16 bit integers, with the appropriate values of BZERO and
BSCALE.  I can live with this interpretation as long as I can store
all the numeric data types that I work with with no loss of precision.
However, this will present a problem when the file format gets
enlarged to handle 64 bit integer quantities.  Then,

   Physical_Value = BZERO + BSCALE * Array_Value

will result in a loss of precision for 64 bit integers.

Some may find this whole debate pointless, but if you want to define a
fits file whose contents have a well defined meaning that is
independent of the software that reads it, then these seemingly
trivial points matter.

--John

---

From: hodge at bowline.stsci.edu (Phil Hodge)
Date: 15 Jul 1998 16:01:58 GMT

> 16) and also swap the order of the bytes on little endian machines like

This is trivial to implement, yes, but it's not transparent to many
people who would read the header.  It's rather esoteric, in fact.
I think that FITS should support unsigned 16-bit and 32-bit integers
using notation (e.g. a DATATYPE keyword) that would be clear to an
intelligent person who is not necessarily an expert on FITS.

Rob Seaman wrote:

> Perhaps rather than worrying about unsigned integers (or 24 bit ints or
> whatever), we should get back to devising a speedy FITS compression scheme
> that would allow using 32 (or even 64) bit integer pixels for intermediate
> precision data in a space-efficient manner.

Would that be 32-bit unsigned pixels?

						Phil

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Wed, 15 Jul 1998 17:06:05 -0400

> intelligent person who is not necessarily an expert on FITS.

The FITS standard allows optional characters following the datatype
character in the TFORMn keyword, so one could adopt a convention
something like:

TFORM1  = '1IU'               / datatype is unsigned 16-bit integer
TZERO1  =               32768

or

TFORM1  = '1I(unsigned)'      / datatype is unsigned 16-bit integer
TZERO1  =               32768

This doesn't require any change to the FITS standard, and the TZEROn
keyword is still required, but at least it would be clearer to  people
reading the header that this is an unsigned integer column. 

____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)

---

From: Thierry Forveille <Thierry.Forveille at obs.ujf-grenoble.fr>
Date: Thu, 16 Jul 1998 09:56:08 +0200 (DFT)

> reading the header that this is an unsigned integer column. 
Really I don't see the point of keeping track of the unsigned origin of
a datum. As mentioned earlier in this discussion, the purpose of FITS is
to transfer numerical information, not bit patterns. Whether a column
originates from unsigned IEEE 16 bits integers or 30 bits one from a CDC
mainframe is irrelevant for the receiving program, as long as it fits
in its specified format. Most modern image processing systems will
convert it to 32 bits integers anyway (if not to floats) for the sake
of efficiency.

Actually, all people I have heard arguing for an unsigned 16 bit format
(nobody seems to demand 32 bits unsigneds) really want to use it to store
floating point data affected by quantisation noise ;)