Subject: RE: Unsigned integer proposal

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Mon, 21 Mar 1994 18:56:25 +0100

------- Start of forwarded message -------
Posted-Date: Mon, 21 Mar 1994 18:32:42 +0100
In-Reply-To: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
References: <9403211428.AA12356 at xebec.gsfc.nasa.gov>
From: forveill at gag.observ-gr.fr (Thierry Forveille)
To: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Subject: Re: Unsigned Integers Proposal
Date: Mon, 21 Mar 1994 18:32:42 +0100

Arnold Rots writes:
 > 
 > My basic point was (and is) that the need for unsigned integer types
 > is fairly general and will be increasing in the future.  It's going to
 > be easier to introduce them now than n years from today.  The XTE
 > project is not the reason for our wanting to have unsigned integers
 > added to the list: I think FITS needs them and high-lighting their
 > usefulness in the context of the XTE software was intended to
 > illustrate how and why these data types are important and will
 > continue to be so in other projects.  The other side of the coins,
 > though, is that I have no great desire to jump through all kinds of
 > hoops now, if unsigned integers are going to be included in FITS
 > anyway.
 > 
I happen to hold just the opposite opinion: the (perceived) need for unsigned
integers is highly dated to a few years around 1993 and will fade away as it
has arrived. The difference between signed and unsigned integers only matters
when you don't fit on 15 bits, so nobody ever cared about FITS integers 
being signed before 16 bits ADC became common-place, and nobody will ever
care again once 18 or 20 bits ADC will be the norm (though I could imagine some
support gathered for 24 bit integers at that time...). Given typical evolution
times in the electronic industry, my prediction is that this is going to
happen before long, though the convenience of 16 bits as the half word of most
computers may delay that evolution by a year or two. One can already see that
need appearing in CCD manuals that state that the device is linear to better
than 1% over the full range of the ADC: that just means the ADC cannot use 
the full range of the CCD and still sample the noise correctly. What you are 
buying with unsigned integers is just one bit (which you could also get more 
cheaply through a BZERO) and this isn't going to last you very long. A new
data type in FITS should bring more than that (I for one would probably buy
64 bit integers).

	Thierry Forveille
	Observatoire de Grenoble

PS: Since this is apparently becoming an issue in the discussion, I should
probably state that I am a radioastronomer by training (is that a liability?), 
but now working on DENIS, a near IR survey of the sky which will produce about 
as many terabytes as the Sloan project (bigger pixels but more steradians and 
filters). The cost of flipping the high order bit was never even discussed and
is dwarfed  y other things like computing a decent median flat-field.
------- End of forwarded message -------

---

From: tjp at bottom.caltech.edu (Tim Pearson)
Date: 21 Mar 1994 20:00:19 GMT

I too am opposed to adding complexity to FITS readers by adding new
unsigned integer data types to the FITS standard.

The purpose of writing in FITS format is to simplify the transport of
data: that's what the T stands for. Files are always read many more
times than they are written (if not, you might question why you are
writing the file in the first place), and so when defining a data format
it is much more important to consider the needs of the reader than those
of the writer. If we change FITS now, many hours - probably many
hundreds of hours - will be devoted to modifying FITS-reading code to
cope. I question whether this is necessary when unsigned integers of up
to 32 bits can already be represented exactly in FITS files with no
change of standards.

"Unsigned" means that all the bits are used for magnitude, and none is
reserved for sign, and usually the range of values allowed is 0 to 
(2^n)-1, where n is the number of bits. In a FITS file, the range of
values represented may be different (owing to BSCALE and BZERO), and
really has to be determined by examining the data. This applies to both
the present FITS integer types and any proposed unsigned type. [As an
example: suppose a FITS file complying with existing standards specifies
BITPIX=16 and BSCALE=5000: the reader must not blindly multiply the
values by 5000 using 16-bit integer arithmetic!]

Unsigned integers up to 15 bits long can be stored in FITS 16-bit
integers with no problem; similarly unsigned integers from 16 to 31 bits
can be stored in FITS 32-bit integers. Problems only arise when trying 
to store 16-bit unsigned integers in FITS 16-bit integers or 32-bit
unsigned integers in FITS 32-bit integers. However, this can be done
with no loss of accuracy or range by using BZERO, as explained in
earlier contributions to this discussion. Problems may arise when the
reading program tries to apply the BZERO correction if the arithmetic is
not done correctly (e.g., assuming that the corrected result can be
stored in 16-bit signed integer), but that is a problem that FITS
readers already have to address.

"Unsigned" is sometimes (e.g., in the C language) used to mean that
arithmetic on these quantities is done modulo 2^n. I don't think it is
appropriate to impose this restriction on a FITS reader: the writer of the
file does not known what arithmetic is going to be done on the values
contained in the file, now what purpose they will be put to; his
responsibility is solely to ensure that the correct values (not bit
patterns) are read.

Finally, I question whether a particular hardware design (the use of a
16-bit A/D converter, for example) should drive software design,
particularly for data structures that are to be transported to a wide
community. Does the particular quantity being sampled with the A/D
converter really require exactly 1 part in 2^16 precision? Or would 1
part in 2^15 really suffice? Or perhaps you would prefer greater
precision, in which case you should maybe get an 18-bit A/D converter.
But please don't suggest we add 18-bit integer types to FITS.

Tim Pearson
-- 

Tim Pearson, Astronomy Dept 105-24, Caltech, Pasadena, California 91125, USA
Internet:                  TJP  at  Deimos.Caltech.Edu, Pearson_T  at  caltech.edu
NSI-Decnet (SPAN):         Deimos::TJP or 15237::TJP

---

From: VOELS at MYSTRY (Stephen A. Voels)
Date: 21 Mar 1994 23:38:28 GMT

Thierry Forveille writes:
>has arrived.

NO.  This question has been around since the FIRST FITS paper -- I
personally have raised it every time a modification or extension has been
proposed or made to FITS.  There have been and are currently several ground
based CCD projects which could use this.  And I am willing to bet that many
of these projects will be around for several years -- not every group
updates or wants to update their equipement when something better comes
along and older devices are often "willed" to insitutes with less
resources.

It is long past time to put these data types in and now is as good a time
as any.  I know too many projects which decided to go with SIMPLE=F just
because of this.

It does make a difference when you want simple programs at the data taking
location that store the data AS GENERATED.  That is THE best way to make
sure data archived directly from the detectors is correct.

If the FITS groups so not want to recognize this, I personally think Arnold
should just go ahead and define his own extension eventhough it may be
close to BINTABLE and do what is best for his project.  With that much data
people will follow, plus other groups are sure to use it.

Stephen A. Voels

---

From: VOELS at MYSTRY (Stephen A. Voels)
Date: 21 Mar 1994 23:37:33 GMT

Stephen A. Voels