Subject: Re: Unsigned Integers Proposal

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 1 Mar 1994 15:25 EDT

In article <2kt28e$kjs at fnnews.fnal.gov>, nicinski at fnal.fnal.gov writes...
>In article <9402232222.AA08514 at xebec.gsfc.nasa.gov>, arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
>|> Proposal to Add Unsigned Integer Data Types to the FITS Standard
>|> ================================================================
>|> 
>!> [deleted material]
>|> 
>|> As far as integer data types are concerned, FITS only recognizes
>|> unsigned 8-bit and signed (2's complement) 16- and 32-bit integers.
>|> Although there may have been good reasons for this selection in the
>|> past, it is arbitrary, inconsistent, and too restrictive for present
>|> day needs.  I propose that 16- and 32-bit unsigned integers be
>|> recognized, as well.
>|> 
>!> [deleted material]
>!> ... possible solutions:
>|> 
>|> 1. Append the letter "U" to the data type.  A 16-bit unsigned integer
>|> would be represented by "IU", a 32-bit one by "JU".  This has the
>|> advantage that it does not violate the current rules.  However, it is
>|> somewhat distasteful since it looks like another ad hoc kludge.
>|> 
>|> 2. Represent 16-bit unsigned integers by the letter "U", 32-bit ones
>|> by the letter "V".  This choice of letters is reasonably
>|> self-explanatory since it follows the I-J convention.
> 
I think 2. is better.  But the community will have to discuss this 
issue.

>We have taken a different route with unsigned integers under Binary Tables.
>A new logical keyword was created, TSIGNn.  If not present, the current
>"signedness" of the data types is used.  If set to `T', it indicates that
>the field value is signed.  If set to `F', it indicates that the field value
>is unsigned.  Our FITS Table reader will only allow TSIGNn to appear for
>integer data types.
> 
But what about someone else's FITS reader, designed according to the 
definition of BINTABLE in the Cotton, Tody, and Pence proposal?  It 
will ignore the TSIGNn keyword and read the value as a signed integer. 
The whole point of FITS is to create a transport standard that is 
universally recognized.  Local conventions that will not be understood 
by readers elsewhere defeat the purpose of a standard.  Until there is 
some general agreement their use should be limited to those who use 
them.  And they should be designed in a way that will not break 
standard FITS readers.

For any file with the above convention to be FITS conformant, the
binary tables would have to have the value of XTENSION set to other
than BINTABLE, and the new name should be registered with the IAU FITS
Working Group. BINTABLE is already reserved for the Cotton et al.
proposal. Alternatively, the entire file could start with SIMPLE = F,
to indicate that the included binary table does not conform to the
rules for BINTABLE. 

>This type of convention would also work for the PDU.  It's completely backwards
>compatible (as is Arnold's).  We chose a new keyword, rather than a new data type,
>to address PDUs (in case we ever need to do so in the future).  We needed some-
>thing other than TZEROn and TSCALn, because our reader is very generic -- thus
>we cannot afford to perform any data conversions where loss of information is
>possible (we let the user decide how TZEROn and TSCALn are applied).
> 
This proposal is also appropriate for community discussion but should 
not be unilaterally implemented.

				Barry Schlesinger
				NSSDC/NOST FITS Support Office

---

From: oegerle at stsci.edu (Bill Oegerle,326,4744)
Date: Wed, 2 Mar 1994 16:58:49 GMT

In article AA25016 at fits.cv.nrao.edu, forveill at gag.observ-gr.fr (Thierry Forveille) writes:
> I don't find the case for unsigned integers in FITS totally convincing.
> The argument that one should keep the data in as raw a format as possible
> is one that I usually buy, but I would make an exception here since the
> transformation (just an offset, and no scaling) can be inverted exactly,
> to the very last bit included.... (rest deleted)

I agree.  for instance, in IRAF, using the "wfits" task, just use the
parameters:

> wfits file=xxx bscale=1 bzero=32768 bitpix=16 scale=yes

(note that even though you tell IRAF to scale the data, this does not really
happen... you say scale=yes just to get IRAF to use the specified values of
bscale and bzero.  since bscale=1 there is no scaling, just an offset of bzero=32768)

--Bill Oegerle
  STScI

---

From: ron at argus.lpl.Arizona.EDU (Ron Watkins)
Date: 2 Mar 1994 21:08:30 GMT

I am unclear on this whole concept of scaling signed values to unsigned values.

Say I have a value in the range 0..65536, Now if these bits are interpreted
as signed values then they are arranged as follows for two's complement:

Unsigned	Signed		Signed+32768
0		0		32768
.		.		.
.		.		.
32767		32767		65535
32768		-32768		0
.		.		.
.		.		.
65536		-1		32767

Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed
values. This will goof things up, won't it? What's going on? How can I take
bit patterns that I already have as unsigned and put them into a FITS file
and make the come out again correctly AND at the same time, make it so the
FITS file makes sence of those values?
--
Ron Watkins    [ron at argus.lpl.arizona.edu]    /            /~~~~)     /
931 Gould-Simpson                            /            /____/     /
University of Arizona                       /            /          /
Tucson AZ. 85721 -- (602) 621-8606         (____ unar & / lanetary (____ ab.

---

From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Date: Thu, 3 Mar 94 09:11:22 EST

> FITS file makes sence of those values?

You are right, this does not work.  In the current situation, you
first have to subtract 32768 from all your numbers, before you deposit
them into the FITS file.

That is why I recommend FITS recognize unsigned integer data types.

Even so, there is one additional complication: Fortran does not know
unsigned integers.  Hence, if your file contains unsigned 16-bit
integers, and you are working in Fortran, you need to put them into
32-bit integers; if you are working with unsigned 32-bit integers, I
guess the only thing you can do officially, is put them into doubles.
If you are working in C, you don't have any of these problems, of
course, but beware: if you are using a Fortran-based FITS reader, you
might still be in for surprises.

  - Arnold Rots

---

From: drtr at mail.ast.cam.ac.uk (David Robinson)
Date: 3 Mar 1994 14:22:38 GMT

In article <2l2v8e$qca at organpipe.uug.arizona.edu> ron at argus.lpl.Arizona.EDU (Ron Watkins) writes:
>...

It's very simple. You have data with values x = 0 ... 65535.
So x - 32768 is in the range -32768 ... 32767. Thus the (x-32768) values can be
stored in a BITPIX=16 signed 16-bit integer array. The data has not been
scaled, and has had a constant subtracted from it, therefore bscale = 1 and
bzero = 32768.

Any fits reader will apply the bscale/bzero values to the raw data before
presenting it to the user, thus restoring the original 0 ... 65535 values.

Note that a portable program will *never* be able to simply 'take the bit
patterns that it has as unsigned'; on many computers, the bytes of a native
unsigned int will be in the wrong order compared to the required FITS format.

The existing FITS standard, therefore, can support efficient storage of
unsigned 16-bit integers. Adding a new format would give no extra
functionality, generate incompatibility problems, and require more complex FITS
readers.

 David Robinson. (drtr at mail.ast.cam.ac.uk)

---

From: pgrosbol at eso.org
Date: Thu, 03 Mar 94 18:00:11 +0100

On the Proposal to Add Unsigned Integer Data Types
       to the FITS Standard by Arnold Rots

The discussion on unsigned 16-bit integers as a FITS
data type started already in the mid 80's when CCD
acquisition systems started to produce 15-16 bit data.

It has been clearly demonstrated in the previous
discussion that it is possible to offset unsigned intergers
and save them as signed values. No information is lost
and the cost is only two interger operations which at
the time of multi MIPS CPUs is insignificant.

If new data types were introduced all FITS reader in the
world would in principle have to be upgraded to conform
to the new standard. This is a dramatic action which can
only be justified if very significant new functionality
is introduced. I believe it was the case when IEEE floating
point formats were added.

It should be noted that the rules for 'Generalized extensions
and blocking factors for FITS' A&A Suppl. 73,p359 state:
  'Only the binary and character coding conventions specified
   in the FITS standards should be used in FITS extensions.'
Thus, one cannot add unsigned intergers to the BINTABLE extension
without changing the full set of FITS data types. Values of
BITPIX=17 or 33 would violate the formular for the size of the
data matrix: nbit = abs(bitpix)*gcount*(pcount+naxis1*...*naxisn)
and are not acceptable.

The proposal would only bring a marginal advantadge (i.e. saving
two integer operations per pixel) but require very significant
code modifications. Thus, I cannot support the proposal.

Preben Grosbol
ESO

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Fri, 4 Mar 94 15:22:42 EST

Regarding the unsigned integer proposal by A. Rots, my opinion it that
the lack of support for unsigned integers is a serious deficiency of
FITS that needs to be addressed.  This issue is not going to go away no
matter how much we preach that unsigned integer values must be
transformed into scaled integers with the appropriate TSCAL keyword
value.  Many people, as is already evident from the discussion about
this proposal, will simply not follow this FITS requirement and
will go ahead and write the unsigned value to the FITS file.  Some
might add a keyword like SIGNED = F to try to indicate that they broke
the rules, but many won't even do this.  Some people will do this out
of ignorance, not realizing that it is possible to coerce unsigned
integers into a signed integer datatype.  Others may just refuse to do
it because they think this sort of scaling is all rather mindless and
stupid, and they see no point in it.  Still others may have valid concerns
about the impact on efficiency, or on the added complexity of the
software, if they have to scale the unsigned integer data.  So for
whatever reason, people are breaking the FITS rules on this, and I
think the problem will only get worse in the future if more and more
groups start using FITS as their primary data format.  Other competing
data formats, like HDF, do support unsigned integers, so unless FITS can
adapt to the need of users, users will not use FITS.

My preference would be to setup a small technical committee that would
consider all the issues and try to come up with a proposal for
supporting unsigned integers that minimizes the impact on existing FITS
readers.  I've already looked at what it would take to  
add full support for unsigned I*2, I*4, and signed I*1  values in both
primary arrays and in binary tables within my FITSIO subroutine
package and estimate this could be done in a week or 2.  Any
FITS software that uses FITSIO could then transparently read or
write unsigned integers without any change to the application
program, simply by linking to the new FITSIO library.  I think
that the disruption to existing FITS readers is a managable problem,
and the benefits of adding this capability to FITS outweighs the
problems.

-Bill Pence
 HEASARC

---

From: forveill at gag.observ-gr.fr (Thierry Forveille)
Date: Fri, 4 Mar 1994 21:57:09 +0100

Arnold Rots writes:
 > In the current situation, you
I don't really see what is wrong with subtracting 32768 before writing. The
cost is small compared with the CPU overhead of the physical I/O, and depending
on the computer you are using, you may have to swap the bytes (also a more
costly operation). On RISC architectures, I think instructions for unsigneds 
are actually emulated (16 bits signed instructions too), so CPU efficiency
isn't obtained that way. The additional complexity of the code doesn't seem 
a very serious issue either. Is that rather a philosophical issue of whether 
one is allowed to alter the bits at all?

 > might still be in for surprises.
Most processing packages would in practice put the data in a float,
not in a 32-bit integer. Integers are most frequently used to store what is
really a fixed point real number, at least when there are enough of them
that the size matters. Even photon counts (which are intrinsically integers)
have to be converted to floats for any serious processing or calibration
(rather than just data display). As soon as you need to convert counts
to photons/s/m**2, calibrate the relative response of a spectrograph, or
remove an instrumental background, the data become reals, not integers.
This is another reason why one need not care that the data are stored
as signed with an offset rather than unsigned: what will really be used
is a float or a double.

		Thierry Forveille
		Observatoire de Grenoble

---

From: mo at cfa234.harvard.edu (Maureen Conroy)
Date: Thu, 10 Mar 94 16:52:24 EST

If there is a well-defined and supported way to
write UNSIGNED integers to FITS files, I would think that the
most important issue is to be sure that the technique is clear
and well-documented.  The NOST FITS office has a FITS users guide
to provide guidelines in using FITS files.  Judging
from the varied responses it would seem to be an excellent example
to cite specifically in the guide, since it seems to be a 
frequently needed feature.  
	Barry - what do you think?

				Maureen Conroy
				SAO

---

From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER)
Date: 14 Mar 1994 09:40 EDT

In article <9403102152.AA11370 at cfa234.harvard.edu>, mo at cfa234.harvard.edu (Maureen Conroy) writes...
>	Barry - what do you think?

Anything done in the near term has to be done in a way that will not
be misinterpreted by a reasonable FITS reader.  Use of BZERO and
BSCALE satisfies this criterion.  The technique is discussed in
general terms in the first FITS paper.  I will add specific discussion
in the next *full* revision of the User's Guide (version 4.0) about
how unsigned integers can be handled through the use of scaling. 

The TSIGNn proposal does not satisfy this criterion, as FITS files 
using it would be misinterpreted by a FITS reader constructed on the
basis of the current rules of FITS.  Similarly, the proposed new data
types in BINTABLE would not be understood by a current FITS reader.
Such proposed new structures for unsigned integers must be discussed
by the FITS community.  If a consensus should emerge that a new
rule is necessary, then the normal procedures of consideration by
the regional and international FITS committees and demonstration of
its use in data tranport must be followed before it can be approved as
part of FITS.  Present discussion suggests that there is not currently
a consensus as to whether specific provisions for unsigned integers
are desirable.  

				Barry Schlesinger

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Tue, 15 Mar 94 18:09:11 EST

To further support the need for a FITS unsigned datatype in binary
tables, the format for the data coming from the X-ray Timing Explorer
Satellite (XTE) is briefly described bolow.  In most of the previous
discussion of this subject it has been assumed that the entire data
stream consists of unsigned integers, and thus it is simple in
principle to just scale  all the data into signed integers.  But in the
XTE case, both signed and unsigned data values are intertwined in
a complex way within the telemetry stream, so it can be quite
complicated and computationally expensive to identify and scale just
the unsigned telemetry values before writing the data to a FITS binary
table.  It is estimated that XTE will be generating close to 1 Terabyte
of data per year (!), so the computational overhead of having to scale
any unsigned integers before writing them to FITS files is a major
concern.

A somewhat simplified description of the XTE data follows:

The XTE satellite dumps a buffer full of data at periodic time
intervals.  The contents of the buffer depends on the observing mode,
(there are dozens of different possible modes).  The natural way to
store this information in a FITS binary table is to write each buffer
of data, along with a time stamp indicating when the data were
received, into one row of a binary table.  The next buffer of data
would be written to the next row of the table.  All the buffers for a
given observation would be contained in a single table.

Each buffer of data can be futher broken down into individual
components, each of which is logically a different column of the
table.  To give a simple example the, data buffer for a given observing
mode might be 322 bytes long containing 2 scalar 8-bit telemetry
values, followed by a histogram composed of 128 16-bit unsigned values,
followed by another histogram of 64 8-bit unsigned values.  (this is a
hypothetical example, but it is at least representative of what real
XTE data will look like).  So, for this example the most natural lay
out for the binary table would be to have 2 scalar columns followed by
2 vector columns.  The 2 scalar columns  would have a 'B' byte data
type, and the second vector column would have a '64B' datatype.  The
first vector column contains 128 unsigned integers so ideally we would
like to define the data type to be something like '128U' or
'128I*UNSIGNED'.

If FITS directly supports unsigned integers then it is conceptually
simple to convert the XTE data to FITS: as the data comes in, first
identify what observing mode it is in and write the appropriate FITS
header defining all the columns in the table.  Then just copy each
buffer as it comes, byte by byte, into the table, one after another,
until the observation is complete.

But if FITS doesn't directly support unsigned integers, the job is much
more complex.  Before appending each new buffer to the table you have
to figure out which bytes within the buffer represent unsigned
integers, and then scale only them so that they fit within the range of
the FITS signed integer data type.  This process is bound to be rather
computationally expensive. XTE will be generating about 2 Gbytes of
data per day and there is already a lot of concern about how much
computing power will be needed to process this amount of data, so
efficiency is very important.

No final decision has yet been made on how to deal with the XTE
unsigned data values in the FITS files, but it looks rather likely at
this point that the project will simply not have the computer resources
necessary to scale the unsigned values and will have no choice except
to invent a convention for directly encoding the unsigned values in
FITS binary tables.

---

From: dwells at nrao.edu (Don Wells)
Date: Wed, 16 Mar 1994 02:50:09 GMT

In article <9403152309.AA11220 at tetra.Gsfc.NASA.Gov>
    pence at tetra.gsfc.nasa.gov (William Pence) writes:
   ".. XTE will be generating.. 1 Terabyte.. per year .."

The nominal design data rate of the VLBA (my previous RT project) is 
4 GB/day, 1.5 TB/year.  This sort of data rate is indeed large by
historical standards, but it is fairly typical in modern systems.

    ".. [figuring] out which bytes within the buffer represent unsigned
   integers.. [will be].. rather computationally expensive.."

It sounds to me like you have many records with the same structure.  I
suggest that a list structure be computed when the first record of a
given type is encountered, so that the list structure can control
efficient processing of each succeeding record. If the number of types
is small enough, precompile optimized algorithms (e.g. maybe even using
unrolled loops).

    ".. XTE will be generating about 2 Gbytes of data per day.."

2 Gbyte/day is 23 Kbyte/s.  
Even my old PC/XT clone can probably achieve that rate.

    ".. efficiency is very important.. rather likely.. project will
    simply not have the computer resources necessary to scale the
    unsigned values .." 

I doubt that this assertion will stand up to a critical review.

                -*-

Modern RISC processors are sufficiently pipelined that they can
overlap one or more cycles of register operations with the clock
cycles of a memory-to-memory copy.  For high-bandwidth
nearly-I/O-limited systems designers should consider how many times
the bits go into and out of main memory. That is where the action is!
They should also examine the components which are doing the DMA
transfers, how many memory cycles they steal from overlapped CPU
operations, and how many memory-to-memory copies are being done in the
I/O system. One or two register-to-register operations in a hot RISC
engine will pale to insignificance in comparison to several redundant
memory cycles, and such register operations may even be free-of-charge
if pipelining occurs.
--
  Donald C. Wells         Associate Scientist         dwells at nrao.edu
                    http://fits.cv.nrao.edu/~dwells
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road,   Charlottesville, Virginia       22903-2475 USA

---

From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Date: Fri, 18 Mar 94 11:38:13 EST

I have watched the debate over the issue of introducing unsigned
integers into the FITS Binary Tables closely and my conclusion (not
surprising, since I started this round in the discussion) is: we
should do it and we'd better do it quickly.

The Binary Table standard contains a fairly large and fairly complete
set of data types.  The criterion for defining this set cannot have
been to arrive at a minimum set.  Considering the fact that it
contains complex and double complex, the approach must have been to
include all meaningful data types commonly in use among astronomers.
I would submit that unsigned integers ought to be included in this
collection, simply on the basis of generality.

Frankly, I think, there is too much parochial thinking going on.  I
have no intention to pick on anyone in particular, but I understand
very well where Don Wells is coming from: in radio astronomy floats,
doubles, complex, signed integers and scaled integers have
traditionally dominated the zoo of data types.  Among the latter
especially the 16 bit variety, as illustrated by Don's remark:

> I am intrigued by the fact that each number system has a number whose
> negation is arithmetically equal to itself. In the 1-s complement
> system the number is zero ((+0)==(-0) in C-speak), while in 2-s
> complement it is 32768 ((+32768)==(-32768)).

Note that this implicitly assumes that all integers are 16-bit signed
integers.  I am sure that Don did not mean it that way, I but mention
it to illustrate the implicit assumptions we are all making.  My point
is that in the rest (i.e., non-radio) of astronomy, complex (for
instance) does not play such a big role, but unsigned integers do.

Again, if we *can* store original values in FITS tables, I feel it
*should* be done.  Every extra line of code increases complexity and
chances for errors; this is of specific concern to those among us who
work on software that will produce basic archives: the number of
operations should be kept to an absolute minimum, on the one hand,
while the archived product should provide as good data access as
possible.  Unsigned integers would help tremendously in that regard.

All I am asking for is that unsigned integers be recognized in Binary
Tables only.  The Binary Tables proposal has only just been approved,
so, since unsigned integers are unavoidably coming, as Bill Pence
pointed out, the sooner they are recognized, the easier the phasing in
will be.

On to the XTE case.  Don Wells and Tom McGlynn have provided us with
some quick calculations that all of us (I hope) are capable of
carrying out but that, in their simplicity, completely miss the point.
The problem is not in the actual scaling itself, though I might point
out that we will decline Don's offer of his PC/XT clone, since we hope
to be able to process a day's worth of telemetry in considerably less
than 24 hours.  Figuring out when, where, and how to scale, involves at
least the same amount of processing.

More important, though, is the added complexity.  Of course, it can be
determined what needs to be scaled by what.  And, of course, it can be
table driven.  But we are talking about a significant increase in code
complexity.  XTE will generate something of the order of a TByte a
year in FITS files, containing data in something of the order of a
thousand data modes (Bill Pence was a little optimistic with his
"dozens").  In order to keep this from turning into a configuration
control and software maintenance nightmare, we have neatly
encapsulated the various functions: the data management subsystem (in
C++) can provide any client (not just a FITS writer) with data values;
the FITS writer (in C) takes the values and dumps them into FITS
Binary Tables.  We don't want to force the former to comply with FITS
standards (for obvious reasons), while we also want to keep the latter
as simple as possible.  I cannot over-stress the importance of this
point: this is crucial software that creates *the* data archive for
this mission; every increase in its complexity increases the risk of
errors, in implementation and maintenance. Unnecessary capabilities
and sophistication is just not acceptable for this type of software.
The FITS writer currently has no direct knowledge of what it is
writing (at least not while it happens) because there is no need for
that capability; strict encapsulation and design simplicity have made
this piece of software highly reliable - I would like to keep it that
way.

For Binary Tables, I would propose the following data types.  New ones
are indicated with an asterisk (*).  I am not quite sure whether
64-bit integers are warranted at the moment, but considering that we'd
better do this right for the next 10 years, or so, I have included
them.  I hope nobody will mistake "H" for Hollerith.

    (L)  8 bit logical
    (X)  1 bit item
    (A)  8 bit ASCII character
    (E) 32 bit float
    (D) 64 bit double float
    (C) 64 bit complex float (2 times 32 bits)
    (M)128 bit complex float (2 times 64 bits)
  * (H)  8 bit signed integer
    (I) 16 bit signed integer
    (J) 32 bit signed integer
  * (K) 64 bit signed integer
    (B)  8 bit unsigned integer
  * (U) 16 bit unsigned integer
  * (V) 32 bit unsigned integer
  * (W) 64 bit unsigned integer

In addition, of course, there is the P.

There is one more issue that I would like to address.  Both Bill Pence
and Barry Schlesinger have hinted at how unsigned integers could be
used in FITS files until such a time as when we have an official
standard.  I am very hesitant to use devices like:
    TFORMn  = 'I:UNSIGNED'   / this is an 16 bit unsigned integer column
or:
    SIMPLE  =              F / This is my type of FITS file
or:
    XTENSION= 'MYBINTABLE'   / A BINTABLE with unsigned integers
as interim measures.  If a standard got established at any point
during the XTE mission, we would either end up with an archive that
contains different types of files, or face the task of reprocessing
all previous data.  Neither is palatable.  The point is that such
devices are quite acceptable for limited amounts of, more or less,
volatile data.  For definitive archives (which, one hopes, would also
outlive software systems that are currently in use) other
considerations come into play.

I sincerely hope that the community will move forward with this
expeditiously.  As Bill Pence remarked, this problem is not going
away; therefore, the sooner we deal with it, the better.  And to
require everybody to deal with the lack of unsigned integers in the
future, because "we have always managed to do without" in the past,
is, in my view, a dangerous position.

  - Arnold Rots

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Fri, 18 Mar 1994 18:57:27 +0100 (MET)

Here are my 0.02 currency units (Ukrainian coupons ?) about the 
"unsigned integers in FITS" controversy, which I have been following
for a while. I apologize if in the posting below I do not acknowledge
all previous contributors by name, but I am quoting off my memory.

---------

I have always been liking Ockham's razor, in this particular case
"data types non sunt multiplicanda praeter necessitatem" : the number
of data types should be kept to a minimum.

So just on this ground, I have looked favourably at the various
contributors which argued very convincingly that unsigned integers
can be handled with the BSCALE/BZERO convention without an unreasonable
CPU load and therefore remain of the opinion that one should live
with a LIMITED number of data types.

------------

The usage of a minimal subset could also be helpful to developers in
different languages. Personally I use only Fortran, and I would be
annoyed to deal with unsigned integers which are not supported by
the language (see however below for bytes et al.). One contributor
noted that C programmers may feel the same about the COMPLEX data type
(actually I believe I've used it in Fortran only once, and that was
not serious work, a Mandelbrot set game).

In my opinion the really useful set (which does not mean I propose to
delete what is already in the standard beyond it) is :

  - 8-bit characters                               B
          or bytes (unsigned ....)                 C
  - 16-bit signed integers                         B
  - 32-bit signed integers                         C
  - 32-bit IEEE floating                           A
  - 64-bit IEEE floating                           B

The letters ABCD indicates decreasing "priority of use". In fact, talking 
of numeric values, there is usually little reason to prefer INTEGER values 
(with the annoyance of scaling) to REAL values, except perhaps sometimes
for space reasons (or for "conservative" reasons, see below).
E.g. if I want to store a spectrum, I store it in counts/s (REAL) with
associated errors (REAL) and not in counts (INTEGER), etc.

-------------

One argument which was put forward by some contributors, and which I
term here for brevity "the conservative argument" is : "we should store
data as close as they come out from the instrument" (this arose, as far
as I remember, both from CCD users and from XTE)

i) this is sometimes reasonable, but cannot it be done within the
   existing conventions ?

   I often had to deal with 8-bit and 16-bit data from X-ray detectors.
   8-bit values might be some ADC channel level, or counts in an histogram 
   (and in both cases usually are in the full unsigned range 0-255).
   16-bit values may be counts in an histogram, but in this case although
   nominally unsigned the probability of being above 32767 is really low.
   I hardly see the need of using the full range of a 32-bit (signed or
   unsigned) for anything (but on-board clocks, which deserve a special
   treatment anyhow).

   One should separate the way the data are STORED in a file, and the
   way they are ELABORATED in variables in a program. As a matter of
   fact I operate only on INTEGER*4 variables. 
   I note that FITS (both the "original" FITS and the Binary Tables)
   consider 8-bit values to be unsigned (unlike other integers, and
   unlike the convention for the non-standard BYTE data type in some 
   Fortrans; in fact I never use the BYTE data type, but always read
   the unsigned 8-bit value in an INTEGER*4 with some  CHAR/ICHAR
   trick). 
   Copying the two bytes of an INTEGER*2 into an INTEGER*4 preserving
   its value should be reasonably easy with some EQUIVALENCE tricks.
   
   I refer also to previous postings about BSCALE, BZERO etc.

ii) there are cases in which the conservative argument lead to the
    extrema would cause proliferation of strange data types.

   It often happens that satellite data have a "funny" number of bits
   (either because they are "naturally" produced this way in the 
   electronics, or because the need of the telemetry require to chop
   off redundant 0 bits) : so one can have a 10-bit energy, a 3-bit
   detector id, etc.

   In this case I would consider extremely annoying for the general
   observer to deal with such data types. However that is what would
   be implied if one follows strictly the "store data as they come
   out of the instrument" approach.
   For instance, the requirement which was put on the Ground Segment
   Contractor for SAX, was to reformat such data in a simple way, i.e.
   expand all 1- to 7-bit fields into an 8-bit 0-padded field, all
   9- to 15-bit fields into a 16-bit field, and all 17- to 31-bit
   fields into a 32-bit field (the "natural" sizes which can be
   handled as said in (i) above).
   This way data are kept "close to the form they come out of the
   instrument" (e.g. the partitioning in telemetry packets is
   preserved, and no destructive processing is performed), but they
   are reformatted in a friendly way.

   I note that FITS does not consider a true bit data type, since the
   rX binary table type requires the same padding to n*8 bit, but
   unfortunately (unexplainably ?) requires left-justification.

iii) a philosophical argument : when is something to be called FITS.

   As far as I remember the postings about XTE were mentioning "telemetry
   data buffers" downlinked from the spacecraft in a "huge number" (quoted
   as "dozens" to "thousands" sic!) "of data types", which had to be
   "stored as close as possible as they come down from the spacecraft",
   but at the same time be FITS. And the proposed solution was to store
   each "buffer" as a row in a FITS table.

   Now, while I could see good reasons to keep the "buffers" (more or
   less corresponding to the "telemetry packets" mentioned in ii above)
   as untouched as possible (but I hope this does not extend to keeping
   5-bit fields ... again see ii above), what I fail to see is why one
   should try to encapsulate this which is not (and by good reasons)
   a FITS structure into a "nominalistic" FITS wrapper.

   If one had to do some processing, and export processed data, I would
   agree about the case for using an existing FITS structure, or
   devising an easy one.

   But in this and similar cases, I find strange the following line of
   reasoning :

      - Files must be written in FITS
      - My data are not compatible with FITS
      - The FITS convention must be changed so that my data can
        be enclosed unchanged 
        and the result still be called FITS, q.e.d.

   This seems to me a strange argument, like I were saying :

   - English uses the latin alphabet
   - All documents must be written in English
   - My "data" are written in Hungarian.
   - I write my document putting in each page a heading saying "what
     follows is hungarian text" followed by one page of the original
     text. Of course I use the latin alphabet.
   - The document is written in English, q.e.d.
 
---------------------------------------------------------------------------- 
       A member of  G.ASS : Group for Astronomical Software Support          
---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
----------------------------------------------------------------------------

---

From: mcs at goblin.caltech.edu (Martin Shepherd)
Date: 20 Mar 1994 03:43:43 GMT

In article <9403181638.AA11251 at xebec.gsfc.nasa.gov> arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
:I have watched the debate over the issue of introducing unsigned
:integers into the FITS Binary Tables closely and my conclusion (not
:surprising, since I started this round in the discussion) is: we
:should do it and we'd better do it quickly.

I have also been watching the debate, but have come to the opposite
conclusion. My personal viewpoint is that requiring FITS reader
software to become more complex just so that a few type-specific FITS
writers can be written more easily is totally unreasonable.

:The Binary Table standard contains a fairly large and fairly complete
:set of data types.  The criterion for defining this set cannot have
:been to arrive at a minimum set.  Considering the fact that it
:contains complex and double complex, the approach must have been to
:include all meaningful data types commonly in use among astronomers.
:I would submit that unsigned integers ought to be included in this
:collection, simply on the basis of generality.

:Frankly, I think, there is too much parochial thinking going on.

This goes both ways. Requiring that FITS be molded to your specialized
needs without regard to the problems that it would produce for the
rest of the community is surely a parochial viewpoint.

:  I
:have no intention to pick on anyone in particular, but I understand
:very well where Don Wells is coming from: in radio astronomy floats,
:doubles, complex, signed integers and scaled integers have
:traditionally dominated the zoo of data types.

This is unreasonable. It seems far more likely that the chosen types
were based on those then available in FORTRAN-77 rather than a
conspiracy among radio astronomers. While I personally prefer C, I can
see the point of not supporting types that at least one language can't
handle. Admittedly, from this viewpoint, COMPLEX types should not have
been included, but it is undeniably too late to change that now.

:[Text omitted]

:Again, if we *can* store original values in FITS tables, I feel it
:*should* be done.

In an ideal world perhaps, but in the real world it is pointless
because many architectures don't provide types with the same bit
representation as FITS decrees, and are thus unable to recover the
exact value that was written.

At a time in which object oriented programming is showing such
promise, isn't it becoming clear that the particular form in which
data are stored is secondary to the interface used to access them?
FITS allows such abstraction, and should be commended for this. If you
care about how your data is represented at the bit level then don't
use FITS - it wasn't designed for that.

:  Every extra line of code increases complexity and
:chances for errors; this is of specific concern to those among us who
:work on software that will produce basic archives: the number of
:operations should be kept to an absolute minimum, on the one hand,
:while the archived product should provide as good data access as
:possible.  Unsigned integers would help tremendously in that regard.

Again the same argument can be presented from the opposite viewpoint.
Adding new types increases the size and complexity of general FITS
readers. You are requiring that FITS readers be slowed down so
that your simpler, type-specific FITS writer can run faster. To make
this point clearer, consider that in a general FITS reader every new
type must be accompanied by at least three specialized sets of
functions:

1. Machine-specific routines to convert between each FITS type and
   equivalent host-architecture data-types. This makes it hard to
   port to new architectures - more so if there are a lot of types.

2. Conversion functions between all other compatible types supported
   by the FITS library (eg. int to unsigned int, int to float...).
   For N types there are O(NxN) such conversions required. This has a
   significant impact on speed and complexity.

3. Functions to test for and set type-specific blank/null values.

Obviously the more functions there are and the greater the number of
possible routes through a FITS reader, the more difficult it becomes to
write a general FITS library, the harder it is to test the result and
the slower it becomes to read FITS. This is the impact on the rest of
the community that I was referring to above, in turning your accusation
of parochial thinking back on you.

:[Text omitted]

:On to the XTE case.  Don Wells and Tom McGlynn have provided us with
:some quick calculations that all of us (I hope) are capable of
:carrying out but that, in their simplicity, completely miss the point.
:The problem is not in the actual scaling itself, though I might point
:out that we will decline Don's offer of his PC/XT clone, since we hope
:to be able to process a day's worth of telemetry in considerably less
:than 24 hours.  Figuring out when, where, and how to scale, involves at
:least the same amount of processing.

This makes it sound as though XTE was designed without regard for how
its output would be pipe-lined? Should the rest of the community have
to shoulder the burden of such a design flaw? Can the satellite
software not be changed to better match your archival requirements?

:More important, though, is the added complexity.  Of course, it can be
:determined what needs to be scaled by what.  And, of course, it can be
:table driven.  But we are talking about a significant increase in code
:complexity.

And should the rest of the community have to make their software more
complex so that your's need not be?

:[Text omitted]
:that capability; strict encapsulation and design simplicity have made
:this piece of software highly reliable - I would like to keep it that
:way.

I feel the same way about my general FITS library, and since it is
much easier to write a specific form of FITS than to read the general
case, shouldn't the burden be taken by the former? After all, during
data processing, FITS files often have to be read multiple times, so
speed and correctness during reading is very important to the
end-user.

:[More text omitted]
:I sincerely hope that the community will move forward with this
:expeditiously.  As Bill Pence remarked, this problem is not going
:away; therefore, the sooner we deal with it, the better.  And to
:require everybody to deal with the lack of unsigned integers in the
:future, because "we have always managed to do without" in the past,
:is, in my view, a dangerous position.

Given that computers are getting faster all the time, and that storage
facilities are getting more capacious, the argument for unsigned types
may soon be outdated. I can't see how it could get more pressing with
time. Unfortunately the same can't be said for 64 bit integers, but at
least they only represent one extra type to add.

:  - Arnold Rots

No offense intended. I just hope that you can find a simpler way to
fix XTE without requiring radical changes to FITS.

Martin Shepherd  (mcs at phobos.caltech.edu)

---

From: arots at xebec.gsfc.nasa.gov (Arnold Rots)
Date: Mon, 21 Mar 94 09:28:52 EST

Martin Shepherd posted a detailed response to my last message.  I
won't replay to every point, but there are some issues that require
clarification.

My basic point was (and is) that the need for unsigned integer types
is fairly general and will be increasing in the future.  It's going to
be easier to introduce them now than n years from today.  The XTE
project is not the reason for our wanting to have unsigned integers
added to the list: I think FITS needs them and high-lighting their
usefulness in the context of the XTE software was intended to
illustrate how and why these data types are important and will
continue to be so in other projects.  The other side of the coins,
though, is that I have no great desire to jump through all kinds of
hoops now, if unsigned integers are going to be included in FITS
anyway.

> been included, but it is undeniably too late to change that now.

Sorry if this offended you.  However, I was a lot closer to the birth
of FITS than you were.  And I wasn't saying it was a conspiracy; I
said it was designed (with an honest effort to generalize) from a
specific set of needs.  That set did not include unsigned integers,
but it did include complex floats (case in point).

> exact value that was written.

Sorry, I wrote *values*, not *bit patterns*.

> of parochial thinking back on you.

Bill Pence addressed these issues in an earlier post and concluded
they weren't as significant as you make them appear.

> complex so that your's need not be?

As I said in the introduction, I was not trying to argue that the
whole community should change its standards because of XTE, only that
many projects (XTE and the guys at FNAL are a case in point) would
benefit tremendously from this proposal.

> speed and correctness during reading is very important to the

This raises an interesting question on one's fundamental design
approach: should basic utility libraries (like FITS readers) be kept
simple so that they are reliable, but requiring the application
software that uses them to do the work and be more complex; or should
those libraries be designed in such a way that the user-software can
be simpler?  Whatever case can be made against the introduction of
unsigned integers, I don't think it could be supported by this
argument.

  - Arnold Rots

---

From: nicinski at fncasefnal.fnal.gov (Tom Nicinski)
Date: 21 Mar 1994 16:48:57 GMT

arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:

[deleted ...]

|> My basic point was (and is) that the need for unsigned integer types
|> is fairly general and will be increasing in the future.  It's going to
|> be easier to introduce them now than n years from today.

I'd also like to see unsigned integers introduced soon as part of the
BINTABLE extension, before it changes from a draft to a standard.  Letting
this issue sit and then providing another "standard" extension that is the
BINTABLE extension plus unsigned integers (and signed 8-bit integers and
possibly 64-bit integers (signed and unsigned)) would be a mistake.  Talk
about making FITS more complex:  you'd have to worry about maintaining two
extensions in synch with each other.

Tom Nicinski

+--- . .' ----+--------------------------------+----------------------------+
|   /| | |  | | Tom Nicinski / MS 234          | nicinski at fnal.fnal.gov     |
|    +-+-+\ | | Fermi National Accelerator Lab |                         _  |
| .  | | | \| | P.O. Box 500                   |                        [o] |
| `--' | |  | | Batavia, IL 60510-0500         |                        /|\ |
+---- -' -----+--------------------------------+----------------------------+

---

From: mcs at goblin.caltech.edu (Martin Shepherd)
Date: 22 Mar 1994 08:34:34 GMT

In article <9403211428.AA12356 at xebec.gsfc.nasa.gov> arots at xebec.gsfc.nasa.gov (Arnold Rots) writes:
:Martin Shepherd posted a detailed response to my last message.  I
:won't replay to every point, but there are some issues that require
:clarification.

:My basic point was (and is) that the need for unsigned integer types
:is fairly general and will be increasing in the future.  It's going to
:be easier to introduce them now than n years from today.
:...

I understand your viewpoint, but most of the responses that I have
seen contained objections to the proposal, so it was a surprise to see
anybody coming to the conclusion from the discussion, that unsigned
types should be implemented forthwith.

I do agree that, *if* unsigned types have to be added to FITS then
agreement should be reached on how to do it soon, and *if* they are to
be implemented then for the sake of conciseness and simplicity they
should be added to BINTABLE rather than by designing another derived
extension type, as one poster suggested. So far the consensus doesn't
seem to support such a change.

:>[Discussion of how adding unsigned types to FITS would complicate
   general FITS libraries more than not adding them would hurt the
   few specialized programs that find the conversion from unsigned
   to signed+offset to be a bottle-neck].

:Bill Pence addressed these issues in an earlier post and concluded
:they weren't as significant as you make them appear.

Regardless, they are more significant than simply abstracting unsigned
types via a functional interface to existing signed FITS types.

:>[Discussion of where the burden of handling unsigned types should lie]

:This raises an interesting question on one's fundamental design
:approach: should basic utility libraries (like FITS readers) be kept
:simple so that they are reliable, but requiring the application
:software that uses them to do the work and be more complex; or should
:those libraries be designed in such a way that the user-software can
:be simpler?  Whatever case can be made against the introduction of
:unsigned integers, I don't think it could be supported by this
:argument.

I agree that general FITS readers should hide as much of the
implementation as possible, but there are limits. FITS is already
fairly complicated to write a general library for and difficult to
implement correctly, efficiently, and portably with the current zoo of
ad hoc types and rules.

In places where support for unsigned types is desirable, FITS
libraries can easily use encapsulation to hide the fact that
unsigned types are implemented as a signed type and an offset, without
the user ever having to deal with the gory details. This achieves the
same effect without entailing as much extra code as handling unsigned
types at the raw FITS byte level.

My understanding is that proponents of adding unsigned types,
object to abstracting them through existing FITS types because:

1. The required offset to convert unsigned to signed, changes the
   value written.

 Why is this a problem. A simple scalar addition recovers the exact
 values that were written. By the time the user of any reasonable FITS
 reader gets the data, the values will have been restored to their
 original form.

2. It takes time to offset the unsigned type when recording it as
   a signed type.

 I find it hard to believe that this could have any significance
 compared to the overhead of file I/O.

Additionally, in the special case of XTE the argument seems to be that
it is difficult to determine what is being sent by the satellite? This
is hardly a FITS problem.