Subject: 64-bit integers in FITS

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 13 Jul 1998 16:25:33 -0400

I basically have an open mind about the suggestion to support 64-bit
integers in FITS files, but just wanted to make a couple comments:

- it would be a fairly trivial job to modify the FITS standard to
specify the format for 64-bit integer images and binary table columns. 
Images would have BITPIX = 64, tables would have TFORM = 'K' (or
whatever), and the binary data itself would be written as
twos-complement signed binary integers containing 8 bytes.  Not much
more needs to be said.  There is no restriction on the value of integer
keywords, so they already support 64-bit integers (but most
readers/writers only support 32-bit integers).

- supporting the reading or writing of FITS files containing 64-bit
integer data on 32-bit platforms would likely require a lot of work.  If
the hardware/compiler doesn't support reading and writing 64-bit
integers then interface software like CFITSIO would have to have
software emulation routines to do datatype conversion between 32-bit int
<-> 64-bit int, 64-bit float <-> 64-bit int, etc. and also trap any
overflow conditions.  Presumably there already exist routines to do all
this, written in C, that could be used in CFITSIO.  None the less,
adding support for 64-bit integers in CFITSIO, not to mention all the
other existing FITS readers and writers, would be a big job, and would
make the interface software even more complex than it already is.

- There would still remain the problem that it is not possible in
principle to fully represent 64-bit integers on a 32-bit machines
without possible overflow (when converting to 32-bit integers) or loss
of precision (when converting to double precision floating point).  Thus
one could quite easily write FITS files on 64-bit machines that could
not be read on 32-bit machines.  This would defeat one of the main
purposes of FITS to provide a machine-independent data interchange
format.

____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)

---

From: Peter Bunclark <psb at ast.cam.ac.uk>
Date: Tue, 14 Jul 1998 08:27:20 +0100

Isn't the fundamental point that FITS is a data-interchange medium?
Hence, if someone happens to have 64-bit data, they will have a need
to move it between machines and would probably prefer to have used
FITS.  It's certainly not in FITS's remit to declare to
instrumentatation
groups that they are not allowed to invent 64-bit detectors.

Such a hypothetical group would have to invent a cooked-up data 
format of their own;  if it was me I'd put BITPIX=64 in the knowledge
that everyone would know exactly what I meant.  Why not enshrine it
in the standard, perhaps with the advice to use it only when the
data genuinely requires such a precision?

Peter.

---

From: Tim Pearson <tjp at astro.caltech.edu>
Date: Mon, 13 Jul 1998 21:34:30 -0700 (PDT)

Like William Pence, "I basically have an open mind about the
suggestion to support 64-bit integers in FITS files."

But I would like to remind people that the reason for doing so should
not be because 64-bit integers are convenient to use on some computer
systems, but because there is a need to interchange data that cannot
be represented in any of the current data types (BITPIX=8, 16, 32,
-32, or -64). I can't think of any astronomical data that require such
precision, but perhaps that just reflects the poverty of my
imagination.

If we do add BITPIX=64, perhaps we should also consider intermediate
types like BITPIX=40, 48, etc. which could also be accommodated in the
standard FITS structure and which would save space in the file when
full 64-bit precision is not required.

I am surprised that many people appear to think that FITS is designed
for interchanging computer bit patterns. It is not: it is for
interchanging images (and now tabular data), using as few different
conventions as possible.

This misunderstanding perhaps explains why someone might think that
"BSCALE = 2.0" means something different from "BSCALE = 2". BSCALE
specifies a mathematical operation to be performed on the data array
to construct the image pixel values; how it is done, and with what
precision, is up to the reader. There is certainly no implication that
32-bit (say) floating-point arithmetic should be used if that might
lose precision.

My interpretation of the NOST draft standard (Sec. 5.2.4) is that both
"BSCALE = 2" and "BSCALE = 2.0" are allowed, even though BSCALE is
specified to have a "floating point" value (5.4.2.5). They should be
interpreted identically.

Tim Pearson
Astronomy Dept, California Institute of Technology

---

From: Tom McGlynn <tam at silk.gsfc.nasa.gov>
Date: Tue, 14 Jul 1998 13:26:00 -0400

When I wrote a Java FITS reader last year I was considering whether I should
support 64 bit integers and I asked whether there was any need for it.  The response
was underwhelming, but I went ahead and did it anyway.
    (See http://members.home.net/mcglynn/java/nom/tam/fits).

A few points:

   It's very clear how the standard should support this data type, e.g., BITPIX=64
for image data, and TFORM='K' for binary tables.  (16 byte integers will be
a little harder since 'L' is already taken)  This makes adopting it a lot easier.

   I have thought of a few things that people might want to store in 8 byte integers
today or soon.  Presumably this number will increase with time.
For some double precision numbers are not an acceptable alternative.
Note that while we do have ways to indicate that real data is being
encoded as integers  (BZERO/BSCALE, ...) we do not have the inverse: that
value encoded in a double is really an integer.  That's an important thing
to know about an item.

Some things that might need 8 byte integers:
  - As mentioned previously clock times frequently require more than 32 bits.
  - File sizes
  - Verification/encryption keys, e.g., an 8 byte checksum.   Possible rouding errors
     mean that these cannot be stored as doubles.
  - Catalog IDs (e.g., the Monet catalog already has 270 million entries).  Its very useful
     to integer ID's for catalogs but some of the next generations of catalogs will have more than
     4 billion entries.
  - System data (e.g., the one suggestion I had last year was that databases often use
    8 byte quantities).

With regard to Tim Pearson's question of whether we should implement intermediate sizes, the
historical precedent seems to be no:  FITS does not support 24 bit ints.  I think there
are some other problems:  What granularity to support?  What's the format code
in binary tables?    More importantly FITS seems to have evolved such that its
internal formats are compatible with those found on typical hardware.  It would
be a lot more painful to support these intermediate types (e.g., to hande the sign
extension, packing and overflow issues).

There is the question of how to support i*8's on machines which don't have that.
It should be very straightfoward to convert from i*8 to r*8 (not too much worse
than the signed to unsigned conversions) but that won't always be good enough.
This is the one issue that I'd like to understand better before going ahead and
recommending adopting i*8.  What fraction of the community cannot read i*8's?
E.g., do gcc and g77 support it?

            Tom McGlynn

---

From: sla at ucolick.borg (Steve Allen)
Date: 14 Jul 1998 11:00:06 -0700

The performance increase for, e.g., database applications experienced
with Dec Alpha processors is rapidly driving Sun, Intel, HP, and
others to produce hardware and software with 64-bit capabilities.

In article <35AA6D3D.44297AB4 at tetra.gsfc.nasa.gov>,
>specify the format for 64-bit integer images and binary table columns.

It should be noted that introduction of 64-bit integers will probably
create demand for enhancement of the TFORMn = 'P' (Array Descriptor)
data type.  The Appendix for BINTABLE currently suggests this for a
length and offset into a heap following a table.  It's easy to imagine
heaps of more than 2**32 bytes.

Perhaps something as simple as a new TFORMn = 'Q' (64-bit Array Descriptor)
which consists of two 64-bit integers would do the job.

-- 
Steve Allen          UCO/Lick Observatory       Santa Cruz, CA 95064
sla at ucolick.borg     Voice: +1 831 459 3046     FAX (don't): +1 831 459 5244
WWW: http://www.ucolick.borg/~sla               PGP public keys:  see WWW
Junk mail is irrelevant -- my return address has been assimilated.

---

From: seaman at noao.edu (Rob Seaman)
Date: 14 Jul 1998 22:10:06 GMT

> value encoded in a double is really an integer.  That's an important thing

Well, BSCALE and BZERO were originally designed for packing reals into
integer pixels before the floating point convention, but they now can
also mean that 16 bit integers are unsigned, so FITS has no reliable
way to indicate that the original datatype was different than the FITS
datatype in either direction.  This seems like a pretty esoteric thing
to rely on knowing in any event.

>   - As mentioned previously clock times  [...]
>   - File sizes
>   - Verification/encryption keys, e.g., an 8 byte checksum.  [...]
>   - Catalog IDs  [...]
>   - System data  [...]

However, can anybody suggest a current need for images with 64 bit
pixels?  ...and do tables and images have to support the same set of
datatypes?  "Exotic" image data arrays are likely to be written as
binary tables anyway.

On the other hand, 64 bits may not be enough for some of the applications
listed above.  Encryption keys and digital signatures, for instance, can
require hundreds or thousands of bits (and a proportionately large number
of ASCII digits or characters).  A character string may be the only thing
that suffices for table fields (or header keywords) that require extended
precision.

> With regard to Tim Pearson's question of whether we should implement
> intermediate sizes, the historical precedent seems to be no:  FITS does
> not support 24 bit ints.

Note that the hubbub about unsigned integers is also something of a
historical artifact.  If a signed integer datatype is large enough to
contain the entire range of the unsigned integers being generated (e.g.,
packing 15 bit unsigned data into 16 bit signed pixels), then there is
no problem - other than some slight overhead that may perhaps be involved
in this easy type conversion.  And if the signed datatype is not large
enough (trying to fit 18 bits into a short integer), then some higher
precision datatype is needed entirely.

The main reason for worrying about unsigned integers is that the
affordable state of the art for A/D converters is 16 bits and we don't
want to waste the high order bit.  Folks building instruments with
18 bit A/D's won't be concerned with packing them into short integers
using BZERO.

Perhaps rather than worrying about unsigned integers (or 24 bit ints or
whatever), we should get back to devising a speedy FITS compression scheme
that would allow using 32 (or even 64) bit integer pixels for intermediate
precision data in a space-efficient manner.

Rob Seaman
-- 
seaman at noao.edu, http://iraf.noao.edu/~seaman
NOAO, 950 N Cherry Ave, Tucson AZ 85719, 520-318-8248
PGP: 98 8D 8B 49 74 9A 41 88  3A 43 87 54 51 BF 30 4B

---

From: davis at space.mit.edu (John E. Davis)
Date: 15 Jul 1998 17:39:40 GMT

On 14 Jul 1998 09:51:10 -0400, Tim Pearson <tjp at astro.caltech.edu>
>interchanging images (and now tabular data), using as few different

   My main concern is that FITS is not being used to simply exchange
images between systems (which it does well), but is used as a general
purpose data archive format.  I have no problems with this concept as
long as the file format is well defined in the absence of a reader.  I
remain unconvinced that this requirement is fulfilled.

   If FITS should not be used as a general purpose archive format,
then it would be nice if the scope of the file format was spelled out
in the NOST document.  Some people believe that all programs used for
processing astrophysical data should read and write FITS files and
communicate with each other using such files.  I believe that this
practice goes beyond the original scope of the FITS file format.

--John

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Wed, 15 Jul 1998 16:51:29 -0400

>From the implementation point of view, a FITS image is just a special
case of a table with 1 row and 1 vector column.  So if one is going to
support 64-bit integer table columns, then very little else needs to be
added to support 64-bit images.