Subject: Comments on NOST 100-1.2

From: JENNINGS Don <Don.Jennings at obs.unige.ch>
Date: Wed, 08 Jul 1998 11:58:08 +0200

Hello all,

I have briefly reviewed the new draft FITS standard NOST
100-1.2 and have the following three comments. Since I do
not subscribe to the FITSBITS exploder or normally read
sci.astro.fits, please forgive me if my comments have
already been discussed. I ask that the NOST Technical Panel
consider my comments for inclusion into the new Standards
document.

      I. Support for 64-bit integers

I believe that this is the right time to add the 64-bit
integer data type to FITS. Reasons:

  1. Other data formats support 64 bit integers. The lack of
64-bit integers in FITS is a significant obstacle when
importing non-FITS data into FITS.

  2. 64-bit processors are now common. This implies that the
use of the "long int" data type in programs will be more
frequent. FITS must be able to store these integer values if
it is to be a usable format.

  3. 64-bit operating systems are now common. This implies
that there are quantities generated that cannot be expressed
in a 32-bit integer any longer; e.g., memory addresses, file
sizes.

  4. It is extreamly simple to do. Primary Arrays and IMAGE
extensions will accept BITPIX = 64, and BINTABLES will
accept TFORM = 'rKa'. 

  5. 64-bit integer values are at least as useful and worthy
of support as the complex data types. I personally have
never needed to store complex numbers in a FITS file, nor
known anyone with the need, but have needed 64-bit integer
support several times (.e.g., onboard spacecraft clock
times).

  6. The only reasons that I have heard agaist 64-bit
integer support essentially condense into "we can live
without it". But I claim the time has come where we cannot
live without it. I suspect that the programmers that limited
the expression of the year to two digits had a similiar "we
can live without it" argument in the 60's and 70's.
  
If the NOST Technical Panel decides not to include 64-bit
integer support in the new version of the standard, then I
ask that they at least provide their reasons for not doing
so in the new Standards document. This way history can make
them accountable for their decision.

     II. Unsigned Integer Convention

The desire for support of unsigned integers has, I believe,
been expressed before. Fortuantely, there is a way to store
unsigned integers in the current FITS standard using the
BZERO (for primary arrays and IMAGEs) and TZEROn (for
BINTABLEs) keywords. See the CFITSIO Users manual for more
details.

I request that this convention be described in one of the
appendicies of the new Standard. It could be extreamly
useful for the newer generation of FITS readers and writers
to support this convention. Note that: (1) it is already
supported by software (CFITSIO), and (2) it is already in
use within the community.

    III. Alternate Substring Array convention B.3

The array substring convention provided in appendix B3,
where substrings may be expressed as 'rA:SSTRw/nnn', should
also list the alternate convention of 'rAw' where:

r = the total length of the ASCII character field
w = the width of each substring in the character field

Note that this convention: (1) is already suported by
software (CFITSIO), (2) is already in use within the
community, and (3) is much more likely in general to be used
over the 'rA:SSTRw/nnn' convention due to its simplicity.

Thank you,

Don Jennings

-- 

------------------------------------------------------------------
If you understand what you're doing then you're not learning 
anything

Don JENNINGS                     
INTEGRAL Science Data Centre     E-mail:
Don.Jennings at obs.unige.ch 
16, Chemin d'Ecogia              Phone:  +41 22 950-91-23
CH-1290 Versoix                  Fax:    +41 22 950-91-33

------------------------------------------------------------------

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Fri, 10 Jul 1998 18:19:13 +0200 (MET DST)

> use of the "long int" data type in programs will be more

  How common ? AFAIK we have DEC Alphas (will they survive Compaq ?) and
  Sun UltraSparc (but the OS does NOT use 64-bit features, or at least so
  they told me when I inquired about migration, thinking of having similar
  problems as the Ultrix-->Digital Unix transition)

  Is INTEGER*8 in Fortran now standard ? or just supported by a few vendors ?
  (and by the way, why should one need it ?)

> in a 32-bit integer any longer; e.g., memory addresses, file

  See above for comment on how common they are. 
  The quantity you mention are not relevant talking of the CONTENT of DATA
  files.
  And I doubt they are of practical use in common conditions (I see that
  peoples in banks and similar tend to have large databases, but I ever
  doubt to need a > 2GB file  ... or to have > 2GB physical memory ... and
  since I do not have that, why should I need 64-bit addresses ? In fact
  on Alpha I compile EVERYTHING with the -taso option : Truncated Address
  Space Option, so that addresses fit into 32 bits and I have portable
  code)

> without it".

  I still believe we can.
  Unless you want to solve the problem of the inventor of the game of
  chess (if you put a grain of rice on the first square of the chessboard,
  two on the second, four on the third ...) why should one need to COUNT
  such large numbers with ABSOLUTE precision ?  REAL*8 should be enough.

  40- or 48-bit s/c clocks are the only things which comes to my mind,
  but s/c clocks are such a pain in the neck anyhow (also because they are
  unsigned :-( ). And in practice one never has so long elapsed periods or so
  fast timing that one cannot either round to "Unix time seconds" or MJD, or
  use full resolution in a limited interval, or convert to REAL*8 elapsed
  seconds.

I'm a supporter of Ockham's razor :

"file formats, data structures, data types non sunt multiplicanda praeter
necessitatem" 

>      II. Unsigned Integer Convention

Oh Yet Another Data Type !
But fortunately as you say ...

> Fortuantely, there is a way to store
> BINTABLEs) keywords. 

Thus I support the specific inclusion in the documentation of the way how to
use them.

----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR - via Bassini 15 - I-20133 Milano (Italy)      
Member of the Ockhamist's Club, together with 
William of Ockham, Guglielmo da Roveredo, Wilhelm von Eichdorf
----------------------------------------------------------------------------
For more info : http://www.ifctr.mi.cnr.it/~lucio/personal.html             
----------------------------------------------------------------------------

---

From: davis at space.mit.edu (John E. Davis)
Date: 11 Jul 1998 10:44:58 GMT

On 8 Jul 1998 09:56:51 -0400, JENNINGS Don <Don.Jennings at obs.unige.ch>
>BINTABLEs) keywords. See the CFITSIO Users manual for more

I think that this hack should be done away with in favor of real
support for unsigned values.  About the only reason I can see for not
adding such support is that older readers may have problems reading
the files with unsigned values.  But that's life--- software must
evolve.

--John

---

From: davis at space.mit.edu (John E. Davis)
Date: 12 Jul 1998 10:41:41 GMT

On 10 Jul 1998 13:06:38 -0400, Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
>  Sun UltraSparc (but the OS does NOT use 64-bit features, or at least so

  I heard that the `long' integer will be 64 bits in the next release
  of Solaris.  Is this true?

>  since I do not have that, why should I need 64-bit addresses ? In fact

  Less than 20 years ago, Bill Gates thought that no one would ever
  need 640K of memory.  Today, it is not that uncommon to see 100
  Megabyte data files.  How many people made statements twenty years
  ago about never needing 100 Megabyte files?

  If we have learned anything from the computer revolution, then it is
  that we cannot predict the future.  I have no idea whether or not I
  will be using a keyboard or a microphone in 10 years to communicate
  with my computer.

  The FITS file format is nearly 20 years old.  About a year ago a new
  FITS time specification was necessary to handle the year 2000
  problem. The fact is that several modern operating systems support
  64 bit integers and there are more on the horizon.  Let's not make
  the same mistake and ignore 64 bit data types as well as unsigned
  types.

--John

---

From: Thierry Forveille <Thierry.Forveille at obs.ujf-grenoble.fr>
Date: Sun, 12 Jul 1998 15:54:48 +0200 (DFT)

> the files with unsigned values.  But that's life--- software must
I must say I have never seen any convincing argument for supporting
unsigned values. The present "hack" as you call it offers all the
functionalities for no cost, while adding unsigned as an additional
supported format would cost significant extra code to everybody for no
additional value. To me Occam's razor says it should stay out...

64 bit integers are a whole different story, as they bring real
improvements. The question for them is not whether, but when. We
will I think have to include them, but the present revision may or 
may not be timely.

---

From: "John E. Davis" <davis at space.mit.edu>
Date: Sun, 12 Jul 1998 15:36:03 -0400

>additional value. To me Occam's razor says it should stay out...

Please correct me if I am wrong, but my main objection to this
convention is inherently non-portable because the BZERO value that must
be used is outside the range of the integer data type.  For example,
to represent an unsigned 32bit integer, BZERO must be set to
2147483648 in the fits file.  Now, BZERO is an integer and this value
is outside the range of an integer.  Why is this a problem?  It is a
problem because parsing it as a 32 bit integer may result in its
truncation. Specifically, CFITSIO uses the strtol function call to
parse these values and this function will truncate 2147483648 to
2147483647.

To see this, consider:

#include <stdio.h>
#include <stdlib.h>

int main ()
{
   long i;
   char *s;

   s = "2147483648";
   i = strtol (s, NULL, 10);
   fprintf (stdout, "%s = %ld\n", s, i);

   s = "2147483647";
   i = strtol (s, NULL, 10);
   fprintf (stdout, "%s = %ld\n", s, i);
   
   return 0;
}

Compiled with gcc on my Linux system, this produces:

   2147483648 = 2147483647
   2147483647 = 2147483647

The man page for strtol indicates:

RETURN VALUE
       The  strtol()  function  returns the result of the conver-
       sion, unless the value would underflow or overflow.  If an
       underflow  occurs, strtol() returns LONG_MIN.  If an over-
       flow occurs, strtol() returns LONG_MAX.   In  both  cases,
       errno is set to ERANGE.

So, without real support for unsiged types, I maintain that this
convention is flawed.

Thanks,
--John

---

From: davis at space.mit.edu (John E. Davis)
Date: 12 Jul 1998 21:10:58 GMT

On 12 Jul 1998 10:43:53 -0400, Thierry Forveille <Thierry.Forveille at obs.ujf-grenoble.fr>

Please correct me if I am wrong, but my main objection to this
convention is that it is inherently non-portable because the BZERO
value that must be used is outside the range of the integer data type.
For example, to represent an unsigned 32bit integer, BZERO must be set
to 2147483648 in the fits file, which is outside the range of a 32 bit
signed integer.  Why is this a problem?  It is a problem because
parsing it as a 32 bit integer may result in its truncation.
Specifically, CFITSIO uses the strtol function call to parse these
values and this function will truncate 2147483648 to 2147483647.

To see this, consider:

#include <stdio.h>
#include <stdlib.h>

int main ()
{
   long i;
   char *s;

   s = "2147483648";

   s = "2147483647";
   
   return 0;
}

   2147483648 = 2147483647
   2147483647 = 2147483647

RETURN VALUE
       The  strtol()  function  returns the result of the conver¡
       sion, unless the value would underflow or overflow.  If an
       underflow  occurs, strtol() returns LONG_MIN.  If an over¡
       flow occurs, strtol() returns LONG_MAX.   In  both  cases,
       errno is set to ERANGE.

It is possible avoid strtol with a different function, but one would
still need to perform integer multiplications which would result in
overflow.  About the only portable alternative would be to use double
precision arithmetic but even that may not be sufficient if you want
to use this convention to represent 64 bit unsigned values.

So, without real support for unsigned types, I maintain that this
convention is flawed.

Thanks,
--John

---

From: arots at xebec.harvard.edu (Arnold Rots)
Date: Mon, 13 Jul 1998 08:26:16 -0400 (EDT)

Hear, hear!
It's not so much a question of whether FITS will ever include unsigned
integer data types, but when.  Having been at the center of the
previous major effort to get them accepted, some five years ago, I 
would estimate that it's going to be another ten years ;-)

  - Arnold

> the files with unsigned values.  But that's life--- software must

--------------------------------------------------------------------------
Arnold H. Rots                                         AXAF Science Center
Smithsonian Astrophysical Observatory                tel:  +1 617 496 7701
60 Garden Street, MS 81                              fax:  +1 617 495 7356
Cambridge, MA 02138                             arots at head-cfa.harvard.edu
USA                                     http://hea-www.harvard.edu/~arots/
--------------------------------------------------------------------------

---

From: "William F. Wyatt" <REMOVEwyattTHIS at cfa.harvard.edu>
Date: Thu, 16 Jul 1998 11:39:11 -0400

[...]
>  since I do not have that, why should I need 64-bit addresses ? In fact
[...]

And I used to think 64 kilobytes of DG Nova memory was fantastic.

Here at SAO, we're working on a CCD mosaic that, if used at full
resolution, will generate about 680 MBytes per image. Thats about 340
megapixels (16-bit). If you want to mosaic several of these images into
a single file (should that seem useful at the time), it'll easily
exceed 2 GB. And that's without floating it.

-- 
Bill Wyatt (REMOVEwyattTHIS at cfa0.harvard.edu) "remove this" for email
   Smithsonian Astrophysical Observatory  (Cambridge, MA, USA)

---

From: Brian Glendenning <bglenden at cv3.cv.nrao.edu>
Date: Thu, 16 Jul 1998 10:56:41 -0600

In radio astronomy, multi-gigabyte visibility (raw interferometer) are
not uncommon for the VLBA. Large images are also not unheard of, for
example a VLA image of M33 in HI which has been created is 4096x4096x256
(16GB), and it might be reimaged at 8192x8192x256 (64GB).

Of course these examples can be represented in FITS now without a 64-bit
type.

Brian

---

From: arots at xebec.harvard.edu (Arnold Rots)
Date: Mon, 20 Jul 1998 10:39:31 -0400 (EDT)

Unless I am unusually dense (which always remains a possibility o
Monday mornings), the attached two paragraphs from Bill Pence's
message seem to be at odds with each other and do not really address
the issue John Davis was raising.

If John is right in his interpretation of what cfitsio does, things
are far from transparent to the programmer and we have a serious
problem - a problem that cannot be dismissed with "it's your
compiler's fault".  I tried his little test on a Sparc, using gcc
and, not surprisingly, it failed in the same way.  Hence, using gcc on
a 32-bit machine (hardly an uncommon combination) will get one into
trouble.

It means that anybody using the unsigned integer facilities in cfitsio
is at risk: all purportedly unsigned longs will come out one too small
on a common platform-compiler combination.  The fact that we have not
heard any complaints probably has to do with the rather recent
introduction of this feature which undoubtedly has not made it into a
lot of application code.

It would seem prudent, though less elegant, to test whether
BZERO/TZERO represents the signed/unsigned hack (yes, it _is_ a hack)
and then simply flip the most significant bit.

Could Bill comment onm this?  Do we have a serious problem or do we
misunderstand the way things are handled in cfitsio?

Thanks,

  - Arnold Rots

William Pence wrote:
> The more important issue for programmers, I believe, is that standard
> FITS software like the CFITSIO library should provide a transparent
> interface for reading and writing unsigned integers in FITS files.
> CFITSIO provides a whole family of routines for reading or writing data
> of any supported datatype, including unsigned short, unsigned int, and
> unsigned long (in the C language interface) to FITS images or tables. 
> When using CFITSIO, the programmer does not need to be concerned in any
> way with how the data values are internally represented in the FITS
> file.  All the business with setting or reading the BZERO keyword value,
> etc. is all handled internally by the interface routines.
> 
> Finally, John Davis raised the objection that the BZERO keyword =
> 2147483648 that is used to represent unsigned 32-bit integers in FITS is
> not a valid 32-bit signed integer with some particular compilers (it is
> a valid signed integer value with many other compilers, however).  This
> is not a problem with the FITS format itself, but instead just
> illustrates that the implementation of FITS readers and writers on any
> given platform must be able to deal with the limitations of that
> platform.  The similar sorts of issues have to be addressed when
> reading/writing FITS files in Fortran, which doesn't even support an
> unsigned integer datatype at all.

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Mon, 20 Jul 1998 15:55:18 -0400

Regarding Arnold's questions, I have checked the CFITSIO code and run some
test programs to verify that CFITSIO does read and write unsigned integers
correctly.  Internally, CFITSIO reads and writes the TZEROn or BZERO keyword
as a double, not as an integer, so CFITSIO is unaffected by the supported
range of the integer datatype in this case.  Also, as Arnold suggested,
CFITSIO does test for the special values of BZERO or TZEROn that represent
unsigned integers, and then just flips the sign bit, instead of adding
BZERO, for better reading or writing efficiency. In summary, then, there is
no problem using CFITSIO to read or write unsigned integers in FITS file.

John Davis' example, does however show that CFITSIO currently does not check
for overflows when reading keyword values (for both integers or reals). 
Thus, for example, when reading the BZERO = 2147483648 keyword as a signed
4-byte integer, the value may be silently truncated to the maximum value of
a signed integer (this behaviour is compiler dependent).  Similarly, trying
to read a real keyword with a value = 1.5E+32 as an integer, could also
return a value = 2147483647 without any errors.  This behavior is not what
most users would want or expect, so the next release of CFITSIO will contain
more range checking code that will return an error status whenever an
overflow occurs while reading keywords.

-Bill Pence

> misunderstand the way things are handled in cfitsio?

____________________________________________________________________
Dr. William Pence                          pence at tetra.gsfc.nasa.gov
NASA/GSFC Code 662         HEASARC         +1-301-286-4599 (voice)     
Greenbelt MD 20771                         +1-301-286-1684 (fax)