Subject: VAX R*8 to FITS/IEEE R*8 conversion problem

From: jennings at antwrp.antwrp.gsfc.nasa.gov (Don Jennings)
Date: Thu, 17 Dec 1992 13:54:50 GMT

My organization often needs to convert data that is native to the VAX
architecture into a FITS format. Since the VAX real*8 bit pattern is different
>from the FITS/IEEE real*8 bit pattern, it is not possible to completely
encapsulate a VAX real*8 in a IEEE real*8; specificly, a loss of precision
occurs when transforming a VAX real*8 into an IEEE real*8.

One solution we have devised is to save the 7th byte of VAX real*8 values, 
where the loss of precision occurs, and add them to a column of the binary
table where the real*8 values are being stored. The following is a suggested
list of rules for the extra column:

   It would always be the last column of the table.

   Each column entry would contain the 7th byte of all real*8 values
   occurring in a given row. Thus, if a table row contained 5 real*8
   values, then the extra column's TFORM code would be 8B, if the
   row contained any real*8 variable length array entries then the
   extra column's TFORM code would be PB(XXXX).

FITS readers residing on non-VAX architectures would be able to ignore the
extra column, while VAX FITS readers could completely reconstruct the original
real*8 value by reading the extra column's values.

Has anyone out there had this problem? Does anyone out there have a 
different solution? If so, then I would like to hear from you. Please 
send/post any general comments on the above scheme you might have as well. 

                    Don Jennings, COSSC/GSFC

---

From: pence at tetra.gsfc.nasa.gov (William Pence)
Date: Thu, 17 Dec 92 11:30:58 EST

>    values, then the extra column's TFORM code would be 8B, if the

I think you mean 5B, not 8B  (?)

> real*8 value by reading the extra column's values.

My reaction to this is that it is not very useful to put information in a FITS file
that only a particular machine will be able to interpret.  One of the main purposes of 
FITS is to make the information in the file portable to any machine.  So I think
you have 2 choices.

1.  If this added precision in the VAX value is really significant, then you 
need to find a portable way of including it in the FITS file.  One way might be
to express the double precision value as 2 columns:  one containing the integer
part, and the other containing the fractional part of the value.

2.  If the added precision in the VAX value is not really significant (the VAX
G_floating datatype has effectively 4 more bits of precision than an IEEE R*8 value)
then one should not worry about it, and one should accept the fact that the
FITS file cannot preserve every last bit of information on every type of machine.

As an aside, I'm sure you are aware that the VAX has 2 different R*8 data 
representations.  The default D_floating representation which you are probably using
reserves 8 bits for the exponent and 55 bits for the mantissa.  The G_floating
representation on the otherhand is similar to the IEEE R*8  representation: both
have 11 bits for the exponent and 52 bits for the mantissa.  The G_floating
and IEEE values are still not identical though because the VAX interprets the
exponent and mantissa numbers as normalized, ("hidden bit normalization") which
effectively gives you another bit of precision compared to the IEEE value.
If you were to use G_floating on the VAX, you probably would not notice the
loss of precision in converting the data back and forth from FITS.

-Bill Pence

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Thu, 17 Dec 92 17:58:58 ITA

On Thu, 17 Dec 92 11:30:58 EST William Pence said:
>FITS is to make the information in the file portable to any machine.

 I would tend to agree with Bill on this.

>
>1.  If this added precision in the VAX value is really significant, ....
>2.  If the added precision in the VAX value is not really significant ...
>

  I had recently the converse concern of this, that is not about the
  added precision in the Vax representation, but the fact that the IEEE
  representation for REAL*8 allows a wider range of exponents, larger than
  the 1.E-38 to 1.E38 allowed by usual (D_FLOAT) on Vax
  (as clearly stated by Bill here below)

>loss of precision in converting the data back and forth from FITS.

  I recently was engaged in setting up something able to exchange files
  between Vaxes (2-complement byteswapped integers, Vax format REAL*4 and
  REAL*8 D_FLOATs), Suns (2-complement integers, IEEE REAL*4 and REAL*8) and
  DECstations (2-complement byteswapped integers, byteswapped IEEE REAL*4 and
  *8). The idea was to copy the files and convert them in place without
  passing through an intermediate FITS file.

  The idea is to use some simple routines, and to change the file in place
  on the destination machine (therefore only a conversion foreign-->local
  is needed on each operating system)

  The byteswap can be easily handled in Fortran in a portable way. For the
  conversion from IEEE to Vax format on VMSI use some assembler routines kindly
  provided by Bill in his FITSIO packages (on copyright from the IRAF team).
  The R*4 Vax to IEEE can be easily handled in Fortran on Unix, due to the
  fact that all what is needed is to shuffle bytes and adjust the exponent
  bias.  I have for the moment suspended work on the R*8 Vax to IEEE
  conversion because I need to clear my mind.

  The conversion from G_FLOAT would be analogous to the R*4 one, but the
  one for D_FLOAT (the reverse of the ieeed routine in the IRAF VMS assembler
  routine) is not obviously done in Fortran.
  What I am uncertain of, is whether I should continue using D_FLOAT on Vax
  or move to G_FLOAT.

  I actually use quite a limited amount of REAL*8 values, and I was always
  happy with applying some scale factors when exceeding the exponent range
  (1.OE38 for both R*4 and D_FLOAT R*8). From the quoted assembler routine
  I derive that IRAF does the same.

  I would appreciate to know other people's opinion on the matter
  whether G_FLOAT should be recommended on VMS for IEEE compatibility,
  or not.

-----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |
Bitnet:   EXOSAT at IMISIAM                   |
-----------------------------------------------------------------------------

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Thu, 17 Dec 1992 19:53:00 GMT

In article <9212171724.AA01436 at fits.cv.nrao.edu>, Lucio Chiappetti <lucio at ifctr.mi.cnr.it> writes...

	(stuff deleted)

>  whether G_FLOAT should be recommended on VMS for IEEE compatibility,

I believe that G_FLOAT and H_FLOAT (REAL*16) are not supported on the new Alpha
machines.

We use IDL, which doesn't give the user the option of selecting D_FLOAT or
G_FLOAT (and doesn't support quadruple-precision).  On the VAX, it uses D_FLOAT
numbers for double precision.  (As an aside, IDL does provide built-in support
for converting between IEEE and host formats, for all datatypes it supports.)

Bill Thompson

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Thu, 17 Dec 1992 20:04:00 GMT

In article <JENNINGS.92Dec17085450 at antwrp.antwrp.gsfc.nasa.gov>, jennings at antwrp.antwrp.gsfc.nasa.gov (Don Jennings) writes...
>   It would always be the last column of the table.

This statement alone raises warning flags in my mind.  I'm firmly of the
opinion that the structure of binary tables should not in any way depend on the
order in which the columns appear.  If you're going to do something like this,
you should indicate the column with an appropriate label (TTYPE).

>send/post any general comments on the above scheme you might have as well. 

If that last byte is important, than it will be important regardless of what
computer you're analyzing the data on.  If so, then a better, more
machine-independent way of transferring the full significance of the data has
to be devised.  If it isn't important, then one shouldn't worry about it.

Bill Thompson

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Fri, 18 Dec 92 09:16:13 +0100

I received the mail below from Thierry Forveille privately, but then we
both considered it could be of interest to the list, so I am forwarding
it below.
                             Lucio Chiappetti

----------------------------Original message----------------------------
We actually have available some fortran code that performs the transformations
you need and which I can send you if you like. It was developped for exactly
the kind of application you contemplate, transparent access to data files
accross an heterogeneous network. In addition to what you request, we also
support conversion to non-native formats since we also want to modify files
on other machines, not only read them.
The code is written in fortran (I agree the R*8 conversion is non trivial!)
and performs all possible conversions between Vax and both flavours of IEEE
(little and big endians), and has versions for all three types of hardware
(for efficiency, it uses integer multiplications instead of bit shifts where it
can). All IEEE special values are handled (infinities, NaNs...) are handled,
and converted to a user specified blanking value. These routines have been
extensively used for about two years, and work without user complaints...

        Thierry Forveille
        Observatoire de Grenoble
        forveill at gag.observ-gr.fr