Subject: Re: Packing floating point data using HDF

From: [email-masked] ()
Date: 24 Feb 1995 15:19:59 GMT

In article <[email-masked]>, [email-masked] (Jim Rosinski) writes:
|> 
|> I would like to pack my floating point SDS data generated by HDF on a UNICOS
|> operating system.  Obviously you can get effective 2 to 1 packing by
|> specifying DFNT_FLOAT32 when the data get written if you're willing to live
|> with the roughly 7 decimal digits of precision that this provides (Crays use
|> a 64-bit floating point representation).

Be careful with this!  If you have a float64 array which you supply it to the
HDF library and claim that it's a float32 array, HDF will *NOT* convert your
data from 64 to 32 bits.  It will simply take each consecutive 32 bits and
interpret them as a float32.  So, if you have a 100x100 array of float64
(that's 100 x 100 x 8 = 80,000 bytes) and tell HDF that it's a 100x100 array
of float32 (100 x 100 x 4 = 40,000 bytes), HDF will store the first 10,000
32-bit "words" into your SDS.  The result is that half your data has just
disappeared.  The way to safely accomplish your 2:1 compression is to use
a loop to assign and coerce each element of your float64 array to an element
of a float32 array.

|> Does HDF provide the capability to
|> pack/unpack the data, with (obviously) some loss of precision which provides
|> even more than this 2 to 1 factor?  Standard compression utilities which
|> provide no loss of precision (ex: gzip) are useless on the type of data which
|> I'm generating--there aren't many repeated values.

The current version of HDF does not provide any compression, as such, for
SDSs.  Version 4.0 (currently in alpha release) will provide some compression
techniques for SDSs.

What you *can* do with the current version is to "de-calibrate" your data
>from a 64-bit float to a 16-bit integer and store an int16 SDS with standard
"calibration" information (through a routine called SDsetcal).  The calibration
information includes a scale, offset, and resultant data type.  Using the
formula given in the HDF Reference Manual, you can then convert the int16
data set into the resultant float64, using the stored scale and offset.

Note that HDF does not do the scaling operation for you.  It just provides
a standard way of storing the parameters for the scaling operation.

Of course, this strategy will only make sense if your data can stand some
loss of precision *and* dynamic range.

			-Doug Ilg
			 Hughes STX