Subject: Re: Cray performance of HDF, netCDF, etc.

From: [email-masked] (Benjamin Z. Goldsteen)
Date: Fri, 11 Jun 1993 06:09:45 GMT

[email-masked] ( Don Dovey ) writes:

>I'm defining a data format for finite element analysis results
>on unstructured grids.  The analysis codes currently write flat
>(non-portable) binary files.

>I ran some quick timings, writing large arrays to disk, to compare 
>the HDF Vset interface with a straight unformatted FORTRAN write
>statement on a Cray YMP.  The HDF output was several hundred times
>slower (HDF has to convert the data to 32 bit IEEE).  I noticed
>that in the new beta release of HDF, the conversions are supposed
>to have been sped up by a factor of 2 or 3, so let's say the HDF
>write is a hundred times slower.

>Are these timings in the right range, and does netCDF (using Sun's
>XDR) have a similar performance on the Cray?

>A factor of one hundred would impact the I/O performance of our
>analysis codes.  From a performance standpoint, it would be more
>desirable to write the data in the native format and then be able
>to convert it if it is moved to a workstation (disk space is not
>a limiting factor).  On the other hand, HDF's Vset and netCDF
>provide the perfect level of abstraction for the kind of data
>storage we want to do.  PDBlib, which provides a machine-targetable
>portable data format, is more low-level and would require more work
>to develop an interface on top of it.

      I believe you can have HDF write FP data in native Cray format. 
However, only Crays will be able to understand the data (unless this has
changed recently).  Perhaps you could write Cray FP HDF files and then
write a little filter that just read a Cray FP HDF files and outputed an
IEEE FP HDF files.

>So, has any one else had to deal with performance issues on a Cray
>with these data formats?  Any helpful input?

      The conversion to IEEE is going to be fairly expensive on the Cray
(I wonder if it vectorizes...).  I noticed that HDF was expensive
compared to standard I/O in my program, but I/O accounted for very
little of my program that it didn't matter (it went from like 0.0001% to
0.01%).  I would imagine the situation would be different for people
with 8 GW CFD codes.  
-- 
Benjamin Z. Goldsteen

---

From: [email-masked] (S. A. Levin [Stewart])
Date: 11 Jun 1993 21:20:08 GMT

In article <[email-masked]>, [email-masked] (Benjamin Z. Goldsteen) writes:
|> [email-masked] ( Don Dovey ) writes:
|> 
|> >Are these timings in the right range, and does netCDF (using Sun's
|> >XDR) have a similar performance on the Cray?
|> 
|> >A factor of one hundred would impact the I/O performance of our
|> >analysis codes.

At the Stanford Exploration Project, Dave Nichols has rewritten some of the
xdr ieee float conversion routines in order to get acceptable performance
on converting large volumes (100's of megabytes) of seismic data.  The basic
xdr package handles data a byte at a time, calling several layers of subroutines
to retrieve, assemble, and convert each data item of a vector.  This is where
the factor of 100 comes in, I believe.  I do not know how much Cray has
optimized their implementation.
 
|>       The conversion to IEEE is going to be fairly expensive on the Cray
|> (I wonder if it vectorizes...).  I noticed that HDF was expensive
|> compared to standard I/O in my program, but I/O accounted for very
|> little of my program that it didn't matter (it went from like 0.0001% to
|> 0.01%).  I would imagine the situation would be different for people
|> with 8 GW CFD codes.  
|> -- 
|> Benjamin Z. Goldsteen

I have worked on vectorized floating point conversion on the Cray and other machines
and it should be in the noise level (perhaps 5 machine cycles per sample) when done
right.  Cray's library routines do a good job, though there's room for improvement,
but would not perform well if called separately to convert each sample one at a time.

Stewart A. Levin, Mobil  [email-masked]

---

From: [email-masked] (Maurice J. LeBrun)
Date: 13 Jun 93 01:40:49

There is one technique for reducing overhead that I haven't seen mentioned
here.  It works if your data is fairly band-limited (if random I wouldn't try
it), and as a bonus can even reduce your data set size by some limited
factor (e.g. 2x smaller).  I think it works well for most scientific data
situations if you choose your record read/writes appropriately.  Here is a
description (in TeX) I wrote for it a while back:
---

The compression of floating point data used is simply bit-stripping, with a
twist.  Each record write specifies the number of bits to use for storing an
individual data point.  The minimum and maximum for the range is determined,
and the data is mapped onto the range
\begin{verbatim}
    f = (unsigned long) (f - fmin)/(fmax - fmin) * (2^nbits - 1)
\end{verbatim}
This reduces the data to the form:
\begin{verbatim}
    data record:  fmin, fmax, [integer data]  
\end{verbatim} where the integer data is in the range $[0, 2^{\rm nbits}-1]$.
Thus, even without compression, we have reduced the problem of how to
portably write the data to the writes of {\tt fmin} and {\tt fmax}, since the
integer data can be portably written as a byte stream.  We then write {\tt
fmin} and {\tt fmax} as IEEE 32 bit floating point numbers, using standard C
functions.  Although there is significant overhead in this conversion, it
amounts to a negligible overall expense since it is done only twice per
record written.  In principle, one can use any desired number of bits for
storing the data, although 16 bits or 32 bits are the easiest to implement.

You may wonder if a 16 bit representation is acceptable for scientific data
sets.  The answer is a qualified yes, in fact, even fewer bits may be
enough.  The reason for this is that many measured quantities have a fairly
limited dynamic range.  If, for example, we are measuring the coordinates
of a particle in a box, there is a definite physical limit on the
magnitudes, and typically a finite resolution which we are interested in
observing.  In this case 16 bits is typically enough for accurate
diagnostics with a fractional resolution of 1/65536 or 0.000015.  The
shorter the record length, the higher the precision, since {\tt fmin} and
{\tt fmax} are essentially exact.  However, beware of too short of record
lengths, since the write of {\tt fmin} and {\tt fmax} involves overhead in
both disk and cpu usage.  For example, given 32 bit data written using 16
bits, the record length vs the {\em actual} reduction in size for a few
cases is listed below (note in addition to writing {\tt fmin} and {\tt
fmax} we need to write {\tt nbits}):
\begin{verbatim}
        length          compression
        ------          -----------
           8                .78
          16                .64
          32                .57
          64                .54
         128                .52
\end{verbatim}

Note that saying that 16 bits is enough for data representation is {\em not}
the same as saying that 16 bits is adequate for numerical calculation.  This
is because the accuracy required by diagnostics on the data is typically much
less than that required of the computation itself.  Thus, to use this type of
compression effectively, you should constrain your writes to those of limited
dynamic range, while being long enough (approx 32 words or more) to avoid too
much overhead.  As an example in limiting the dynamic range, in the
previously mentioned coordinate problem, one would {\em not} write out each
$(x,y,z)$ triplet as part of the same record.  Instead, the best way is to
write out $(x)$ for all particles, then $(y)$ and then $(z)$.

---

OK, end of previously-written description.  Obviously my interest at the time
was a cheap way of reducing data requirements.  But it works at reducing cpu
overhead for large i/o transfers too (previously mentioned caveats apply).
The drawbacks are that: (a) it really needs to be written in C not Fortran,
(b) your application requires a filter to transform the data into something
that other codes will recognize, and (c) it may not be clear a priori how
much accuracy to retain.

--
Maurice LeBrun    [email-masked]
Institute for Fusion Studies, University of Texas at Austin

Faire de la bonne cuisine demande un certain temps.  Si on vous fait
attendre, c'est pour mieux vous servir, et vous plaire.
                                [menu of restaurant Antoine, New Orleans]

---

From: [email-masked] (Dave Nichols)
Date: 17 Jun 1993 23:28:43 GMT

In article <[email-masked]> [email-masked]  writes:

>   xdr package handles data a byte at a time, calling several layers of 
>   subroutines  to retrieve, assemble, and convert each data item of a 
>   vector.  This is where  the factor of 100 comes in, I believe. 
>   I do not know how much Cray has  optimized their implementation.

Actually I didn't rewrite the xdr routines I just changed what our own
I/O routines called.

If you do I/O to a "FILE*" the standard, portable, xdr distribution
converts one float at a time and then uses putc() 4 times to write the
bytes. On many systems this cost in not significant compared to the
calculation and I/O time, on the Cray you really notice it.

My first attempt to overcome this was to read large blocks of data myself
and then use the xdr routines to convert from memory to memory. This 
improves the speed acceptably on some systems. On the Cray this is no good
because the xdr routines aren't vectorized so I replaced them with the
Cray library routines (IEG2CRAY and CRAY2IEG). I dislike having special
cases in the code but the effort was worth it this time.

Here are some approximate I/O rates for writing 10M of floats to disk on a YMP.
It is writing through the SSD so I/O rates are pretty good.

Raw I/O of cray floats 10MW floats = 80MB
    ~1.4 MW/s

I/O in ieee format using xdr_vector() to an XDR stream that uses FILE* I/O
10MW floats = 40MB in ieee format.
    ~0.14 MW/s !

I/O in ieee format using xdr_vector() to an XDR stream that writes to memory
then raw I/O to disk.
    ~0.16 MW/s 

I/O in ieee format using using CRAY2IEG to convert data and then raw I/O
    ~1.3 MW/s

I am prepared to pay a 10% penalty for having it in ieee format,
especially since it takes half the space, but I am not prepared to pay
a 1,000% penalty. If you want fast I/O in portable format from a Cray
you seem to need to use their conversion routines.
I don't know if the netCDF guys have made any optimizations like this.

In an ideal world Cray would modify their library version of the xdr
routines to use the vectorised conversion routines. Then we would
have the nice uniform xdr interface and reasonable performance. 

-- 
Dave Nichols, Dept. of Geophysics, Stanford University.
[email-masked]