Subject: Compressed Data in FITS files?

From: job5g at algol.astro.Virginia.EDU (Jeffrey Oliver Breen)
Date: Wed, 18 May 1994 17:42:01 GMT

Hi there,

I recently had to deal with some large, sparse FITS arrays (i.e. many of the
pixels were 0).  These files turned out to be highly compressible --
up to 96% using GNU's gzip!  Since one of the goals of the FITS format
is long-term storage, I was wondering if there would be any interest
in a new, compressed array extension.  There are some great
compression algorithms out there in the public domain nowadays so
that shouldn't be a problem.

Thanks,
Jeffrey

+----------
| Jeffrey Oliver Breen                                      Ensure
| Department of Astronomy      "Even logic must give        e-mail
| University of Virginia        way to physics."            privacy:
| Charlottesville, VA 22903         - Capt. Spock,          encrypt
| job5g at Virginia.edu                  Star Trek VI          with PGP

---

From: gsh7w at fermi.clas.Virginia.EDU (Greg Hennessy)
Date: Thu, 19 May 1994 04:06:10 GMT

Jeffrey Oliver Breen <job5g at algol.astro.Virginia.EDU> wrote:
#Hi there,
#
#I recently had to deal with some large, sparse FITS arrays (i.e. many of the
#pixels were 0).  These files turned out to be highly compressible --
#up to 96% using GNU's gzip!  Since one of the goals of the FITS format
#is long-term storage, I was wondering if there would be any interest
#in a new, compressed array extension.

There has been one paper (at least) written on compressed FITS,
written by ARchie Warnock, with Barb Pfarr and Bib Hill as co-authors
in some permutation, but it is a difficult problem. IF you compress
the header, a human has difficulty determining what is in the file,
and just because a compression program exists on the hot unix box that
WRITES the files, it does not mean that a decompression program exists
on the IBM 650 that is trying to *READ* the file. It makes little use
to write a tape that others can't read. I personally almost *always*
compress my FITS disk files though, even though I find in many
circumstances that compress gets better reduction, and works much
faster than the GNU gzip program. 

Greg Hennessy

---

From: job5g at algol.astro.Virginia.EDU (Jeffrey Oliver Breen)
Date: Sun, 22 May 1994 01:16:15 GMT

In article <Cq17EA.FEE at murdoch.acc.virginia.edu>,
Greg Hennessy <gsh7w at fermi.clas.Virginia.EDU> wrote:
>in some permutation, but it is a difficult problem. 

Thanks for the pointer -- I'll take a look at their paper.

>IF you compress
>to write a tape that others can't read. 

Indeed.  That's exactly why "new FITS extension" popped into my head.
That way the header is left as is, and only the array would be
compressed.  Plus, since it could be part of the next FITS revision,
it would have to be supported on any platform boasting FITS
compatability.  If the IBM 650 supports FITS, it would have to read
compressed arrays just as it is now expected to read binary tables.
It is precisely because slick compression programs probably won't be
ported to older machines that incorporating compression into the FITS
standard is an attractive solution.

>I personally almost *always*

Well, my experience is just the reverse, but I brought up gzip not for
its performance, but for its status as free, public domain code
(unlike compress) which we could pilfer to our hearts' content.

In the realm of "lossy compression", in which you are willing to
sacrifice some information for smaller files ("quick look" images,
digital sky surveys, etc.), it seems necessary for us to design our
own algorithms anyway.  Bill Press gave an interesting talk on
wavelet-based compression algorithms a couple of years ago in which he
made the case that astronomers will never be satisfied with
"commercial" lossy compression standards like JPEG or MPEG.  This is
mostly due to the fact that astronomical images are unlike the
"terrestrial" images for which these standards are being developed:
most of the image is uninteresting (=sky background) and the brightest
pixels are _not_ flukes but rather are often the focus of interest.

While I don't propose settling on an absolute standard for all
types of data compression in astronomy right now, it might be worth
the effort to think about a framework which could be built upon later.
We could start, for instance, with a new type "Z" for compressed data
and a new FITS header like "ZTYPE".  Start off with ZTYPE = 0 for
gzip-like non-lossy compression and work from there.  Then down the
road, maybe add a ZTYPE = 1 for JPEG or 2 for FitsPress-like wavelet
compression or whatever.

Anyway, this is just a thought.  I'd be curious to hear from anyone
who's thought more about this.  In the meantime, I'll find that paper and
do some homework.

Cheers,
Jeffrey

+----------

---

From: "Selden E. Ball, Jr." <SEB at LNS62.LNS.CORNELL.EDU>
Date: Sun, 22 May 1994 07:35:40 -0400 (EDT)

job5g at algol.astro.Virginia.EDU (Jeffrey Oliver Breen) wrote

>(unlike compress) which we could pilfer to our hearts' content.

Sorry, that is not the case. gzip is "Freely Redistributable,"
not "Public Domain."  Since it is covered by the Free Software Foundation's
"CopyLeft" license, many companies selling commercial imaging products
that include Fits support would be extremely reluctant to use it.

Selden

---

From: asr at beach.cis.ufl.edu (Allen S. Rout)
Date: 23 May 1994 02:05:38 GMT

-=> Regarding: Re: Compressed Data in FITS files?: 
-=> gsh7w at fermi.clas.Virginia.EDU (Greg Hennessy) sez:
> # [ compress FITS files! gzip does wonders! ] 
>
> [ ... ] just because a compression program exists on the hot unix box that
> on the IBM 650 that is trying to *READ* the file. 

Umm.. Call me a Unix geek, but...

If you can find an architecture on which gzip will not compile succsessfully,
you'll shortly have a great deal of attention from a lot of people trying to
fix it.  In general, the FSF and the GNU project are very, very, VERY quick to
respond.

Worrying about avaliability of gzip on <platform X> should be, I contest, way
down on your list of worries.

--
<    asr at cis.ufl.edu	>| Please feel free to respond to anything I 
<     (904)373-0906	>| say: I have strong opinions, but a wide-open
< 1936 N.W. 2nd Avenue	>|  mind.
< Gainesville, FL 32603	>| BUT KEEP THE FLAMES TO E-MAIL!  (geez!)
-"My marriage is on the rocks, but only because we can't find a blanket."
<a href=http://www.cis.ufl.edu/~asr/asr.html>My Home Page</a>

---

From: job5g at algol.astro.Virginia.EDU (Jeffrey Oliver Breen)
Date: Mon, 23 May 1994 02:17:48 GMT

In article <01HCMR87X0O2A74B5I at lns62.lns.cornell.edu>,
>that include Fits support would be extremely reluctant to use it.

Thanks for pointing out my error (and thanks to those who sent me
e-mail, too).  While the gzip source code is copyrighted (or
copylefted), its compression algorithms are not patented, and that's
what really matters.  If a compressed data extension were adopted into
FITS, there wouldn't be any reason to even mention gzip in the specs.
It's the algorithms which matter, not a specific implementation of
them.

Cheers,
Jeffrey

+----------

---

From: tennant at alph.msfc.nasa.gov (Allyn Tennant)
Date: 24 May 1994 01:07:22 GMT

In article <Cq6JJ3.ALC at murdoch.acc.Virginia.EDU>  
> road, maybe add a ZTYPE = 1 for JPEG or 2 for FitsPress-like wavelet

Aaargh!  Since FITS forces you to use 80 columns to describe a
single parameter, there is no advantage to numbers as parameter
values.  Thus it would be much better to use ZTYPE='GZIP' for gzip
and ZTYPE='JPEG' for...

Since I've felt it necessary to comment on this, let me point out
that I do *not* endorse adding such compression standards to FITS.
First, anyone can use gzip (or compress) to compress files, and
likewise use zcat to view the header.  Thus the existing use of
gzip/zcat is a standard in that someone who has never heard of
FITS would be able to compress/uncompress the file.  Likewise
if you want to jpeg compress an image, you should write a *.jpg
file and there are lots of existing tools that can manipulate
*.jpg files and zero tools (to my knowledge at least :-) that can
manipulate JPEG data inside a FITS file.  There is no reason for
FITS users to reinvent these tools.

					Allyn