Subject: compression of fits

From: veran at cfht.hawaii.edu (Jean-Pierre Veran)
Date: Thu, 3 Oct 1991 00:09:13 GMT

The Canada-France-Hawaii Telescope has large format CCDs of 2K by 2K pixels
(2 bytes per pixel even if only 15 bits are used at the moment). An average of
100 images are taken each nignt ie almost 1 gigabytes each night.
The 8 megabytes images are stored in FITS format on optical disk, after being
compressed with the Unix COMPRESS program. The performances of COMPRESS are
generally poor so we decided to initiate a research project to find out how it
would be possible to improve the compression ratio of our FITS files. I am the
one in charge for that.
First I tried compression programs available in the public network,
(COMPRESS 4.1, COMPACT 1.0, ARC 5.21, LHARC 1.0, ZOO 2.10, FREEZE 2.2, ZIP 0.9)
on a bunch of objects, flats and bias images.
I have found that the best was generally COMPACT 1.0, using adaptative
Huffman coding.
The images are generally very noisy and I have found that I can improve
considerably the compression if I get rid of a certain number of the least
significant bits, then use COMPACT 1.0 to compress the remaining most
significant bits and finally just append the least significant bits to
the file,
without trying any compression on them. The problem is that the optimal
number
of LSB to get rid of depends on the image and I'm trying to work out a
criterion
based on simple statistic properties. This seems not very difficult,
needs some
more calculation but only during compression, not during decompression
and it is
possible to perform these calculations only on a part of the image, the
central
part for example.
Before compressing the MSB, I tried to use a linear predictor. The
results show
that the compression ratio is generally slightly better but not always.
It is certainly possible to improve the performances of the predictor
with a
larger neighborhood for example but it will not lead to miracles
anyway.
I have also tested the previous pixel compression method suggested in
the FITS
Compression proposal. Well, the algorithm is very simple and the results
are
pretty good. Sometimes, it is possible to even reduce the size of the
obtained
file using a compressing programm (COMPRESS 4.1 seems to be the best).
But still the results show that better performances can simply be
achieved
separating the least significants bits from the most significant ones as
said
above and I do think that some improvements can still be made in that
way.
You will find below the results of some tests. If anybody is interested,
I
can mail some more figures. Any comments or suggestions are very
welcome.

Jean-Pierre Veran
Canada-France-Hawaii Telescope
Email: veran at cfht.hawaii.edu

------------------------------------------------------------------------
---------
This document gives the results of compression tests of 3 FITS object
images,
without header.

First the SHIFT least significant bits were removed from the image ie
Image = Image >> SHIFT has been performed.

Then a linear prediction was computed, using the predictor:
P(x) = 0.351*A + 0.346*C + 0.292*B		B C D
						A X
and each pixel was replaced by its error of prediction X-P(X).

Then this image was compressed with an adaptative Huffman coding (Compact 1.0)

Finally all the bits first removed were compacted and added to the file without
any compression.

The tests have been carried out without and with prediction.

          Size of the original image    2048*2048*2     8388608
RATIO = ----------------------------- = ----------- = -----------
	 Size of the compressed image   KBYTES*1000   KBYTES*1000

<-- indicates a global maximum of the compression ratio.

For comparisons, the performances of the old version of COMPRESS and
COMPACT 1.0
used alone are reminded.

Then the previous pixel compression algorithm has been tried (PREVPIX),
followed by a compression of the file with COMPRESS 4.1 (PREVPIX +
COMPRESS 4.1)
which turned out to be the best compression algorithm for those files.

**********************************************
* Source image: 112069o.fits  Size = 8388608 *
**********************************************
Un. COMPRESS: Ratio = 1.25	KBytes = 6719
COMPACT 1.0 : Ratio = 1.31	KBytes = 6398

PREVPIX:		Ratio = 1.53	Kbytes = 5481
PREVPIX + COMPRESS 4.1:	Ratio = 1.53	Kbytes = 5481

	    ### SHIFT + COMPACT 1.0 ### 

SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	1.31	 6398		  4	1.70	 4936
  1	1.48	 5684		  5	1.77	 4750
  2	1.52	 5506		  6	1.81	 4624
  3	1.62	 5178		  7	1.83	 4594	<--

	### SHIFT + PREDICTOR + COMPACT 1.0 ###
SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	1.40	 5996		  4	1.77	 4737
  1	1.57	 5326		  5	1.83	 4564
  2	1.60	 5253		  6	1.89	 4444	<--
  3	1.70	 4943		  7	1.85	 4533

**********************************************
* Source image: 111907o.fits  Size = 8388608 *
**********************************************
Un. COMPRESS: Ratio = 1.79	KBytes = 4696
COMPACT 1.0 : Ratio = 1.87	KBytes = 4485

PREVPIX:		Ratio = 1.98	Kbytes = 4250
PREVPIX + COMPRESS 4.1:	Ratio = 1.98	Kbytes = 4250

	    ### SHIFT + COMPACT 1.0 ###
SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	1.87	 4485		  4	2.31	 3636
  1	2.01	 4177		  5	2.33	 3597	<--
  2	2.12	 3954		  6	2.30	 3645
  3	2.23	 3766		  7	2.14	 3912

	### SHIFT + PREDICTOR + COMPACT 1.0 ###
SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	1.95	 4304		  4	2.34	 3580
  1	2.09 	 4005		  5	2.38	 3528	<--	 
  2	2.21	 3795		  6	2.31	 3631
  3	2.33	 3607		  7	2.21	 3800

**********************************************
* Source image: 121685o.fits  Size = 8388608 *
**********************************************
Un. COMPRESS: Ratio = 2.86	KBytes = 2937
COMPACT 1.0 : Ratio = 3.02	KBytes = 2777

PREVPIX:		Ratio = 1.97	Kbytes = 4251
PREVPIX + COMPRESS 4.1:	Ratio = 2.97	Kbytes = 2825

	    ### SHIFT + COMPACT 1.0 ###
SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	3.02	 2777		  4	3.19	 2627
  1	3.19	 2630		  5	3.06	 2737
  2	3.32	 2523		  6	2.58	 3250
  3	3.35	 2501	<--	  7	2.23	 3763

	### SHIFT + PREDICTOR + COMPACT 1.0 ###
SHIFT	RATIO	KBYTES		SHIFT	RATIO	KBYTES
-----	-----	------		-----	-----	------
  0	2.97	 2827		  4	3.15	 2665
  1	3.13	 2673		  5	3.07	 2736
  2	3.23	 2589	<--	  6	2.59	 3243
  3	3.20	 2619		  7	2.23	 3753

Jean-Pierre Veran
Canada-France-Hawaii Telescope
Email: veran at cfht.hawaii.edu