Subject: Re: FITS checksum proposal

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Wed, 4 Jan 1995 18:02:49 -0500

There have been no comments on the FITS CHECKSUM proposal posted here on
20 December by Rob Seaman, so before it fades away, it may be worth
posting a reminder about this proposal to groups that maintain archives
of FITS data, or that distribute FITS files to users, and urge them to
consider adding the CHECKSUM and DATASUM keywords to their FITS files.

These 2 keywords allow the recipient of the FITS file to verify the
integrity of the file and ensure (within some reasonable probability)
that the copied file is identical to the original.  It would be easy to
modify existing FITS reading programs to check for these keywords and
verify the checksum values.  If the computed checksums are not
consistent with the keyword values then the user should be warned that
the file has either been modified since the checksums were inserted, or
that the file is corrupted in some way.

It is not difficult to calculate the checksum from first principles,
but for added convenience, the latest version of the FITSIO subroutine
package (V3.7) contains 2 new subroutines for (a) computing and
inserting the CHECKSUM and DATASUM keywords into any FITS HDU, and (b)
verifying that the computed checksum in a FITS HDU is consistent with
the checksum keywords, if they exist.  The 2 new subroutines are:

        call ftpcks( unit, status)     (Put ChecKSum)

which computes and writes the 2 keywords into the opened FITS file, and
 
        call ftvcks( unit, dataok, hduok, status)  (Verify ChecKSum)

which looks for the 2 keywords in the opened FITS file, and if they exist,
verifies the checksum value.  The dataok and hduok output parameters 
indicate whether the checksum for the data unit alone, and the entire
HDU, respectively, are correct or not.

Finally, the latest version of the VERIFITS program that checks any
FITS file for conformance with the FITS Standard will also verify the
checksum if these keywords exist.

The new versions of FITSIO and VERIFITS that support the checksum
keywords will be made available via anonymous ftp on
legacy.gsfc.nasa.gov in the /software/fitsio and
/software/fitsio/verifits directories by the end of this week.

-Bill Pence

---

From: Arnold Rots <arots at xebec.gsfc.nasa.gov>
Date: Tue, 31 Jan 1995 11:52:46 -0500 (EST)

There is one issue that I would like to raise about the checksum
proposal by Seaman and Pence; I do not necessarily disagree with the
proposal as it is, but I believe that the following ought to be
considered.

The question is whether the checksum operations should be done at the
HDU level or at the file level.  In the former case (as proposed) one
would put the checksum keywords in all headers; in the latter only in
the primary header.

I believe that, from the viewpoint of practical use of the checksum,
it does not matter which approach is chosen.  To me, the important
thing is that the checksum over the entire FITS file is supposed to be
zero, in both cases.  That means that one can use any checksum
checking tool that one happens to have lying around and that such a
tool does not need to know anything about FITS files.
Some have made the argument that the current proposal allows one to
find out in which extension the checksum does not check.  I would
consider the value of such information rather minimal: if the
checksum of a FITS file is not zero, I would consider the entire file
suspect and would want a new copy, regardless of where the checksum
error occurred.

So, if it doesn't matter a great deal from the standpoint of
functionality, are there any arguments for or against either approach
on the basis of implementation or operation?

There are some, but I would consider them fairly marginal:
-In the current proposal, when one extension is modified, one does not
have to calculate the checksum over the entire file; the strength of
this argument depends on the frequency of such operations and the size
of the files.
-If there is one checksum per file, one has to go all the way back to
the primary header to update the checksum keyword; in either case one
has to give up the notion of FITS files as sequential files - whether
one then goes back to the last extension header or to the primary
header makes no difference anymore.
-With a single checksum per file, the checksum calculation and update
can be hidden rather elegantly in the file closing utility of any FITS
package; one can do a similar thing with checksums for each HDU.
-With a single checksum per file it is easier to retrofit a checksum
into existing FITS files since one does not have to look into each
individual HDU; though sort of marginal, this is the strongest
argument that I am aware of.

This leads me to the reason why I brought this issue up.  Given the
fact (at least in my mind) that there is no good functional reason to
prefer one approach over the other, is anybody aware of any other,
strong arguments in favor or against either choice?  If not, then I
don't care particularly which way the decision goes, but it would be a
pity if we adopted one and discovered a year from now that the other
would have saved us considerable headaches.

  - Arnold Rots

---

From: "Jonathan McDowell" <jcm at urania.harvard.edu>
Date: Tue, 31 Jan 95 12:20:03 EST

Arnold Rots argues for one checksum per file, not one per HDU.
However,
The advantage of having one checksum per HDU is that an individual HDU can
be moved from one file to another without modification.
(Sometimes one considers the FITS file as an extra directory level
in which the HDUs play the role of files).

  - Jonathan McDowell