Subject: 16 bit unsigned data is taking over the world!

From: newberry at kepler.as.arizona.edu (Mike Newberry)
Date: 25 Nov 92 00:19:44 GMT

This is a lengthy thread, but I thought it sould be of use to readers to know
the history of this discussion. The original posting regarded someone wanting
to read FITS format images without actually having FITS reading software.  I
told them how to do this and noted that the pixels were unsigned 16 bit
integers. This has opened up a related, important discussion which needs to
be addressed: what to do with the proliferation of 16 bit unsigned "FITS" 
data. Unsigned 16 bit pixels are quite common nowadays, masquerading in
images having ostensible FITS headers and blocking structure. This results
from the widespread use of 16 bit A/D converters, especially in the commercial
CCD camera market.

Software authors who don't have the FITS documents look at a FITS header and
naively say "Yes, I see how this is coded". Then they write software to drive
CCD cameras or to write CCD images into "FITS" format. The problem is that
quasi-FITS data is proliferating and we need to do something about it!
I have seen a number of other problems with image headers masquerading as
FITS format which I won't go into here.

--------
In article <24NOV199217101339 at nssdca.gsfc.nasa.gov> bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) writes:
>I'm coming in here in the middle, as I somehow missed the earlier 
>posts in this thread.  I'm repeating the whole previous text, because 
>I'm crossposting to sci.astro.fits, whose readers may not have seen 
>the earlier discussion.
>
> In article <1992Nov23.215025.22192 at organpipe.uug.arizona.edu>, newberry at aquarius.as.arizona.edu (Mike Newberry) writes...
>>In article <1992Nov19.225316.24612 at nrao.edu> cflatter at nrao.edu writes:
>>>In article 6479 at organpipe.uug.arizona.edu, newberry at kepler.as.arizona.edu (Mike Newberry) writes:
>>>>Skip the first 2880 bytes of the file, as this contains the file information
>>>>"header" in FITS format.
>
>For the files under discusssion here, that procedure might work but
>for regular FITS files, the header should not be skipped as it
>is where the information on data type and matrix dimensions is 
>supposed to be found. 
>
>>>>Starting at byte offset 2880 will be found binary,
>>>>uncompressed, unpadded pixel data. The pixels are unsigned 16-bit integers,
>>>                                                   ^^^^^^^^^^^^^^^^^^^^^^^^
>>>>so watch out if unsigned integers might choke your software--these images do
>>>>have values above 32767. To turn this binary raster file into a 2-dimensional
>>>>image, note that there are 375 2-byte pixels in each row. There are 242 rows.
>>>
>>>If this description is accurate the files do not conform to accepted FITS
>>>standards.  Either 16-bit integers are twos complement or SIMPLE = F.
>>>
>>>	Chris Flatters
>>>	cflatter at nrao.edu
>> 
>>Absolutely, Chris.  
>
>And that should be the rule.
>
>>I don't know how many images you've seen lately coming
>>out of CCD systems at major observatories, but images with values above
>>32767 in short integer format and the header flag SIMPLE='T' are typical.
>>They do not conform to FITS standards.  
>
>Period.
>
>>On the other hand, since 16 bit A/D
>>converters are the de facto standard for professional CCD cameras as well as
>>for many used by amateurs these days, you'll have to get used to seeing
>>16 bit unsigned pixels. I've complained about the problem and tried holding
>>out on writing software that would accomodate it. 
>
>Keep complaining.
>
>>The problem is that people
>>who write the CCD controller software refuse to chop values above 32767 and
>>allow their "WFITS" programs to write files or tapes that have SIMPLE='T',
>>thereby assuming that the image processing software will know how to deal
>>with it. If observers want to keep pixels in short integer format throughout
>>data reduction to save disk space and provide much faster processing speed,
>>then they will need to allow for the negative values given by signed integers.
>>With unsigned integer data, they'll either have to live with big negative
>>holes in the centers of bright images or they will have to manually chop
>>their pixels values at 32767 anyway! Since real world images must allow for
>>negative intensity, the founding fathers of FITS, Wells and Griesen, made
>>the right decision in allowing only for signed integers for 2-byte pixels.
>>However, the world hasn't followed the rules. 
>
>The "world", the IAU, has endorsed FITS as the standard data transfer 
>system in astronomy, as defined in the four FITS papers and the 
>Floating Point Agreement.  SIMPLE=T as described in the first FITS 
>paper is the signature of FITS; it implies to the reader who picks up 
>the file that the format is FITS.  If the file following is out of 
>conformance, then whoever is creating the file is not following the 
>IAU standard.  
>
>What the producers of the files could do is to have the original file
>in a local format with SIMPLE=F and then create a file in proper FITS
>format with a proper data type, using BZERO and BSCALE if necessary to
>keep the full range. 
>
>> 
>>To accomodate this ambiguity with 2-byte integers, we have a flag for
>>specifying unsigned integer data in the "read FITS file" function of MIRA.
>>This is better than forcing people to write their data using SIMPLE='F'.
>>Recall that setting SIMPLE='F' doesn't then tell the FITS translator
>>specifically what in the heck is wrong with the image format. For this
>>reason, it is better, I think, to simply allow the user to use SIMPLE='T'
>>and then specify with a flag that their images contain unsigned 16 bit
>>integers. The reader can then chop the values at 32767 to get the pixels
>>into the domain of signed integers. 
>
>No, because then SIMPLE=T is meaning whatever the writer wants it to 
>mean rather than being precisely defined.  Readers should 
>return non-compliant files with SIMPLE=T to the sender.  If a file is 
>sent with SIMPLE=F, then it is up to the sender to define the format 
>for readers.  SIMPLE=T has documentation -- the FITS literature.

Barry, you are absolutely correct; I agree with everything you've said. I
don't want to be misinterpreted as endorsing the violation of FITS format
(and I do understand the details--I have all of the documents), but people
are producing 16 bit unsigned data "right and left" these days. Let's look
at a not-so-hypothetical situation:  How would you respond if you were
deluged with telephone calls by cranky, wild-eyed astronomers on the other
end of the line complaining about large negative "holes" in the middle of all
bright objects?  You can explain to them that their FITS images aren't true
FITS images. You can tell them that it`s not the fault of your FITS reading
software because it conforms to the NOST standard.  You can tell them that
it's the fault of their camera control software or FITS *writing* software.
When enough of them want to send your software back for a refund, you tell
them "Yes, we can write a quick fix that will allow you to work with the
quasi-FITS images you have". I think you get my intent here.

The problem here is that astronomers are stuck with image files which are 
claimed to be genuine FITS format (i.e., SIMPLE='T').  Our intermediate
solution, which I believe to be the only practical one, is to allow people to
specify whether they have 16 bit unsigned data. This way, they can use the
images produced by something they have no control over. Meanwhile, we have
to keep harping on camera manufacturers and software authors about modifying
their software to use only 15 bits of the 16 bit A/D converter. I believe
this to be the only practical solution for the time being.

I propose that FITS might adopt a value of -16 for BITPIX which would signify
unsigned 16 bit pixel values, just as we've standardized -32 as meaning
short real floating point while 32 signifies 32 bit long integer pixel values.

Mike Newberry

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Wed, 25 Nov 1992 13:59:00 GMT

In article <1992Nov25.001944.8002 at organpipe.uug.arizona.edu>, newberry at kepler.as.arizona.edu (Mike Newberry) writes...

	(lengthy thread deleted)

>short real floating point while 32 signifies 32 bit long integer pixel values.

I agree that this is a valuable suggestion.  Currently we use 12 bit A/Ds for
our CCDs, but would like to be able to use 16 bit A/Ds in the future.  Having
an unsigned 16-bit integer type would be very helpful.  (I currently can't
imagine a use for an unsigned 32-bit integer type.)

Let me also suggest that there be an unsigned 16-bit integer type in binary
tables (BINTABLE).  This could be identified as the type "U" in TFORM records.

Programming environments that don't support unsigned integer types, such as
Fortran and IDL, could read 16-bit integer types into 32 bit integer arrays.

Please note that currently, as far as I am aware, using 'BITPIX=-16' as
suggested above would still require 'SIMPLE=F'.

Bill Thompson