Subject: Re: st6 images wanted

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger)
Date: Tue, 24 Nov 1992 22:10:00 GMT

I'm coming in here in the middle, as I somehow missed the earlier 
posts in this thread.  I'm repeating the whole previous text, because 
I'm crossposting to sci.astro.fits, whose readers may not have seen 
the earlier discussion.

 In article <1992Nov23.215025.22192 at organpipe.uug.arizona.edu>, newberry at aquarius.as.arizona.edu (Mike Newberry) writes...
>In article <1992Nov19.225316.24612 at nrao.edu> cflatter at nrao.edu writes:
>>In article 6479 at organpipe.uug.arizona.edu, newberry at kepler.as.arizona.edu (Mike Newberry) writes:
>>>Skip the first 2880 bytes of the file, as this contains the file information
>>>"header" in FITS format.

For the files under discusssion here, that procedure might work but
for regular FITS files, the header should not be skipped as it
is where the information on data type and matrix dimensions is 
supposed to be found. 

>>>Starting at byte offset 2880 will be found binary,
>>>uncompressed, unpadded pixel data. The pixels are unsigned 16-bit integers,
>>                                                   ^^^^^^^^^^^^^^^^^^^^^^^^
>>>so watch out if unsigned integers might choke your software--these images do
>>>have values above 32767. To turn this binary raster file into a 2-dimensional
>>>image, note that there are 375 2-byte pixels in each row. There are 242 rows.
>>
>>If this description is accurate the files do not conform to accepted FITS
>>standards.  Either 16-bit integers are twos complement or SIMPLE = F.
>>
>>	Chris Flatters
>>	cflatter at nrao.edu
> 
>Absolutely, Chris.  

And that should be the rule.

>I don't know how many images you've seen lately coming
>out of CCD systems at major observatories, but images with values above
>32767 in short integer format and the header flag SIMPLE='T' are typical.
>They do not conform to FITS standards.  

Period.

>On the other hand, since 16 bit A/D
>converters are the de facto standard for professional CCD cameras as well as
>for many used by amateurs these days, you'll have to get used to seeing
>16 bit unsigned pixels. I've complained about the problem and tried holding
>out on writing software that would accomodate it. 

Keep complaining.

>The problem is that people
>who write the CCD controller software refuse to chop values above 32767 and
>allow their "WFITS" programs to write files or tapes that have SIMPLE='T',
>thereby assuming that the image processing software will know how to deal
>with it. If observers want to keep pixels in short integer format throughout
>data reduction to save disk space and provide much faster processing speed,
>then they will need to allow for the negative values given by signed integers.
>With unsigned integer data, they'll either have to live with big negative
>holes in the centers of bright images or they will have to manually chop
>their pixels values at 32767 anyway! Since real world images must allow for
>negative intensity, the founding fathers of FITS, Wells and Griesen, made
>the right decision in allowing only for signed integers for 2-byte pixels.
>However, the world hasn't followed the rules. 

The "world", the IAU, has endorsed FITS as the standard data transfer 
system in astronomy, as defined in the four FITS papers and the 
Floating Point Agreement.  SIMPLE=T as described in the first FITS 
paper is the signature of FITS; it implies to the reader who picks up 
the file that the format is FITS.  If the file following is out of 
conformance, then whoever is creating the file is not following the 
IAU standard.  

What the producers of the files could do is to have the original file
in a local format with SIMPLE=F and then create a file in proper FITS
format with a proper data type, using BZERO and BSCALE if necessary to
keep the full range. 

> 
>To accomodate this ambiguity with 2-byte integers, we have a flag for
>specifying unsigned integer data in the "read FITS file" function of MIRA.
>This is better than forcing people to write their data using SIMPLE='F'.
>Recall that setting SIMPLE='F' doesn't then tell the FITS translator
>specifically what in the heck is wrong with the image format. For this
>reason, it is better, I think, to simply allow the user to use SIMPLE='T'
>and then specify with a flag that their images contain unsigned 16 bit
>integers. The reader can then chop the values at 32767 to get the pixels
>into the domain of signed integers. 

No, because then SIMPLE=T is meaning whatever the writer wants it to 
mean rather than being precisely defined.  Readers should 
return non-compliant files with SIMPLE=T to the sender.  If a file is 
sent with SIMPLE=F, then it is up to the sender to define the format 
for readers.  SIMPLE=T has documentation -- the FITS literature.

>This is an issue I had planned on
>bringing up at the FITS committee meeting at the AAS in January.

I don't think there is a formal WGAS meeting until the summer AAS.  
However, this posting to sci.astro.fits will serve the same purpose.  
Much of the discussion is by electronic mail.

Until such time as some provision may be made for unsigned 16-bit 
integers, they should not appear in the primary data array of a file 
with SIMPLE = T.

>It would make life easy again if people would use only 15 bits of their
>16 bit A/D converts, but that doesn't seem to be in the cards yet.
> 
>Mike Newberry
>Axiom Research, Inc.
>..

Standards are developed so that readers know what 
to expect.  If people create data sets in the form of the standard 
that are not in compliance, then they defeat the purpose of creating 
the standard.

				Barry Schlesinger
				NOST FITS Support Office

---

From: newberry at kepler.as.arizona.edu (Mike Newberry)
Date: 24 Nov 92 23:40:11 GMT

>for readers.  SIMPLE=T has documentation -- the FITS literature.

Barry, you are absolutely correct; I have no argument with anything you've
said.  My point is not that I'm endorsing the violation of FITS format (and
I do understand the details--I have all of the documents), but people are 
producing 16 bit unsigned data "right and left" these days. How would you
respond if you were deluged with telephone calls by cranky, wild-eyed guys on
the other end of the line complaining about "holes" in the middle of all
bright objects?  You can explain to them that their FITS images aren't true
FITS images. You can tell them that it`s not the fault of your FITS reading
software because it conforms to the NOST standard.  You can tell them that
it's the fault of their camera control software or FITS *writing* software.
When enough of them want to send *your* software back, you tell them "yes,
we can write in a quick fix that will allow you to work with the not-to-true 
FITS images you have".

That was my point. I am not endorsing violation of the FITS standard. What
I am doing is allowing people to work with data they have or produce using
software someone else wrote for them or sold to them. Meanwhile, we have to
keep harping on camera manufacturers and software writers about modifying
their software to use only 15 bits of the 16 bit A/D converter. I believe
this to be the only practical solution for the time being.

Mike Newberry

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Wed, 25 Nov 1992 01:25:00 GMT

In article <1992Nov24.234011.7663 at organpipe.uug.arizona.edu>, newberry at kepler.as.arizona.edu (Mike Newberry) writes...

	(stuff deleted)

>this to be the only practical solution for the time being.

The idea of changing the science to accomodate FITS files just floors me.  One
either needs to change FITS files to meet the needs of the science, or (gasp!)
use something other than FITS, like CDF, HDF, etc.

However, as Barry pointed out, there are ways to write the data out to FITS
files, and still be legal.  You could,

	1.  Write the data out as SIMPLE=F.  You have to do this anyways unless
	    you fix the problem in some way consistent with FITS.

	2.  Subtract 32767 from the data, and set BZERO = -32767.

	3.  Divide the data by 2 to scale it into 15 bits, if that's all you
	    want.

	4.  Write the data out in 32 bit long integers.

	5.  Define a FITS extension that does allow for unsigned 16 bit
	    integers.

However, if you don't solve it in a "legal" way, then you *have* to set SIMPLE
to F, because it *isn't* a FITS file.

Bill Thompson

---

From: dbriggs at zia.aoc.nrao.edu (Daniel Briggs)
Date: Wed, 25 Nov 92 07:47:29 GMT

In article <1992Nov24.234011.7663 at organpipe.uug.arizona.edu>
  newberry at kepler.as.arizona.edu (Mike Newberry) writes:
>Meanwhile, we have to

Hmmmm.  Wouldn't setting BZERO to 32768 and BSCALE to 1.0 map the range of
the signed integer into the range of an unsigned integer?  Granted, you'll
lose some of the distinction between array values and true physical values.
But for any desired mapping of 16 bit unsigned array values into physical
values, there will be a proper BZERO and BSCALE that operates on 16 bit
signed array values instead.

I grant you that having the pixel values be true unsigned integers is
aesthetically cleaner, but at least this way we've got valid FITS files.

-- 
| Daniel Briggs  (dbriggs at nrao.edu)                      |  USPA B-14993
| New Mexico Tech / National Radio Astronomy Observatory |      DoD #387
| P.O. Box O / Socorro, NM 87801   (505) 835-7391        | 
Support the League for Programming Freedom   (info from lpf at uunet.uu.net)

---

From: zrepachol at cc.curtin.edu.au
Date: Fri, 27 Nov 1992 13:23:57 GMT

> In article <24NOV199217101339 at nssdca.gsfc.nasa.gov> bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) writes:
...
>>>>In article 6479 at organpipe.uug.arizona.edu, newberry at kepler.as.arizona.edu (Mike Newberry) writes:
...
> this to be the only practical solution for the time being.

So do you keep using 15 of your 16, next year 18, then 20 bit converters? If
it does not conform to the standard, its NOT FITS. Beat the daylights out of
the supplier, cancel the cheque or what ever it takes to pound through the
skulls of the programers to get it right.

There is a right way and a wrong way to fix something thats broken. Fixing it
when its not broken is one of the more stupid...

~Paul