From fitsbits-request Tue Mar 1 13:49:23 1994 X-VM-Message-Order: (1 3 2 18 19 4 20 6 7 5 21 22 8 9 23 26 25 10 29 11 24 27 31 28 32 33 30 12 14 34 13 35 15 16 36 37 39 38 40 17 41 42 44 46 45 43) X-VM-Summary-Format: "%n %*%a %-17.17F %-3.3m %2d %4l/%-5c %I\"%s\"\n" X-VM-Labels: nil X-VM-VHeader: ("Resent-" "From:" "Sender:" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:") nil X-VM-Bookmark: 45 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["122" "Tue" " 1" "March" "1994" "17:19:25" "GMT" "frankw at sron.ruu.nl" "frankw at sron.ruu.nl" "<1994Mar1.171925.16767 at cc.ruu.nl>" "9" "Q: FTOOLS for HP" "^From:" nil nil "3" "1994030117:19:25" "Q: FTOOLS for HP" (number " " mark " frankw at sron.ruu.n Mar 1 9/122 " thread-indent "\"Q: FTOOLS for HP\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA27677; Tue, 1 Mar 94 13:49:23 EST Return-Path: Message-Id: <1994Mar1.171925.16767 at cc.ruu.nl> Organization: ACCU Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!EU.net!sun4nl!news.nic.surfnet.nl!ruu.nl!usenet Reply-To: frankw at sron.ruu.nl From: frankw at sron.ruu.nl Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Q: FTOOLS for HP Date: Tue, 1 Mar 1994 17:19:25 GMT Hi, I'm looking for a version of FTOOLS for HP. Does it exist? If so, where can I get it? Thanks, Frankw at sron.ruu.nl From fitsbits-request Tue Mar 1 15:50:01 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3402" "" " 1" "March" "1994" "15:25" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<1MAR199415254559 at nssdca.gsfc.nasa.gov>" "69" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030119:25:00" "Unsigned Integers Proposal" (number " " mark " BARRY M. SCHLESIN Mar 1 69/3402 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<2kt28e$kjs at fnnews.fnal.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA27825; Tue, 1 Mar 94 15:50:01 EST Return-Path: Message-Id: <1MAR199415254559 at nssdca.gsfc.nasa.gov> Organization: NASA - Goddard Space Flight Center Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!swrinde!ihnp4.ucsd.edu!ames!news.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger References: <9402232222.AA08514 at xebec.gsfc.nasa.gov> <2kt28e$kjs at fnnews.fnal.gov> From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 1 Mar 1994 15:25 EDT In article <2kt28e$kjs at fnnews.fnal.gov>, nicinski at fnal.fnal.gov writes... >In article <9402232222.AA08514 at xebec.gsfc.nasa.gov>, arots at xebec.gsfc.nasa.gov (Arnold Rots) writes: >|> Proposal to Add Unsigned Integer Data Types to the FITS Standard >|> ================================================================ >|> >!> [deleted material] >|> >|> As far as integer data types are concerned, FITS only recognizes >|> unsigned 8-bit and signed (2's complement) 16- and 32-bit integers. >|> Although there may have been good reasons for this selection in the >|> past, it is arbitrary, inconsistent, and too restrictive for present >|> day needs. I propose that 16- and 32-bit unsigned integers be >|> recognized, as well. >|> >!> [deleted material] >!> ... possible solutions: >|> >|> 1. Append the letter "U" to the data type. A 16-bit unsigned integer >|> would be represented by "IU", a 32-bit one by "JU". This has the >|> advantage that it does not violate the current rules. However, it is >|> somewhat distasteful since it looks like another ad hoc kludge. >|> >|> 2. Represent 16-bit unsigned integers by the letter "U", 32-bit ones >|> by the letter "V". This choice of letters is reasonably >|> self-explanatory since it follows the I-J convention. > I think 2. is better. But the community will have to discuss this issue. >We have taken a different route with unsigned integers under Binary Tables. >A new logical keyword was created, TSIGNn. If not present, the current >"signedness" of the data types is used. If set to `T', it indicates that >the field value is signed. If set to `F', it indicates that the field value >is unsigned. Our FITS Table reader will only allow TSIGNn to appear for >integer data types. > But what about someone else's FITS reader, designed according to the definition of BINTABLE in the Cotton, Tody, and Pence proposal? It will ignore the TSIGNn keyword and read the value as a signed integer. The whole point of FITS is to create a transport standard that is universally recognized. Local conventions that will not be understood by readers elsewhere defeat the purpose of a standard. Until there is some general agreement their use should be limited to those who use them. And they should be designed in a way that will not break standard FITS readers. For any file with the above convention to be FITS conformant, the binary tables would have to have the value of XTENSION set to other than BINTABLE, and the new name should be registered with the IAU FITS Working Group. BINTABLE is already reserved for the Cotton et al. proposal. Alternatively, the entire file could start with SIMPLE = F, to indicate that the included binary table does not conform to the rules for BINTABLE. >This type of convention would also work for the PDU. It's completely backwards >compatible (as is Arnold's). We chose a new keyword, rather than a new data type, >to address PDUs (in case we ever need to do so in the future). We needed some- >thing other than TZEROn and TSCALn, because our reader is very generic -- thus >we cannot afford to perform any data conversions where loss of information is >possible (we let the user decide how TZEROn and TSCALn are applied). > This proposal is also appropriate for community discussion but should not be unilaterally implemented. Barry Schlesinger NSSDC/NOST FITS Support Office From fitsbits-request Tue Mar 1 16:20:12 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["580" "Tue" " 1" "March" "1994" "16:20:08" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403012120.AA23008 at tetra.Gsfc.NASA.Gov>" "15" "Re: Q: FTOOLS for HP" "^From:" nil nil "3" "1994030121:20:08" "Q: FTOOLS for HP" (number " " mark " William Pence Mar 1 15/580 " thread-indent "\"Re: Q: FTOOLS for HP\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA27860; Tue, 1 Mar 94 16:20:12 EST Return-Path: Message-Id: <9403012120.AA23008 at tetra.Gsfc.NASA.Gov> From: pence at tetra.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Q: FTOOLS for HP Date: Tue, 1 Mar 94 16:20:08 EST > I'm looking for a version of FTOOLS for HP. Does it exist? If so, > where can I get it? > > Thanks, > > Frankw at sron.ruu.nl As far as we know, the standalone version of FTOOLS has not been ported to an HP. In principle, the IRAF version of FTOOLS should run on an HP (assuming IRAF has been ported to your machine), but this has not been tested. The ftools distribution .tar files are available from legacy.gsfc.nasa.gov in the /software/ftools/release directory If you do get it to run, we would be interested in hearing about it. -Bill Pence (pence at tetra.gsfc.nasa.gov) From fitsbits-request Wed Mar 2 12:52:35 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["847" "Wed" " 2" "March" "1994" "16:58:49" "GMT" "Bill Oegerle" "oegerle at stsci.edu" "<1994Mar2.165849.19013 at stsci.edu>" "18" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030216:58:49" "Unsigned Integers Proposal" (number " " mark " Bill Oegerle Mar 2 18/847 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9402281928.AA25016 at fits.cv.nrao.edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA29656; Wed, 2 Mar 94 12:52:35 EST Return-Path: Message-Id: <1994Mar2.165849.19013 at stsci.edu> Organization: Space Telescope Science Institute, Baltimore MD Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news-feed-2.peachnet.edu!gatech!swrinde!elroy.jpl.nasa.gov!ncar!noao!stsci!dorado.stsci.edu!oegerle References: <9402281928.AA25016 at fits.cv.nrao.edu> Reply-To: oegerle at stsci.edu From: oegerle at stsci.edu (Bill Oegerle,326,4744) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Wed, 2 Mar 1994 16:58:49 GMT In article AA25016 at fits.cv.nrao.edu, forveill at gag.observ-gr.fr (Thierry Forveille) writes: > I don't find the case for unsigned integers in FITS totally convincing. > The argument that one should keep the data in as raw a format as possible > is one that I usually buy, but I would make an exception here since the > transformation (just an offset, and no scaling) can be inverted exactly, > to the very last bit included.... (rest deleted) I agree. for instance, in IRAF, using the "wfits" task, just use the parameters: > wfits file=xxx bscale=1 bzero=32768 bitpix=16 scale=yes (note that even though you tell IRAF to scale the data, this does not really happen... you say scale=yes just to get IRAF to use the specified values of bscale and bzero. since bscale=1 there is no scaling, just an offset of bzero=32768) --Bill Oegerle STScI From fitsbits-request Wed Mar 2 17:17:26 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["995" "" " 2" "March" "1994" "21:08:30" "GMT" "Ron Watkins" "ron at argus.lpl.arizona.edu" "<2l2v8e$qca at organpipe.uug.arizona.edu>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030221:08:30" "Unsigned Integers Proposal" (number " " mark " Ron Watkins Mar 2 25/995 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<1994Mar2.165849.19013 at stsci.edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA01621; Wed, 2 Mar 94 17:17:26 EST Return-Path: Message-Id: <2l2v8e$qca at organpipe.uug.arizona.edu> Organization: Lunar and Planetary Lab, U of AZ Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!europa.eng.gtefsd.com!library.ucla.edu!csulb.edu!nic-nac.CSU.net!news.Cerritos.edu!news.Arizona.EDU!math.arizona.edu!CS.Arizona.EDU!organpipe.uug.arizona.edu!argus.lpl.Arizona.EDU!ron References: <9402281928.AA25016 at fits.cv.nrao.edu> <1994Mar2.165849.19013 at stsci.edu> From: ron at argus.lpl.Arizona.EDU (Ron Watkins) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 2 Mar 1994 21:08:30 GMT I am unclear on this whole concept of scaling signed values to unsigned values. Say I have a value in the range 0..65536, Now if these bits are interpreted as signed values then they are arranged as follows for two's complement: Unsigned Signed Signed+32768 0 0 32768 . . . . . . 32767 32767 65535 32768 -32768 0 . . . . . . 65536 -1 32767 Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed values. This will goof things up, won't it? What's going on? How can I take bit patterns that I already have as unsigned and put them into a FITS file and make the come out again correctly AND at the same time, make it so the FITS file makes sence of those values? -- Ron Watkins [ron at argus.lpl.arizona.edu] / /~~~~) / 931 Gould-Simpson / /____/ / University of Arizona / / / Tucson AZ. 85721 -- (602) 621-8606 (____ unar & / lanetary (____ ab. From fitsbits-request Thu Mar 3 09:11:26 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1162" "Thu" " 3" "March" "1994" "09:11:22" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403031411.AA02798 at xebec.gsfc.nasa.gov>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030314:11:22" "Unsigned Integers Proposal" (number " " mark " Arnold Rots Mar 3 25/1162 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA03689; Thu, 3 Mar 94 09:11:26 EST Return-Path: Message-Id: <9403031411.AA02798 at xebec.gsfc.nasa.gov> From: arots at xebec.gsfc.nasa.gov (Arnold Rots) Sender: fitsbits-request at fits.CV.NRAO.EDU To: ron at argus.lpl.arizona.edu Cc: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Thu, 3 Mar 94 09:11:22 EST ron at argus.lpl.Arizona.EDU (Ron Watkins) writes: > Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed > values. This will goof things up, won't it? What's going on? How can I take > bit patterns that I already have as unsigned and put them into a FITS file > and make the come out again correctly AND at the same time, make it so the > FITS file makes sence of those values? You are right, this does not work. In the current situation, you first have to subtract 32768 from all your numbers, before you deposit them into the FITS file. That is why I recommend FITS recognize unsigned integer data types. Even so, there is one additional complication: Fortran does not know unsigned integers. Hence, if your file contains unsigned 16-bit integers, and you are working in Fortran, you need to put them into 32-bit integers; if you are working with unsigned 32-bit integers, I guess the only thing you can do officially, is put them into doubles. If you are working in C, you don't have any of these problems, of course, but beware: if you are using a Fortran-based FITS reader, you might still be in for surprises. - Arnold Rots From fitsbits-request Thu Mar 3 09:46:43 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1489" "" " 3" "March" "1994" "14:22:38" "GMT" "David Robinson" "drtr at mail.ast.cam.ac.uk" "<2l4rre$87m at lyra.csx.cam.ac.uk>" "29" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030314:22:38" "Unsigned Integers Proposal" (number " " mark " David Robinson Mar 3 29/1489 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<2l2v8e$qca at organpipe.uug.arizona.edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA03716; Thu, 3 Mar 94 09:46:43 EST Return-Path: Message-Id: <2l4rre$87m at lyra.csx.cam.ac.uk> Organization: Institute of Astronomy, Cambridge Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!gatech!howland.reston.ans.net!pipex!lyra.csx.cam.ac.uk!pavo.csi.cam.ac.uk!drtr References: <9402281928.AA25016 at fits.cv.nrao.edu> <1994Mar2.165849.19013 at stsci.edu> <2l2v8e$qca at organpipe.uug.arizona.edu> From: drtr at mail.ast.cam.ac.uk (David Robinson) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 3 Mar 1994 14:22:38 GMT In article <2l2v8e$qca at organpipe.uug.arizona.edu> ron at argus.lpl.Arizona.EDU (Ron Watkins) writes: >I am unclear on this whole concept of scaling signed values to unsigned >values. >... >Now, if I say bscale=1 and bzero=32768, then it will add 32768 to the signed >values. This will goof things up, won't it? What's going on? How can I take >bit patterns that I already have as unsigned and put them into a FITS file >and make the come out again correctly AND at the same time, make it so the >FITS file makes sence of those values? It's very simple. You have data with values x = 0 ... 65535. So x - 32768 is in the range -32768 ... 32767. Thus the (x-32768) values can be stored in a BITPIX=16 signed 16-bit integer array. The data has not been scaled, and has had a constant subtracted from it, therefore bscale = 1 and bzero = 32768. Any fits reader will apply the bscale/bzero values to the raw data before presenting it to the user, thus restoring the original 0 ... 65535 values. Note that a portable program will *never* be able to simply 'take the bit patterns that it has as unsigned'; on many computers, the bytes of a native unsigned int will be in the wrong order compared to the required FITS format. The existing FITS standard, therefore, can support efficient storage of unsigned 16-bit integers. Adding a new format would give no extra functionality, generate incompatibility problems, and require more complex FITS readers. David Robinson. (drtr at mail.ast.cam.ac.uk) From fitsbits-request Thu Mar 3 12:00:27 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1561" "Thu" " 3" "March" "1994" "18:00:11" "+0100" "pgrosbol at eso.org" "pgrosbol at eso.org" "<9403031700.AA00687 at ns2.hq.eso.org>" "37" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030317:00:11" "Unsigned Integers Proposal" (number " " mark " pgrosbol at eso.org Mar 3 37/1561 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA04040; Thu, 3 Mar 94 12:00:27 EST Return-Path: Message-Id: <9403031700.AA00687 at ns2.hq.eso.org> From: pgrosbol at eso.org Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Thu, 03 Mar 94 18:00:11 +0100 On the Proposal to Add Unsigned Integer Data Types to the FITS Standard by Arnold Rots The discussion on unsigned 16-bit integers as a FITS data type started already in the mid 80's when CCD acquisition systems started to produce 15-16 bit data. It has been clearly demonstrated in the previous discussion that it is possible to offset unsigned intergers and save them as signed values. No information is lost and the cost is only two interger operations which at the time of multi MIPS CPUs is insignificant. If new data types were introduced all FITS reader in the world would in principle have to be upgraded to conform to the new standard. This is a dramatic action which can only be justified if very significant new functionality is introduced. I believe it was the case when IEEE floating point formats were added. It should be noted that the rules for 'Generalized extensions and blocking factors for FITS' A&A Suppl. 73,p359 state: 'Only the binary and character coding conventions specified in the FITS standards should be used in FITS extensions.' Thus, one cannot add unsigned intergers to the BINTABLE extension without changing the full set of FITS data types. Values of BITPIX=17 or 33 would violate the formular for the size of the data matrix: nbit = abs(bitpix)*gcount*(pcount+naxis1*...*naxisn) and are not acceptable. The proposal would only bring a marginal advantadge (i.e. saving two integer operations per pixel) but require very significant code modifications. Thus, I cannot support the proposal. Preben Grosbol ESO From fitsbits-request Fri Mar 4 15:22:48 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2248" "Fri" " 4" "March" "1994" "15:22:42" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403042022.AA13581 at tetra.Gsfc.NASA.Gov>" "38" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030420:22:42" "Unsigned Integers Proposal" (number " " mark " William Pence Mar 4 38/2248 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02503; Fri, 4 Mar 94 15:22:48 EST Return-Path: Message-Id: <9403042022.AA13581 at tetra.Gsfc.NASA.Gov> From: pence at tetra.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Fri, 4 Mar 94 15:22:42 EST Regarding the unsigned integer proposal by A. Rots, my opinion it that the lack of support for unsigned integers is a serious deficiency of FITS that needs to be addressed. This issue is not going to go away no matter how much we preach that unsigned integer values must be transformed into scaled integers with the appropriate TSCAL keyword value. Many people, as is already evident from the discussion about this proposal, will simply not follow this FITS requirement and will go ahead and write the unsigned value to the FITS file. Some might add a keyword like SIGNED = F to try to indicate that they broke the rules, but many won't even do this. Some people will do this out of ignorance, not realizing that it is possible to coerce unsigned integers into a signed integer datatype. Others may just refuse to do it because they think this sort of scaling is all rather mindless and stupid, and they see no point in it. Still others may have valid concerns about the impact on efficiency, or on the added complexity of the software, if they have to scale the unsigned integer data. So for whatever reason, people are breaking the FITS rules on this, and I think the problem will only get worse in the future if more and more groups start using FITS as their primary data format. Other competing data formats, like HDF, do support unsigned integers, so unless FITS can adapt to the need of users, users will not use FITS. My preference would be to setup a small technical committee that would consider all the issues and try to come up with a proposal for supporting unsigned integers that minimizes the impact on existing FITS readers. I've already looked at what it would take to add full support for unsigned I*2, I*4, and signed I*1 values in both primary arrays and in binary tables within my FITSIO subroutine package and estimate this could be done in a week or 2. Any FITS software that uses FITSIO could then transparently read or write unsigned integers without any change to the application program, simply by linking to the new FITSIO library. I think that the disruption to existing FITS readers is a managable problem, and the benefits of adding this capability to FITS outweighs the problems. -Bill Pence HEASARC From fitsbits-request Fri Mar 4 15:59:26 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2151" "Fri" " 4" "March" "1994" "21:57:09" "+0100" "Thierry Forveille" "forveill at gag.observ-gr.fr" "<9403042059.AA02558 at fits.cv.nrao.edu>" "40" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994030420:57:09" "Unsigned Integers Proposal" (number " " mark " Thierry Forveille Mar 4 40/2151 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403031411.AA02798 at xebec.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA02564; Fri, 4 Mar 94 15:59:26 EST Return-Path: Message-Id: <9403042059.AA02558 at fits.cv.nrao.edu> Posted-Date: Fri, 4 Mar 1994 21:57:09 +0100 In-Reply-To: <9403031411.AA02798 at xebec.gsfc.nasa.gov> References: <9403031411.AA02798 at xebec.gsfc.nasa.gov> From: forveill at gag.observ-gr.fr (Thierry Forveille) Sender: fitsbits-request at fits.CV.NRAO.EDU To: arots at xebec.gsfc.nasa.gov (Arnold Rots) Cc: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Fri, 4 Mar 1994 21:57:09 +0100 Arnold Rots writes: > In the current situation, you > first have to subtract 32768 from all your numbers, before you deposit > them into the FITS file. > > That is why I recommend FITS recognize unsigned integer data types. > I don't really see what is wrong with subtracting 32768 before writing. The cost is small compared with the CPU overhead of the physical I/O, and depending on the computer you are using, you may have to swap the bytes (also a more costly operation). On RISC architectures, I think instructions for unsigneds are actually emulated (16 bits signed instructions too), so CPU efficiency isn't obtained that way. The additional complexity of the code doesn't seem a very serious issue either. Is that rather a philosophical issue of whether one is allowed to alter the bits at all? > Even so, there is one additional complication: Fortran does not know > unsigned integers. Hence, if your file contains unsigned 16-bit > integers, and you are working in Fortran, you need to put them into > 32-bit integers; if you are working with unsigned 32-bit integers, I > guess the only thing you can do officially, is put them into doubles. > If you are working in C, you don't have any of these problems, of > course, but beware: if you are using a Fortran-based FITS reader, you > might still be in for surprises. > Most processing packages would in practice put the data in a float, not in a 32-bit integer. Integers are most frequently used to store what is really a fixed point real number, at least when there are enough of them that the size matters. Even photon counts (which are intrinsically integers) have to be converted to floats for any serious processing or calibration (rather than just data display). As soon as you need to convert counts to photons/s/m**2, calibrate the relative response of a spectrograph, or remove an instrumental background, the data become reals, not integers. This is another reason why one need not care that the data are stored as signed with an offset rather than unsigned: what will really be used is a float or a double. Thierry Forveille Observatoire de Grenoble From fitsbits-request Tue Mar 8 13:19:51 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["662" "Tue" " 8" "March" "1994" "05:35:43" "GMT" "BRIAN STAPLES" "brian.staples at digcir.cts.com" "<94030804001821 at digcir.cts.com>" "19" "Need FITS conversion" "^From:" nil nil "3" "1994030805:35:43" "Need FITS conversion" (number " " mark " BRIAN STAPLES Mar 8 19/662 " thread-indent "\"Need FITS conversion\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA09787; Tue, 8 Mar 94 13:19:51 EST Return-Path: Message-Id: <94030804001821 at digcir.cts.com> Organization: Digital Circus BBS (619) 223-5348 Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!gatech!udel!news.sprintlink.net!crash!digcir!brian.staples From: brian.staples at digcir.cts.com (BRIAN STAPLES) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Need FITS conversion Date: Tue, 8 Mar 1994 05:35:43 GMT Greetings one and all. I am looking for a program that will convert GIF, TIF, and/or BMP files (or other common graphic files) into FITS format. I needed it to run either in MS WIndows or DOS. A slick program is not what I need, just a basic conversion program that will allow me to do convert common files to FITS so I can do some basic image processing (I'm an amateur astronomer looking to practice). If anyone out there has any suggestions, recommendations, and sources, I would truly appreciate it. Many thanks in advance. Brian Staples __________________________________________ bstaples at digcir.cts.com __________________________________________ From fitsbits-request Thu Mar 10 09:17:39 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["274" "Thu" "10" "March" "1994" "10:15:15" "GMT" "John Boyer" "boyer at rd.eng.bbc.co.uk" "" "10" "please help with imdisp79" "^From:" nil nil "3" "1994031010:15:15" "please help with imdisp79" (number " " mark " John Boyer Mar 10 10/274 " thread-indent "\"please help with imdisp79\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15584; Thu, 10 Mar 94 09:17:39 EST Return-Path: Message-Id: Organization: British Broadcasting Corporation, UK Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!math.ohio-state.edu!howland.reston.ans.net!pipex!bbc!ant!boyer From: boyer at rd.eng.bbc.co.uk (John Boyer) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: please help with imdisp79 Date: Thu, 10 Mar 1994 10:15:15 GMT I have imdisp79 and a fits file which i got from stdatu. I have tried to display this file with imdisp and I get junk. Imdisp says 1024*1024 16 sample bits and the label says 32 sample bits! I am confused can some one help me please.? john b john.boyer at rd.eng.bbc.co.uk From fitsbits-request Thu Mar 10 16:52:29 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["504" "Thu" "10" "March" "1994" "16:52:24" "EST" "Maureen Conroy" "mo at cfa234.harvard.edu" "<9403102152.AA11370 at cfa234.harvard.edu>" "13" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031021:52:24" "Unsigned Integers Proposal" (number " " mark " Maureen Conroy Mar 10 13/504 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA17854; Thu, 10 Mar 94 16:52:29 EST Return-Path: Message-Id: <9403102152.AA11370 at cfa234.harvard.edu> From: mo at cfa234.harvard.edu (Maureen Conroy) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Thu, 10 Mar 94 16:52:24 EST If there is a well-defined and supported way to write UNSIGNED integers to FITS files, I would think that the most important issue is to be sure that the technique is clear and well-documented. The NOST FITS office has a FITS users guide to provide guidelines in using FITS files. Judging from the varied responses it would seem to be an excellent example to cite specifically in the guide, since it seems to be a frequently needed feature. Barry - what do you think? Maureen Conroy SAO From fitsbits-request Mon Mar 14 10:01:01 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1823" "" "14" "March" "1994" "09:40" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<14MAR199409402991 at nssdca.gsfc.nasa.gov>" "35" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031413:40:00" "Unsigned Integers Proposal" (number " " mark " BARRY M. SCHLESIN Mar 14 35/1823 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403102152.AA11370 at cfa234.harvard.edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA09104; Mon, 14 Mar 94 10:01:01 EST Return-Path: Message-Id: <14MAR199409402991 at nssdca.gsfc.nasa.gov> Organization: NASA - Goddard Space Flight Center Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!saimiri.primate.wisc.edu!nntp.msstate.edu!emory!swrinde!ihnp4.ucsd.edu!ames!news.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger References: <9403102152.AA11370 at cfa234.harvard.edu> From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 14 Mar 1994 09:40 EDT In article <9403102152.AA11370 at cfa234.harvard.edu>, mo at cfa234.harvard.edu (Maureen Conroy) writes... > > If there is a well-defined and supported way to >write UNSIGNED integers to FITS files, I would think that the >most important issue is to be sure that the technique is clear >and well-documented. The NOST FITS office has a FITS users guide >to provide guidelines in using FITS files. Judging >from the varied responses it would seem to be an excellent example >to cite specifically in the guide, since it seems to be a >frequently needed feature. > Barry - what do you think? > Anything done in the near term has to be done in a way that will not be misinterpreted by a reasonable FITS reader. Use of BZERO and BSCALE satisfies this criterion. The technique is discussed in general terms in the first FITS paper. I will add specific discussion in the next *full* revision of the User's Guide (version 4.0) about how unsigned integers can be handled through the use of scaling. The TSIGNn proposal does not satisfy this criterion, as FITS files using it would be misinterpreted by a FITS reader constructed on the basis of the current rules of FITS. Similarly, the proposed new data types in BINTABLE would not be understood by a current FITS reader. Such proposed new structures for unsigned integers must be discussed by the FITS community. If a consensus should emerge that a new rule is necessary, then the normal procedures of consideration by the regional and international FITS committees and demonstration of its use in data tranport must be followed before it can be approved as part of FITS. Present discussion suggests that there is not currently a consensus as to whether specific provisions for unsigned integers are desirable. Barry Schlesinger NSSDC/NOST FITS Support Office From fitsbits-request Mon Mar 14 11:31:26 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["7402" "" "14" "March" "1994" "15:49:54" "GMT" "kent at steve.fnal.gov" "kent at steve.fnal.gov" "<2m2132$k78 at fnnews.fnal.gov>" "131" "Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031415:49:54" "Yet another unsigned integers proposal" (number " " mark " kent at steve.fnal.g Mar 14 131/7402 " thread-indent "\"Yet another unsigned integers proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA09255; Mon, 14 Mar 94 11:31:26 EST Return-Path: Message-Id: <2m2132$k78 at fnnews.fnal.gov> Organization: FERMILAB, Batavia, IL Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!rsg1.er.usgs.gov!stc06r.CTD.ORNL.GOV!fnnews.fnal.gov!steve.fnal.gov!KENT Reply-To: kent at steve.fnal.gov From: kent at steve.fnal.gov Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Yet another unsigned integers proposal Date: 14 Mar 1994 15:49:54 GMT The Sloan Digital Sky Survey (SDSS) project will generate of order 10 terabytes of raw data and 200 gigabytes of reduced data over a period of 5 years. As computing coordinator for the survey, one of my responsiblities is to specify the formats for the various types of data that will be collected. As you can imagine, it is important to do this job properly at the beginning. These formats will be used not only for internal purposes but also for the distribution of data to the external community. Our project was strongly advised to use FITS wherever possible, and we have been assidiously pursuing that track. We will use use a combination of basic FITS images, Ascii tables, and binary tables to store and exchange a variety of raw and processed data. Our raw data will all be - yes - 16 bit unsigned integers. We are therefore faced with the same dilemma as Arnold Rots described in a recent posting, and we have come to the same conclusion - it is very desirable that the FITS standard provide explicit support for unsigned data types, and the place to do so is in binary tables. In spite of the recent thread arguing that FITS currently handles unsigned types properly (flip the high order bit and set TSCAL=1, TZERO = 32768), we have, in fact, found that this is not at all the case. At present we are faced with writing all of our FITS files with SIMPLE = F in order that we can adopt some nonstandard conventions to deal with the inadequacies. This is highly unsatisfactory. The first problem we face is one of hard economics - we will be collecting data at the rate of 7 Mbytes per second, and the current data acquisition system will not handle the extra compute steps for even a simple bit flip. We are not about to spend the several thousand dollars needed to add the extra compute muscle for this meaningless task. At present we will write the raw data as FITS images with the unsupported BITPIX=-16 convention and SIMPLE = F. The second (and more serious) problem arose when we tried to write a general purpose reader and writer for FITS images and tables that translates the data into their most "natural" format - what is the exact meaning to be applied to data values in the primary array or in tables? The FITS standard, unfortunately, blurs the distinction between the physical value that one wants to represent and the FITS table representation of that value, particularly when scaling is involved. The FITS standard seems to imply that a physical value is a pure number plus physical dimensions, with no connotation of int, float, or signed attached nor any specific binary representation. While this connotation might be adequate for describing data in the primary array (which ought to be IMAGE data), it is not necessarily appropriate for data in tables (e.g., a table entry that contains an integer flag to represent a quality code). Furthermore, while the standard does define the native data types of a SIMPLE machine in exquisite detail, it is unclear what, if any, SIMPLE machine representation is implied when the BSCALE and BZERO (or TSCAL and TZERO for tables) keywords are present. The definitions of BSCALE and BZERO (Section 5.2.2.5 of the NOST standard) imply that, if anything, after scaling, the result is to be interpreted as a "floating point" number. The standard does not specify the exact bit pattern that results after scaling - presumably a FITS reader must be prepared to deal with an infinite precision floating point number or with floating point numbers with exponents that are beyond the range of their IEEE representation. Although it is stated in the draft binary table extension standard (and has been echoed many times in various supporting documentation for FITS) that unsigned 16 bit integers may be represented by using signed 16 bit integers and setting BSCALE=1, BZERO=32768, this is not really what the main standard says. At best, one can say that the use of BSCALE = 1 and BZERO = 32768 results in a floating point number that can, if the user desires, be represented internally as a 16 bit unsigned integer, but one should not assume that the scaled FITS data value has that meaning attached. We will have situations where greater precision of meaning is needed - i.e., we want to make it clear that a particular item has a specific type and value. It would seem that binary tables ought to be able to do this. Although unsigned types are not native to most Fortrans, they are native to ANSI C compilers (and indeed much of the SDSS software is being written in C, in part because of the support for unsigned data types). We will also have need for a variable length storage mechanism - a typical purpose will be to store cutout images of objects detected in the survey, which vary in size for each object, with the object parameters stored in the main table. The heap facility that is proposed for binary tables serves our needs well but again with one exception - unsigned types are not supported in heap storage (one cannot even apply TSCAL and TZERO to heap data). Having ranted and raved for so long, I should at least make a specific proposal. IN THE MAIN STANDARD: 1. The definition of "physical value" should be clarified (is it is a pure number with dimensions attached?). The definition and use of "floating point" should be clarified, since it is used to mean both the IEEE format and the ASCII format, which are not uniquely related. 2. The exact meaning of scaling with BSCALE and BZERO be clarified. Does it imply that an exact bit pattern should result? 3. In the definitions of BSCALE and BZERO, the statements that values of 1. and 0. are implied if the keywords are not present should be omitted. [The reason for omitting them is that scaling implies a type conversion, which may not be the desired effect]. 4. In chapter 6 of the main standard, which defines the "official" data types, the statement that only these types may appear in both the primary array and extension tables should be changed to omit the reference on extension tables. (As it now stands, it would appear that the "X" and "L" data types in binary tables already violate this chapter). IN THE DRAFT BINARY TABLE STANDARD: 1. The goal of binary tables is to provide for the interchange of binary data. Each datum has a unique binary representation and a type. All "commonly used" data types are supported, including: (L) 8 bit Boolean T or F (A) 8 bit ASCII characters (I) 16 bit signed integers (U) 16 bit unsigned integers (J) 32 bit signed integers (V) 32 bit unsigned integers (E) 32 bit IEEE floating point numbers (K) 64 bit signed integers (W) 64 bit unsigned integers (D) 64 bit IEEE floating point numbers (X) A byte string of arbitrary length of undefined type (e.g., a bit mask). where the letter in parenthesis is the TFORM value (I have revised Rots' suggested designations for unsigned integers - sorry about that). 3. The presence of TSCAL and TZERO implies that the table data value is to be scaled and interpreted as a numerical value of data type float with no bit pattern implied. Their use is deprecated if the desired result is a unique binary representation and/or type. Are these proposals sensible? Radical? Do we have to invent yet another extension table type? How "Flexible" is FITS? Steve Kent From fitsbits-request Tue Mar 15 13:22:33 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1541" "" "15" "March" "1994" "17:55:35" "GMT" "Steve Allen" "sla at umbra.ucsc.edu" "<2m4sqn$3ae at darkstar.UCSC.EDU>" "29" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031517:55:35" "Yet another unsigned integers proposal" (number " " mark " Steve Allen Mar 15 29/1541 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m4pup$js7 at paperboy.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA12921; Tue, 15 Mar 94 13:22:33 EST Return-Path: Message-Id: <2m4sqn$3ae at darkstar.UCSC.EDU> Organization: UCO/Lick Observatory, Santa Cruz Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!agate!darkstar.UCSC.EDU!umbra.UCSC.EDU!sla References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov> From: sla at umbra.UCSC.EDU (Steve Allen) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: 15 Mar 1994 17:55:35 GMT In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>, Robert Hill wrote: >kent at steve.fnal.gov writes: >>(flip the high order bit and set TSCAL=1, TZERO = 32768), > >Actually, this is not quite right, I think. You don't `flip' the >high order bit, you just treat it as a data bit. No, you simply flip the high order bit and insert BSCALE and BZERO cards into the header. That is all. If your machine can actually add or subtract faster than it can flip a bit; good, then do that! This same subject has occurred in the Keck data acquisition system. The ADCs produce 16-bit unsigned numbers which are all shoved through a DSP on their way to the permanent storage device. With the DSP the processing required to flip the high bit would mean one more operation on the data while they are in the DSP. The end result is that the first processed bits would come out a few microseconds later, but the overall throughput need not be affected. The complication for Keck arises further down the software chain because all successive software, specifically the image preview, must understand that the data are unsigned. This requires systems integration between geographically distinct parallel development teams. -- -- -- -- -- -+- -- -- -- -- -- -- -- -+- -- -- -- -- -- Steve Allen | "Shameless Morris Enthusiasm" | sla at lick.ucsc.edu UCO/Lick Observatory | Yep, that's me. 6 up for | +1 408 459 3046 Santa Cruz, CA 95064 | Orange in Bloom, Sherborne, eh? | Seabright Morris From fitsbits-request Tue Mar 15 13:01:33 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4409" "" "15" "March" "1994" "17:06:33" "GMT" "Robert Hill" "bhill at trifle.gsfc.nasa.gov" "<2m4pup$js7 at paperboy.gsfc.nasa.gov>" "88" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031517:06:33" "Yet another unsigned integers proposal" (number " " mark " Robert Hill Mar 15 88/4409 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m2132$k78 at fnnews.fnal.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA12893; Tue, 15 Mar 94 13:01:33 EST Return-Path: Message-Id: <2m4pup$js7 at paperboy.gsfc.nasa.gov> Organization: NASA Goddard Space Flight Center -- InterNetNews site Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!news.umbc.edu!haven.umd.edu!ames!news.gsfc.nasa.gov!bhill References: <2m2132$k78 at fnnews.fnal.gov> From: bhill at trifle.gsfc.nasa.gov (Robert Hill) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: 15 Mar 1994 17:06:33 GMT kent at steve.fnal.gov writes: >The Sloan Digital Sky Survey (SDSS) project will generate of order 10 >terabytes >of raw data and 200 gigabytes of reduced data over a period of 5 years. [...] >We will use use a combination of basic FITS images, Ascii tables, and binary >tables to store and exchange a variety of raw and processed data. >Our raw data will all be - yes - 16 bit unsigned integers. >We are therefore faced with the same dilemma as Arnold Rots described in >a recent posting, and we have come to the same conclusion - it is very >desirable that the FITS standard provide explicit support for unsigned >data types, and the place to do so is in binary tables. In spite of the >recent thread arguing that FITS currently handles unsigned types properly >(flip the high order bit and set TSCAL=1, TZERO = 32768), Actually, this is not quite right, I think. You don't `flip' the high order bit, you just treat it as a data bit. A typical easy way to do this upon writing is to stuff the two bytes into the low-order locations within a zeroed-out four-byte word. Then you subtract 32768 (in four-byte arithmetic) and write out the low-order two bytes. The range 0..65535 is mapped directly into the range -32768..32767. [...] >The first problem we face is one of hard economics - we will be collecting >data at the rate of 7 Mbytes per second, and the current data acquisition >system will not handle the extra compute steps for even a simple bit >flip. Are you using off-the-shelf hardware interfaces with canned software, or do you have custom hardware interfaces with at least the inner acquisition routines written by your project in assembler? Your comments made me idly curious about your situation. In any case, with a conservative hardware scenario, e.g. a 25 MHz PC massaging the incoming data, I can see the conversion costing from 1/2 to 1 second per 7 Mbyte chunk, so yes, you might have a problem trying to do the conversion at that stage. [...] >The second (and more serious) problem arose when we tried to write >a general purpose reader and writer for FITS images and tables that >translates the data into their most "natural" format - what is the exact >meaning to be applied to data values in the primary array or in tables? >The FITS standard, unfortunately, blurs the distinction between the physical >value that one wants to represent and the FITS table representation of that >value, particularly when scaling is involved. [...] Actually, I interpret your question in a less philosophical way. What you want is for your project internal data files to also be exportable. That's fine, we do that too, and many projects these days seem to take the same approach. You can assure bit-by-bit recoverability for internal uses by forcing your own reading software to do the conversion y=1*x+32768 in integer arithmetic. The question is whether you really need to assure bit-by-bit recoverability for the export user -- in effect, whether you really need the capability of forcing the user to read the files in the same way you would. E.g., for most people, what's wrong with reading 16 bits into a 32-bit f.p. number with a 24-bit mantissa? Eight extra bits! I'm open minded about this proposal, not convinced either way as yet, and would like to see more discussion. Instrumentally oriented people (I'm one of them) have a tendency to say, `These are the bits we got, so they are the bits you're going to get.' If your instrumental format happens to be the same as one of the FITS representations, you get to have things both ways -- you can supply a physical calibration in BSCALE and BZERO. Many FITS readers allow the user, as an option, to read the FITS data array into a memory array of exactly the same type with no conversion. Thus bit-by-bit recoverability is a _de facto_ feature of FITS for many of us already, for certain data types. (Leaving aside the fact that it is an implicit feature of ASCII and of X-type bit fields in BINTABLEs.) Should this capability be given some kind of official status? If so, then the FITS community is pretty much obligated to try to support the most common instrumental formats as FITS data representations. Bob Hill -- Robert.S.Hill at gsfc.nasa.gov Not speaking for any of the following: Hughes STX Corp., Code 681, NASA/GSFC, Greenbelt, MD 20771 "Don't agree with me until I'm finished talking!" -- Darryl F. Zanuck From fitsbits-request Tue Mar 15 18:09:16 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3709" "Tue" "15" "March" "1994" "18:09:11" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403152309.AA11220 at tetra.Gsfc.NASA.Gov>" "66" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031523:09:11" "Unsigned Integers Proposal" (number " " mark " William Pence Mar 15 66/3709 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA13764; Tue, 15 Mar 94 18:09:16 EST Return-Path: Message-Id: <9403152309.AA11220 at tetra.Gsfc.NASA.Gov> From: pence at tetra.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Cc: pence at tetra.gsfc.nasa.gov Subject: Re: Unsigned Integers Proposal Date: Tue, 15 Mar 94 18:09:11 EST To further support the need for a FITS unsigned datatype in binary tables, the format for the data coming from the X-ray Timing Explorer Satellite (XTE) is briefly described bolow. In most of the previous discussion of this subject it has been assumed that the entire data stream consists of unsigned integers, and thus it is simple in principle to just scale all the data into signed integers. But in the XTE case, both signed and unsigned data values are intertwined in a complex way within the telemetry stream, so it can be quite complicated and computationally expensive to identify and scale just the unsigned telemetry values before writing the data to a FITS binary table. It is estimated that XTE will be generating close to 1 Terabyte of data per year (!), so the computational overhead of having to scale any unsigned integers before writing them to FITS files is a major concern. A somewhat simplified description of the XTE data follows: The XTE satellite dumps a buffer full of data at periodic time intervals. The contents of the buffer depends on the observing mode, (there are dozens of different possible modes). The natural way to store this information in a FITS binary table is to write each buffer of data, along with a time stamp indicating when the data were received, into one row of a binary table. The next buffer of data would be written to the next row of the table. All the buffers for a given observation would be contained in a single table. Each buffer of data can be futher broken down into individual components, each of which is logically a different column of the table. To give a simple example the, data buffer for a given observing mode might be 322 bytes long containing 2 scalar 8-bit telemetry values, followed by a histogram composed of 128 16-bit unsigned values, followed by another histogram of 64 8-bit unsigned values. (this is a hypothetical example, but it is at least representative of what real XTE data will look like). So, for this example the most natural lay out for the binary table would be to have 2 scalar columns followed by 2 vector columns. The 2 scalar columns would have a 'B' byte data type, and the second vector column would have a '64B' datatype. The first vector column contains 128 unsigned integers so ideally we would like to define the data type to be something like '128U' or '128I*UNSIGNED'. If FITS directly supports unsigned integers then it is conceptually simple to convert the XTE data to FITS: as the data comes in, first identify what observing mode it is in and write the appropriate FITS header defining all the columns in the table. Then just copy each buffer as it comes, byte by byte, into the table, one after another, until the observation is complete. But if FITS doesn't directly support unsigned integers, the job is much more complex. Before appending each new buffer to the table you have to figure out which bytes within the buffer represent unsigned integers, and then scale only them so that they fit within the range of the FITS signed integer data type. This process is bound to be rather computationally expensive. XTE will be generating about 2 Gbytes of data per day and there is already a lot of concern about how much computing power will be needed to process this amount of data, so efficiency is very important. No final decision has yet been made on how to deal with the XTE unsigned data values in the FITS files, but it looks rather likely at this point that the project will simply not have the computer resources necessary to scale the unsigned values and will have no choice except to invent a convention for directly encoding the unsigned values in FITS binary tables. From fitsbits-request Tue Mar 15 19:30:00 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2766" "Tue" "15" "March" "1994" "19:29:54" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403160029.AA11252 at tetra.Gsfc.NASA.Gov>" "55" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031600:29:54" "Yet another unsigned integers proposal" (number " " mark " William Pence Mar 15 55/2766 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA13858; Tue, 15 Mar 94 19:30:00 EST Return-Path: Message-Id: <9403160029.AA11252 at tetra.Gsfc.NASA.Gov> From: pence at tetra.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Cc: pence at tetra.gsfc.nasa.gov Subject: Re: Yet another unsigned integers proposal Date: Tue, 15 Mar 94 19:29:54 EST To follow up on Steve Kent's (14 March 1994) proposals for adding support for additional data types within FITS, I would make the following specific proposals: 1. Use BITPIX = -16 to represent unsigned 16-bit integers in primary arrays and IMAGE extensions. The SIMPLE keyword would need to be set = F until this convention is officially recognized. I think various groups have suggested and used this convention in the past, so approving this will simply legalize what is already common practice. 2. In binary tables, add the following new data types: (H) 8 bit signed integers (U) 16 bit unsigned integers (V) 32 bit unsigned integers Existing FITS readers will of course not understand these data types, and thus will be unable to parse the binary table at all until they are rewritten because they will not know how many bytes wide these columns are. To ease the transition, the following interium convention should be used until such time as these (H,U and V) data types are officially approved by the FITS committies. 3. As an interium solution, the following convention should be used to represent the new types of integers in binary tables: TFORMn = 'B:SIGNED' / this is a signed byte column TFORMn = 'I:UNSIGNED' / this is an 16 bit unsigned integer column TFORMn = 'J:UNSIGNED' / this is an 32-bit unsigned integer column The use of this convention should be deprecated once the H, U and V datatypes are approved. Note that this convention is entirely consistent with the current binary table definition document which allows any additional characters to follow the main datatype character (B, I or J in this case). The use of the ':' is consistent with usage in the 'Substring Array' convention in Appendix C of the Binary Table definition document. Unlike the case with the previous proposal (for using H, U and V to represent these datatypes) existing FITS readers will still be able to read the data in the table, although the reader may incorrectly interpret the values. FITS files using this convention would need to have SIMPLE = F Steve Kent also proposed supporting 64 bit signed and unsigned integers but I would hesitate to support this simply on the grounds that I don't know how to generate or read 64 bit integers on machines that don't support this as a native datatype. If someone could provide a robust set of routines for converting between 64-bit integers and other data types (e.g., 64-bit floats) then I might reconsider this. These proposals do not allow for 32-bit unsigned or 8-bit signed primary arrays, but I don't think the current need for these formats is very great. -Bill Pence From fitsbits-request Tue Mar 15 21:51:03 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2287" "Wed" "16" "March" "1994" "02:50:09" "GMT" "Don Wells" "dwells at nrao.edu" "" "48" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031602:50:09" "Unsigned Integers Proposal" (number " " mark " Don Wells Mar 16 48/2287 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403152309.AA11220 at tetra.Gsfc.NASA.Gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA13980; Tue, 15 Mar 94 21:51:03 EST Return-Path: Message-Id: Organization: nrao Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells References: <9403152309.AA11220 at tetra.Gsfc.NASA.Gov> From: dwells at nrao.edu (Don Wells) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Wed, 16 Mar 1994 02:50:09 GMT In article <9403152309.AA11220 at tetra.Gsfc.NASA.Gov> pence at tetra.gsfc.nasa.gov (William Pence) writes: ".. XTE will be generating.. 1 Terabyte.. per year .." The nominal design data rate of the VLBA (my previous RT project) is 4 GB/day, 1.5 TB/year. This sort of data rate is indeed large by historical standards, but it is fairly typical in modern systems. ".. [figuring] out which bytes within the buffer represent unsigned integers.. [will be].. rather computationally expensive.." It sounds to me like you have many records with the same structure. I suggest that a list structure be computed when the first record of a given type is encountered, so that the list structure can control efficient processing of each succeeding record. If the number of types is small enough, precompile optimized algorithms (e.g. maybe even using unrolled loops). ".. XTE will be generating about 2 Gbytes of data per day.." 2 Gbyte/day is 23 Kbyte/s. Even my old PC/XT clone can probably achieve that rate. ".. efficiency is very important.. rather likely.. project will simply not have the computer resources necessary to scale the unsigned values .." I doubt that this assertion will stand up to a critical review. -*- Modern RISC processors are sufficiently pipelined that they can overlap one or more cycles of register operations with the clock cycles of a memory-to-memory copy. For high-bandwidth nearly-I/O-limited systems designers should consider how many times the bits go into and out of main memory. That is where the action is! They should also examine the components which are doing the DMA transfers, how many memory cycles they steal from overlapped CPU operations, and how many memory-to-memory copies are being done in the I/O system. One or two register-to-register operations in a hot RISC engine will pale to insignificance in comparison to several redundant memory cycles, and such register operations may even be free-of-charge if pipelining occurs. -- Donald C. Wells Associate Scientist dwells at nrao.edu http://fits.cv.nrao.edu/~dwells National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road, Charlottesville, Virginia 22903-2475 USA From fitsbits-request Wed Mar 16 11:31:50 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["9132" "" "16" "March" "1994" "10:54" "EDT" "BARRY M. SCHLESINGER" "bschlesinger at nssdca.gsfc.nasa.gov" "<16MAR199410544747 at nssdca.gsfc.nasa.gov>" "194" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031614:54:00" "Yet another unsigned integers proposal" (number " " mark " BARRY M. SCHLESIN Mar 16 194/9132 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m2132$k78 at fnnews.fnal.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15454; Wed, 16 Mar 94 11:31:50 EST Return-Path: Message-Id: <16MAR199410544747 at nssdca.gsfc.nasa.gov> Organization: NASA - Goddard Space Flight Center Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!convex!cs.utexas.edu!swrinde!elroy.jpl.nasa.gov!ames!newsfeed.gsfc.nasa.gov!nssdca.gsfc.nasa.gov!bschlesinger References: <2m2132$k78 at fnnews.fnal.gov> From: bschlesinger at nssdca.gsfc.nasa.gov (BARRY M. SCHLESINGER) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: 16 Mar 1994 10:54 EDT In article <2m2132$k78 at fnnews.fnal.gov>, kent at steve.fnal.gov writes... > ... > Our project was strongly advised to use FITS wherever possible... > Our raw data will all be - yes - 16 bit unsigned integers... > ... the current data acquisition > system will not handle the extra compute steps for even a simple bit > flip. We are not about to spend the several thousand dollars needed > to add the extra compute muscle for this meaningless task. At present > we will write the raw data as FITS images with the unsupported BITPIX=-16 > convention and SIMPLE = F. Translating data from a form immediately readable only at the originating location to a standard form easily read elsewhere is not meaningless. In the original FITS paper, Wells, Greisen and Harten (1981, Astron. Astrophys. Suppl. 44,363) write "The usage of SIMPLE=F may be used for data storage or interchange within an institution or between users with similar interests and computers in those cases in which convenience overwhelms the need for standardization, but in which the outward form of this standard is still convenient to implement. We do not want to encourage any such departures from the basic FITS standard...." I interpret the intent to be that if a data set is going to be part of a general distributional archive, it should not include data with SIMPLE=F; that usage is for individual institutions or a relatively small common interest group. Discouraging SIMPLE=F arises from the fundamental purpose of standardization -- avoiding the need to develop new software each time a data set is received from a different location. While writing with a local format may cut costs at the originating installation, the requirement for receiving locations to write their own software will result in a greater total cost to the community. One approach that might be proposed would be to incorporate the proposed new format in some way in a new extension type, with a registered name, and distribute a detailed proposal. Those aspects of the data that do not use unsigned integers can be isolated and read by a standard FITS reader. And the new effort can be restricted to development of routines to read the new extensions, which could be distributed. But the purpose would not be to develop a special format for this one data set, but to start the process of extending the FITS standard. > ... what is the exact > meaning to be applied to data values in the primary array or in tables? > The FITS standard seems to > imply that a physical value is a pure number plus > physical dimensions, with no connotation of int, float, or signed attached > nor any specific binary representation.... > The definitions of BSCALE and BZERO (Section 5.2.2.5 of > the NOST standard) imply that, if anything, after scaling, the result is to > be interpreted as a "floating point" number. > The standard does not specify > the exact bit pattern that results after scaling FITS specifies the bit pattern for data transport. It provides information to the reader to determine whether the data before scaling are to be interpreted as integers (countable, discrete values), real (uncountable, continuous values), or text. There is no specification as to the bit pattern the reading software uses to store the data from the FITS file in its home machine *even before scaling*. That will depend on such issues as to whether the machine is most significant bit first or last, ones- or twos-complement, ASCII or EBCDIC. A physical value is a number plus its units. The units tell whether the value is inherently integer (number of counts), or real (Hertz). (The NOST document uses the term "floating point" rather than "real" because of possible confusion between real vs. imaginary and real vs. integer.) The interpretation is independent of whether or not the value is stored in a computer somewhere. > - presumably a FITS reader > must be prepared to deal with an infinite precision floating point number or > with floating point numbers with exponents that are beyond the range of their > IEEE representation. Using scaling with IEEE floating point is not recommended, precisely because of the danger of overflows. The User's Guide discusses the overflow problem specifically. Use of scaling does not rule out the possibility that the physical value is to be interpreted as an integer. Now, one potential problem here is in scaling from one number intended as an integer to another intended as an integer using a scale factor defined as floating point; round-off may introduce an error. This question needs to be discussed. > Although it is stated in the draft binary table extension > standard (and has been echoed many times in various supporting documentation > for FITS) that unsigned 16 bit integers may be represented by using signed 16 > bit integers and setting BSCALE=1, BZERO=32768, > this is not really what the main > standard says. At best, one can say that the use of BSCALE = 1 and BZERO = > 32768 results in a floating point number that can, if the user desires, be > represented internally as a 16 bit unsigned integer, but one should not > assume that the scaled FITS data value has that meaning attached. That's because this particular method of representing unsigned 16-bit integers is the one that is most widely recommended in the community consistent with the rules of FITS. It is not required; therefore, it is not part of the current standard. > We will have situations where greater precision of meaning is > needed - i.e., we want to make it clear that a particular item has a > specific type and value. It would seem that binary tables ought to be able > to do this. The TDISPn keywords of binary tables can be used to signify whether a table entry is to be considered an integer or a real number, and, if real, the number of significant digits. > ...We will also have need for a variable length storage mechanism - a typical > purpose will be to store cutout images > of objects detected in the survey, which > vary in size for each object, with the object parameters stored in the main > table. The heap facility that is proposed for binary tables serves our > needs well but again with one exception - unsigned types are not supported in > heap storage (one cannot even apply TSCAL and TZERO to heap data).... Not being able to scale heap values is a problem and probably an oversight. One possible proposal would be to change the rule to allow the use of TSCALn and TZEROn in fields with P data types, with the understanding that the scaling is to be applied to the heap data. > IN THE MAIN STANDARD: > 1. The definition of "physical value" should > be clarified (is it is a pure number with dimensions attached?). > The definition and use of "floating point" should be clarified, since > it is used to mean both the IEEE format and the ASCII format, which are not > uniquely related. The definitions should probably be discussed by the technical panel that revises the NOST standard to incorporate BINTABLE, IMAGE, and blocking. > 2. The exact meaning of scaling with BSCALE and BZERO be clarified. Does it > imply that an exact bit pattern should result? No. See above. FITS does not specify the bit patterns resulting from scaling. > 3. In the definitions of BSCALE and BZERO, the statements that values of > 1. and 0. are implied if the keywords are not present should be omitted. > [The reason for omitting them is that scaling implies a type conversion, > which may not be the desired effect]. Scaling does not imply conversion. > 4. In chapter 6 of the main standard, which defines the "official" data types, > the statement that only these types may appear in both the primary array > and extension tables should be changed to omit the reference on extension > tables. (As it now stands, it would appear that the "X" > and "L" data types in binary tables already violate this chapter). The requirement originated in the paper on generalized extensions. But the above question about X and L data types makes a good point. > IN THE DRAFT BINARY TABLE STANDARD: > > 1. .... All "commonly > used" data types are supported, including: > > ... > (U) 16 bit unsigned integers > ... > (V) 32 bit unsigned integers > ... > (K) 64 bit signed integers > (W) 64 bit unsigned integers The community must reach consensus on whether these new types are desirable. > 3. The presence of TSCAL and TZERO implies that the table data value is > to be scaled and interpreted as a numerical value of data type float with > no bit pattern implied. Their use is deprecated if the desired > result is a unique binary representation and/or type. Scaling does not imply that the data value is to be interpreted as real. FITS does not specify the binary representation of data once it leaves the FITS file. It specifies the way the data are to be interpreted, but scaling does not imply a floating point interpretation of the scaled values. Barry Schlesinger NOST FITS Support Office From fitsbits-request Wed Mar 16 13:20:39 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["717" "" "16" "March" "1994" "16:23:20" "GMT" "Robert Hill" "bhill at trifle.gsfc.nasa.gov" "<2m7bpo$a4v at paperboy.gsfc.nasa.gov>" "22" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031616:23:20" "Yet another unsigned integers proposal" (number " " mark " Robert Hill Mar 16 22/717 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m4sqn$3ae at darkstar.UCSC.EDU>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15581; Wed, 16 Mar 94 13:20:39 EST Return-Path: Message-Id: <2m7bpo$a4v at paperboy.gsfc.nasa.gov> Organization: NASA Goddard Space Flight Center -- InterNetNews site Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!news.umbc.edu!haven.umd.edu!ames!newsfeed.gsfc.nasa.gov!bhill References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov> <2m4sqn$3ae at darkstar.UCSC.EDU> From: bhill at trifle.gsfc.nasa.gov (Robert Hill) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: 16 Mar 1994 16:23:20 GMT >>kent at steve.fnal.gov writes: >>>(flip the high order bit and set TSCAL=1, TZERO = 32768), >Robert Hill wrote: >>Actually, this is not quite right, I think. You don't `flip' the >>high order bit, you just treat it as a data bit. >In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>, sla at umbra.UCSC.EDU (Steve Allen) writes: >No, you simply flip the high order bit and insert BSCALE and BZERO cards >into the header. That is all. By golly, you're right. Thanks for the lesson. Bob Hill -- Robert.S.Hill at gsfc.nasa.gov Not speaking for any of the following: Hughes STX Corp., Code 681, NASA/GSFC, Greenbelt, MD 20771 "Don't agree with me until I'm finished talking!" -- Darryl F. Zanuck From dwells Wed Mar 16 14:06:04 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2363" "Wed" "16" "March" "1994" "14:06:03" "EST" "Tom McGlynn" "MCGLYNN at grovx1.gsfc.nasa.gov" "<9403161906.AA15704 at fits.cv.nrao.edu>" "54" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031619:06:03" "Yet another unsigned integers proposal" (number " " mark " Tom McGlynn Mar 16 54/2363 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15710; Wed, 16 Mar 94 14:06:04 EST Return-Path: Message-Id: <9403161906.AA15704 at fits.cv.nrao.edu> From: Tom McGlynn Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits Subject: Re: Yet another unsigned integers proposal Date: Wed, 16 Mar 94 14:06:03 EST Forwarded from the fitsbits-request alias. -Don ------- start of forwarded message (RFC 934) ------- Message-Id: <940316134907.29200784 at grovx1.gsfc.nasa.gov> From: Tom McGlynn To: fitsbits-request at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: Wed, 16 Mar 1994 13:49:07 -0500 (EST) With regard to the unsigned int/scaling controversy, I've just done a few simple benchmarks on my DEC 5000/200. On my machine (and I would suspect on many others), the scaling is a simple subtraction, underflows just wrap around. **************************************************** Rate of scaling unsigned I*2's in megabytes/s. Using IDL: 1.8 Unoptimized C: 1.6 Optimized C (-O4): 10.2 I put a few checks in the loops and generally converted blocks of 10K-20K bytes. I also looked at the assembly code to ensure that the basic loops didn't get optimized out in the optimized C version. I don't believe this represents the best one can do if one is really pressed but already it seems clear that the time required to perform the conversions will be small compared to do I/O. I've never gotten more than 500K bytes/sec I/O rates in real applications. In practice we'd be converting smaller chunks which would take longer but this will be balanced by the fact that not everything is getting converted and one can probably use tricks to speed things up. At the 10 Mb/s rate it will take only 200 s per day to handle XTE data assuming that all data is transformed -- and this is by no means on state of the art computers. I think Bill's argument that it's hard to know which data to convert is specious. If we know enough to write out the FITS header with appropriate keyword values to describe which words are going to be unsigned, then obviously we know which words are going to be unsigned. Just as FITSIO uses similar information to handle host to IEEE conversions when writing FITS files, software can use user supplied keywords to automagically perform the byte scaling. I would be surprised if we could not add such a capability within FITSIO itself without violating the current FITS standards. All this puts me in the camp of those who feel that the case for incorporating unsigned int's is not demonstrated. Tom McGlynn Compton Observatory Science Support Center ------- end ------- From fitsbits-request Wed Mar 16 15:47:54 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2955" "Wed" "16" "March" "1994" "20:46:46" "GMT" "Don Wells" "dwells at nrao.edu" "" "58" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031620:46:46" "Yet another unsigned integers proposal" (number " " mark " Don Wells Mar 16 58/2955 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<2m7bpo$a4v at paperboy.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15829; Wed, 16 Mar 94 15:47:54 EST Return-Path: Message-Id: Organization: nrao Path: saips.cv.nrao.edu!news.cv.nrao.edu!dwells References: <2m2132$k78 at fnnews.fnal.gov> <2m4pup$js7 at paperboy.gsfc.nasa.gov> <2m4sqn$3ae at darkstar.UCSC.EDU> <2m7bpo$a4v at paperboy.gsfc.nasa.gov> From: dwells at nrao.edu (Don Wells) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: Wed, 16 Mar 1994 20:46:46 GMT In article <2m7bpo$a4v at paperboy.gsfc.nasa.gov> bhill at trifle.gsfc.nasa.gov (Robert Hill) writes: ... RH> In article <2m4pup$js7 at paperboy.gsfc.nasa.gov>, sla at umbra.UCSC.EDU (Steve Allen) writes: SA> No, you simply flip the high order bit and insert BSCALE and SA> BZERO cards into the header. That is all. RH> By golly, you're right. Thanks for the lesson. The bit pattern to be used is 8000-hex. In 16-bit unsigned it is 32768 (8*4096). In 16-bit 2-s complement it is both +32768 *and* -32768. The conversion can be produced by *three* different machine opcodes: XOR 8000 (exclusive-OR to flip the high bit) SUB 8000 (subtract +32768) ADD 8000 (add -32768) Tutorial for newcomers: In 2-s complement arithmetic you negate a number by complementing the bit pattern and adding one; the complement of 8000 is 7fff and adding 1 generates a carry and you are back to 8000. In 2-s complement arithmetic you throw away carries on addition. If your unsigned number is 32769=8001, and you use the SUB (subtract) opcode, the hardware subtracts 8000=32768 by complementing to get 7fff, adding 1 to get 8000, and then adding that to 8001 to get 0001 (the value to put in the FITS file), after the carry (10000) is discarded. The ADD and XOR opcodes produce the same result as the SUB opcode. Another example is 0001 unsigned, which becomes 8001=-32767 (+32767=7fff, negation is 8000+1) after conversion. Note that the exact same set of three operators are used to perform the reverse conversion from 2-s complement 16-bit signed to 16-bit unsigned. The two examples cited in the previous paragraph were chosen to demonstrate this invertibility property of the conversion operation. Historical note about arithmetic: The first FITS tape passed in interchange in May 1979 was processed by a CDC 6400, a 60-bit *ONES-COMPLEMENT* computer. In the 1-s complement arithmetic system you negate by complementing the bits, and when adding you add carries from the most significant bit back into the least significant bit (often called "end-around" carry). During the early days of FITS a major challenge for people writing and reading FITS files on various machines was to understand how to interchange between 1-s complement and 2-s complement architectures. Mercifully, almost all of the 1-s complement machines have gone off to computer Valhalla. FITS itself was carefully defined to use 2-s complement. I am intrigued by the fact that each number system has a number whose negation is arithmetically equal to itself. In the 1-s complement system the number is zero ((+0)==(-0) in C-speak), while in 2-s complement it is 32768 ((+32768)==(-32768)). -- Donald C. Wells Associate Scientist dwells at nrao.edu http://fits.cv.nrao.edu/~dwells National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road, Charlottesville, Virginia 22903-2475 USA From fitsbits-request Wed Mar 16 16:30:55 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1172" "Wed" "16" "March" "1994" "16:30:40" "EST" "William Pence" "pence at tetra.gsfc.nasa.gov" "<9403162130.AA00273 at tetra.Gsfc.NASA.Gov>" "22" "TSCAL and TZEROn with P datatypes" "^From:" nil nil "3" "1994031621:30:40" "TSCAL and TZEROn with P datatypes" (number " " mark " William Pence Mar 16 22/1172 " thread-indent "\"TSCAL and TZEROn with P datatypes\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA15915; Wed, 16 Mar 94 16:30:55 EST Return-Path: Message-Id: <9403162130.AA00273 at tetra.Gsfc.NASA.Gov> From: pence at tetra.gsfc.nasa.gov (William Pence) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: TSCAL and TZEROn with P datatypes Date: Wed, 16 Mar 94 16:30:40 EST Barry Schlesinger wrote: >> table. The heap facility that is proposed for binary tables serves our >> needs well but again with one exception - unsigned types are not supported in >> heap storage (one cannot even apply TSCAL and TZERO to heap data).... > >Not being able to scale heap values is a problem and probably an >oversight. One possible proposal would be to change the rule to allow >the use of TSCALn and TZEROn in fields with P data types, with the >understanding that the scaling is to be applied to the heap data. I also think this was just an oversight. The main text of the binary table proposal is correct in stating that the use of the TSCALn keyword is not defined for the P format fields, because the use of the P format is also not defined in the main text. The use of TSCALn and TZEROn to refer to the heap data in P type fields probably should have been discussed in appendix A, however, where the 'variable length array' facility is described in detail. Since the convention described in Appendix A is "still undergoing trials" and has not been offically voted on, there should be little problem in correcting this oversight. -Bill Pence From fitsbits-request Wed Mar 16 21:52:36 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1078" "Wed" "16" "March" "1994" "21:52" "EST" "Wayne H. Warren, Jr." "W3WHW at gibbs.gsfc.nasa.gov" "<9403170252.AA16169 at fits.cv.nrao.edu>" "26" "1's Complement Arithmetic" "^From:" nil nil "3" "1994031702:52:00" "1's Complement Arithmetic" (number " " mark " Wayne H. Warren, Mar 16 26/1078 " thread-indent "\"1's Complement Arithmetic\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA16175; Wed, 16 Mar 94 21:52:36 EST Return-Path: Message-Id: <9403170252.AA16169 at fits.cv.nrao.edu> From: "Wayne H. Warren Jr. 474-0814" (Code 681/GSFC) Sender: fitsbits-request at fits.CV.NRAO.EDU To: FITSBITS Subject: 1's Complement Arithmetic Date: Wed, 16 Mar 94 21:52 EST On 16 Mar at 15:47:54EST, dwells at NRAO.EDU(Don Wells) wrote: > . . . Mercifully, almost all of the 1-s complement machines have >gone off to computer Valhalla. FITS itself was carefully defined to >use 2-s complement. >I am intrigued by the fact that each number system has a number whose >negation is arithmetically equal to itself. In the 1-s complement >system the number is zero ((+0)==(-0) in C-speak), while in 2-s >complement it is 32768 ((+32768)==(-32768)). Not disagreeing with Don's statement, it certainly was convenient for an observational astronomer working with declinations in sexigessimal form that the 1's complement CDC machines distinguished between +0 and -0 and one didn't need to treat the signs independently. I can't recount how many times I've seen errors in extinction coefficients and other results because of the -0 bug in someone's program. Wayne H. Warren Jr. Hughes STX Corporation Laboratory for Astronomy and Solar Physics Code 681 NASA GSFC Greenbelt MD 20771-0001 (301)286-5419 (GSFC); (301)441-4086 (HSTX) w3whw at gibbs.gsfc.nasa.gov From fitsbits-request Fri Mar 18 11:18:43 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3932" "" "18" "March" "1994" "15:50:01" "GMT" "Tom Nicinski" "nicinski at fncasefnal.fnal.gov" "<2mcij9$p9s at fnnews.fnal.gov>" "78" "Re: Yet another unsigned integers proposal" "^From:" nil nil "3" "1994031815:50:01" "Yet another unsigned integers proposal" (number " " mark " Tom Nicinski Mar 18 78/3932 " thread-indent "\"Re: Yet another unsigned integers proposal\"\n") "<9403160029.AA11252 at tetra.Gsfc.NASA.Gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA19703; Fri, 18 Mar 94 11:18:43 EST Return-Path: Message-Id: <2mcij9$p9s at fnnews.fnal.gov> Organization: FERMILAB, Batavia, IL Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!darwin.sura.net!rsg1.er.usgs.gov!stc06r.CTD.ORNL.GOV!fnnews.fnal.gov!fncase!nicinski References: <9403160029.AA11252 at tetra.Gsfc.NASA.Gov> Reply-To: nicinski at fnal.fnal.gov From: nicinski at fncasefnal.fnal.gov (Tom Nicinski) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Yet another unsigned integers proposal Date: 18 Mar 1994 15:50:01 GMT I am also a big supporter of a richer environment -- in particular, supporting additional primitive data types that are quite "natural" and common. In this, thread, that means unsigned integers and signed 8-bit integers. Thus, I agree with Steve Kent's proposal. I also have some comments on William Pence's post. pence at tetra.gsfc.nasa.gov (William Pence) writes: [deleted ...] |> 1. Use BITPIX = -16 to represent unsigned 16-bit integers in primary |> arrays and IMAGE extensions. The SIMPLE keyword would need to |> be set = F until this convention is officially recognized. |> I think various groups have suggested and used this convention in the |> past, so approving this will simply legalize what is already common practice. Although I do not have any suggestions, this convention will only confuse the standard. There are two problems: 1) negative BITPIX values indicate floating point values and 2) how do you handle 32-bit or 64-bit or n-bit integers. |> 2. In binary tables, add the following new data types: |> |> (H) 8 bit signed integers I chose W for 8-bit signed integers for my Binary Table reader. [deleted ...] |> 3. As an interium solution, the following convention should be used |> to represent the new types of integers in binary tables: |> |> TFORMn = 'B:SIGNED' / this is a signed byte column |> TFORMn = 'I:UNSIGNED' / this is an 16 bit unsigned integer column |> TFORMn = 'J:UNSIGNED' / this is an 32-bit unsigned integer column !> |> The use of this convention should be deprecated once the H, U and V |> datatypes are approved. I have an aversion to interim proposals, since they kind of become standard. If you're willing to use SIMPLE=F, then using proposal point 2. (above) might as well be used from the start. [deleted ...] |> Steve Kent also proposed supporting 64 bit signed and unsigned integers |> but I would hesitate to support this simply on the grounds that I don't |> know how to generate or read 64 bit integers on machines that don't |> support this as a native datatype. If someone could provide a robust |> set of routines for converting between 64-bit integers and other |> data types (e.g., 64-bit floats) then I might reconsider this. |> |> These proposals do not allow for 32-bit unsigned or 8-bit signed primary |> arrays, but I don't think the current need for these formats is very great. These last two paragraphs highlight what I see as a severe problem with the how people think about FITS. Rather than trying to provide a richer and consistent environment, peoples' own needs are applied to the stan- dard. Why don't people forsee the need for 8-bit signed primary arrays? Maybe your application doesn't need them, but someone else's might. Why argue against 64-bit integers? If your platform doesn't like them, that should not preclude their inclusion in the standard. You might argue then that the Binary Table complex numbers should be removed. I never use FORTRAN and none of the ANSI C compilers and libraries I use recognize complex numbers as a primitive data type. Now, here's a significant part of the computing world that cannot handle complex numbers naturally; so, why are they in the proposed standard? I'm new to the FITS world, but it certainly seems that the 'F' in Flexible certainly does not apply. This standard is not evolving to take into account real world problems and needs. +--- . .' ----+--------------------------------+----------------------------+ | /| | | | | Tom Nicinski / MS 234 | nicinski at fnal.fnal.gov | | +-+-+\ | | Fermi National Accelerator Lab | _ | | . | | | \| | P.O. Box 500 | [o] | | `--' | | | | Batavia, IL 60510-0500 | /|\ | +---- -' -----+--------------------------------+----------------------------+ From fitsbits-request Fri Mar 18 11:38:31 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["6655" "Fri" "18" "March" "1994" "11:38:13" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403181638.AA11251 at xebec.gsfc.nasa.gov>" "129" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994031816:38:13" "Unsigned Integers Proposal" (number " " mark " Arnold Rots Mar 18 129/6655 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA19750; Fri, 18 Mar 94 11:38:31 EST Return-Path: Message-Id: <9403181638.AA11251 at xebec.gsfc.nasa.gov> From: arots at xebec.gsfc.nasa.gov (Arnold Rots) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Cc: arots at xebec.gsfc.nasa.gov Subject: Re: Unsigned Integers Proposal Date: Fri, 18 Mar 94 11:38:13 EST I have watched the debate over the issue of introducing unsigned integers into the FITS Binary Tables closely and my conclusion (not surprising, since I started this round in the discussion) is: we should do it and we'd better do it quickly. The Binary Table standard contains a fairly large and fairly complete set of data types. The criterion for defining this set cannot have been to arrive at a minimum set. Considering the fact that it contains complex and double complex, the approach must have been to include all meaningful data types commonly in use among astronomers. I would submit that unsigned integers ought to be included in this collection, simply on the basis of generality. Frankly, I think, there is too much parochial thinking going on. I have no intention to pick on anyone in particular, but I understand very well where Don Wells is coming from: in radio astronomy floats, doubles, complex, signed integers and scaled integers have traditionally dominated the zoo of data types. Among the latter especially the 16 bit variety, as illustrated by Don's remark: > I am intrigued by the fact that each number system has a number whose > negation is arithmetically equal to itself. In the 1-s complement > system the number is zero ((+0)==(-0) in C-speak), while in 2-s > complement it is 32768 ((+32768)==(-32768)). Note that this implicitly assumes that all integers are 16-bit signed integers. I am sure that Don did not mean it that way, I but mention it to illustrate the implicit assumptions we are all making. My point is that in the rest (i.e., non-radio) of astronomy, complex (for instance) does not play such a big role, but unsigned integers do. Again, if we *can* store original values in FITS tables, I feel it *should* be done. Every extra line of code increases complexity and chances for errors; this is of specific concern to those among us who work on software that will produce basic archives: the number of operations should be kept to an absolute minimum, on the one hand, while the archived product should provide as good data access as possible. Unsigned integers would help tremendously in that regard. All I am asking for is that unsigned integers be recognized in Binary Tables only. The Binary Tables proposal has only just been approved, so, since unsigned integers are unavoidably coming, as Bill Pence pointed out, the sooner they are recognized, the easier the phasing in will be. On to the XTE case. Don Wells and Tom McGlynn have provided us with some quick calculations that all of us (I hope) are capable of carrying out but that, in their simplicity, completely miss the point. The problem is not in the actual scaling itself, though I might point out that we will decline Don's offer of his PC/XT clone, since we hope to be able to process a day's worth of telemetry in considerably less than 24 hours. Figuring out when, where, and how to scale, involves at least the same amount of processing. More important, though, is the added complexity. Of course, it can be determined what needs to be scaled by what. And, of course, it can be table driven. But we are talking about a significant increase in code complexity. XTE will generate something of the order of a TByte a year in FITS files, containing data in something of the order of a thousand data modes (Bill Pence was a little optimistic with his "dozens"). In order to keep this from turning into a configuration control and software maintenance nightmare, we have neatly encapsulated the various functions: the data management subsystem (in C++) can provide any client (not just a FITS writer) with data values; the FITS writer (in C) takes the values and dumps them into FITS Binary Tables. We don't want to force the former to comply with FITS standards (for obvious reasons), while we also want to keep the latter as simple as possible. I cannot over-stress the importance of this point: this is crucial software that creates *the* data archive for this mission; every increase in its complexity increases the risk of errors, in implementation and maintenance. Unnecessary capabilities and sophistication is just not acceptable for this type of software. The FITS writer currently has no direct knowledge of what it is writing (at least not while it happens) because there is no need for that capability; strict encapsulation and design simplicity have made this piece of software highly reliable - I would like to keep it that way. For Binary Tables, I would propose the following data types. New ones are indicated with an asterisk (*). I am not quite sure whether 64-bit integers are warranted at the moment, but considering that we'd better do this right for the next 10 years, or so, I have included them. I hope nobody will mistake "H" for Hollerith. (L) 8 bit logical (X) 1 bit item (A) 8 bit ASCII character (E) 32 bit float (D) 64 bit double float (C) 64 bit complex float (2 times 32 bits) (M)128 bit complex float (2 times 64 bits) * (H) 8 bit signed integer (I) 16 bit signed integer (J) 32 bit signed integer * (K) 64 bit signed integer (B) 8 bit unsigned integer * (U) 16 bit unsigned integer * (V) 32 bit unsigned integer * (W) 64 bit unsigned integer In addition, of course, there is the P. There is one more issue that I would like to address. Both Bill Pence and Barry Schlesinger have hinted at how unsigned integers could be used in FITS files until such a time as when we have an official standard. I am very hesitant to use devices like: TFORMn = 'I:UNSIGNED' / this is an 16 bit unsigned integer column or: SIMPLE = F / This is my type of FITS file or: XTENSION= 'MYBINTABLE' / A BINTABLE with unsigned integers as interim measures. If a standard got established at any point during the XTE mission, we would either end up with an archive that contains different types of files, or face the task of reprocessing all previous data. Neither is palatable. The point is that such devices are quite acceptable for limited amounts of, more or less, volatile data. For definitive archives (which, one hopes, would also outlive software systems that are currently in use) other considerations come into play. I sincerely hope that the community will move forward with this expeditiously. As Bill Pence remarked, this problem is not going away; therefore, the sooner we deal with it, the better. And to require everybody to deal with the lack of unsigned integers in the future, because "we have always managed to do without" in the past, is, in my view, a dangerous position. - Arnold Rots From fitsbits-request Fri Mar 18 12:53:02 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["7632" "Fri" "18" "March" "1994" "18:57:27" "+0100" "Lucio Chiappetti" "lucio at ifctr.mi.cnr.it" "" "153" "Re: Unsigned Integers Proposal" "^Resent-Date:" nil nil "3" "1994031817:57:27" "Unsigned Integers Proposal" (number " " mark " Lucio Chiappetti Mar 18 153/7632 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403181638.AA11251 at xebec.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA19849; Fri, 18 Mar 94 12:53:02 EST Return-Path: Organization: Istituto di Fisica Cosmica e Tecnologie Relative Reply-To: Lucio Chiappetti In-Reply-To: <9403181638.AA11251 at xebec.gsfc.nasa.gov> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Resent-Date: Fri, 18 Mar 1994 18:57:27 +0100 (MET) Resent-From: fitsbits-request Resent-Message-Id: <9403181753.AA19849 at fits.cv.nrao.edu> Resent-Sender: Lucio Chiappetti From: Lucio Chiappetti Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Fri, 18 Mar 1994 18:57:27 +0100 (MET) Here are my 0.02 currency units (Ukrainian coupons ?) about the "unsigned integers in FITS" controversy, which I have been following for a while. I apologize if in the posting below I do not acknowledge all previous contributors by name, but I am quoting off my memory. --------- I have always been liking Ockham's razor, in this particular case "data types non sunt multiplicanda praeter necessitatem" : the number of data types should be kept to a minimum. So just on this ground, I have looked favourably at the various contributors which argued very convincingly that unsigned integers can be handled with the BSCALE/BZERO convention without an unreasonable CPU load and therefore remain of the opinion that one should live with a LIMITED number of data types. ------------ The usage of a minimal subset could also be helpful to developers in different languages. Personally I use only Fortran, and I would be annoyed to deal with unsigned integers which are not supported by the language (see however below for bytes et al.). One contributor noted that C programmers may feel the same about the COMPLEX data type (actually I believe I've used it in Fortran only once, and that was not serious work, a Mandelbrot set game). In my opinion the really useful set (which does not mean I propose to delete what is already in the standard beyond it) is : - 8-bit characters B or bytes (unsigned ....) C - 16-bit signed integers B - 32-bit signed integers C - 32-bit IEEE floating A - 64-bit IEEE floating B The letters ABCD indicates decreasing "priority of use". In fact, talking of numeric values, there is usually little reason to prefer INTEGER values (with the annoyance of scaling) to REAL values, except perhaps sometimes for space reasons (or for "conservative" reasons, see below). E.g. if I want to store a spectrum, I store it in counts/s (REAL) with associated errors (REAL) and not in counts (INTEGER), etc. ------------- One argument which was put forward by some contributors, and which I term here for brevity "the conservative argument" is : "we should store data as close as they come out from the instrument" (this arose, as far as I remember, both from CCD users and from XTE) i) this is sometimes reasonable, but cannot it be done within the existing conventions ? I often had to deal with 8-bit and 16-bit data from X-ray detectors. 8-bit values might be some ADC channel level, or counts in an histogram (and in both cases usually are in the full unsigned range 0-255). 16-bit values may be counts in an histogram, but in this case although nominally unsigned the probability of being above 32767 is really low. I hardly see the need of using the full range of a 32-bit (signed or unsigned) for anything (but on-board clocks, which deserve a special treatment anyhow). One should separate the way the data are STORED in a file, and the way they are ELABORATED in variables in a program. As a matter of fact I operate only on INTEGER*4 variables. I note that FITS (both the "original" FITS and the Binary Tables) consider 8-bit values to be unsigned (unlike other integers, and unlike the convention for the non-standard BYTE data type in some Fortrans; in fact I never use the BYTE data type, but always read the unsigned 8-bit value in an INTEGER*4 with some CHAR/ICHAR trick). Copying the two bytes of an INTEGER*2 into an INTEGER*4 preserving its value should be reasonably easy with some EQUIVALENCE tricks. I refer also to previous postings about BSCALE, BZERO etc. ii) there are cases in which the conservative argument lead to the extrema would cause proliferation of strange data types. It often happens that satellite data have a "funny" number of bits (either because they are "naturally" produced this way in the electronics, or because the need of the telemetry require to chop off redundant 0 bits) : so one can have a 10-bit energy, a 3-bit detector id, etc. In this case I would consider extremely annoying for the general observer to deal with such data types. However that is what would be implied if one follows strictly the "store data as they come out of the instrument" approach. For instance, the requirement which was put on the Ground Segment Contractor for SAX, was to reformat such data in a simple way, i.e. expand all 1- to 7-bit fields into an 8-bit 0-padded field, all 9- to 15-bit fields into a 16-bit field, and all 17- to 31-bit fields into a 32-bit field (the "natural" sizes which can be handled as said in (i) above). This way data are kept "close to the form they come out of the instrument" (e.g. the partitioning in telemetry packets is preserved, and no destructive processing is performed), but they are reformatted in a friendly way. I note that FITS does not consider a true bit data type, since the rX binary table type requires the same padding to n*8 bit, but unfortunately (unexplainably ?) requires left-justification. iii) a philosophical argument : when is something to be called FITS. As far as I remember the postings about XTE were mentioning "telemetry data buffers" downlinked from the spacecraft in a "huge number" (quoted as "dozens" to "thousands" sic!) "of data types", which had to be "stored as close as possible as they come down from the spacecraft", but at the same time be FITS. And the proposed solution was to store each "buffer" as a row in a FITS table. Now, while I could see good reasons to keep the "buffers" (more or less corresponding to the "telemetry packets" mentioned in ii above) as untouched as possible (but I hope this does not extend to keeping 5-bit fields ... again see ii above), what I fail to see is why one should try to encapsulate this which is not (and by good reasons) a FITS structure into a "nominalistic" FITS wrapper. If one had to do some processing, and export processed data, I would agree about the case for using an existing FITS structure, or devising an easy one. But in this and similar cases, I find strange the following line of reasoning : - Files must be written in FITS - My data are not compatible with FITS - The FITS convention must be changed so that my data can be enclosed unchanged and the result still be called FITS, q.e.d. This seems to me a strange argument, like I were saying : - English uses the latin alphabet - All documents must be written in English - My "data" are written in Hungarian. - I write my document putting in each page a heading saying "what follows is hungarian text" followed by one page of the original text. Of course I use the latin alphabet. - The document is written in English, q.e.d. ---------------------------------------------------------------------------- A member of G.ASS : Group for Astronomical Software Support ---------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR | Ma te' vugl' da' quost avis a ti' Orsign via Bassini 15 - I-20133 Milano | Buttet rabios intant te se' pisnign Internet: LUCIO at IFCTR.MI.CNR.IT | Decnet: IFCTR::LUCIO | (Rabisch, II 46, 119-120) ---------------------------------------------------------------------------- From fitsbits-request Fri Mar 18 16:58:49 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["202" "" "18" "March" "1994" "16:56:12" "GMT" "Tom Lipovits" "ac448 at cleveland.freenet.edu" "<2mcmfc$g3d at usenet.INS.CWRU.Edu>" "7" "Is there a FITS image viewer for DOS?" "^From:" nil nil "3" "1994031816:56:12" "Is there a FITS image viewer for DOS?" (number " " mark " Tom Lipovits Mar 18 7/202 " thread-indent "\"Is there a FITS image viewer for DOS?\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA20108; Fri, 18 Mar 94 16:58:49 EST Return-Path: Message-Id: <2mcmfc$g3d at usenet.INS.CWRU.Edu> Organization: Case Western Reserve University, Cleveland, Ohio (USA) Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!usenet.ins.cwru.edu!cleveland.Freenet.Edu!ac448 From: ac448 at cleveland.Freenet.Edu (Tom Lipovits) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Is there a FITS image viewer for DOS? Date: 18 Mar 1994 16:56:12 GMT I am recently interested in the FITS images and am unable to find any image viewers on local bulletin boards. Any information would be helpful. Thank you _I_don't_have_an_employer_ .Tom. From fitsbits-request Sat Mar 19 23:00:54 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["7008" "" "20" "March" "1994" "03:43:43" "GMT" "Martin Shepherd" "mcs at goblin.caltech.edu" "" "145" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032003:43:43" "Unsigned Integers Proposal" (number " " mark " Martin Shepherd Mar 20 145/7008 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403181638.AA11251 at xebec.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA21952; Sat, 19 Mar 94 23:00:54 EST Return-Path: Message-Id: Organization: California Institute of Technology. Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!usenet.cis.ufl.edu!usenet.ufl.edu!gatech!swrinde!elroy.jpl.nasa.gov!netline-fddi.jpl.nasa.gov!nntp-server.caltech.edu!nntp-server.caltech.edu!mcs References: <9403181638.AA11251 at xebec.gsfc.nasa.gov> From: mcs at goblin.caltech.edu (Martin Shepherd) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 20 Mar 1994 03:43:43 GMT In article <9403181638.AA11251 at xebec.gsfc.nasa.gov> arots at xebec.gsfc.nasa.gov (Arnold Rots) writes: :I have watched the debate over the issue of introducing unsigned :integers into the FITS Binary Tables closely and my conclusion (not :surprising, since I started this round in the discussion) is: we :should do it and we'd better do it quickly. I have also been watching the debate, but have come to the opposite conclusion. My personal viewpoint is that requiring FITS reader software to become more complex just so that a few type-specific FITS writers can be written more easily is totally unreasonable. :The Binary Table standard contains a fairly large and fairly complete :set of data types. The criterion for defining this set cannot have :been to arrive at a minimum set. Considering the fact that it :contains complex and double complex, the approach must have been to :include all meaningful data types commonly in use among astronomers. :I would submit that unsigned integers ought to be included in this :collection, simply on the basis of generality. :Frankly, I think, there is too much parochial thinking going on. This goes both ways. Requiring that FITS be molded to your specialized needs without regard to the problems that it would produce for the rest of the community is surely a parochial viewpoint. : I :have no intention to pick on anyone in particular, but I understand :very well where Don Wells is coming from: in radio astronomy floats, :doubles, complex, signed integers and scaled integers have :traditionally dominated the zoo of data types. This is unreasonable. It seems far more likely that the chosen types were based on those then available in FORTRAN-77 rather than a conspiracy among radio astronomers. While I personally prefer C, I can see the point of not supporting types that at least one language can't handle. Admittedly, from this viewpoint, COMPLEX types should not have been included, but it is undeniably too late to change that now. :[Text omitted] :Again, if we *can* store original values in FITS tables, I feel it :*should* be done. In an ideal world perhaps, but in the real world it is pointless because many architectures don't provide types with the same bit representation as FITS decrees, and are thus unable to recover the exact value that was written. At a time in which object oriented programming is showing such promise, isn't it becoming clear that the particular form in which data are stored is secondary to the interface used to access them? FITS allows such abstraction, and should be commended for this. If you care about how your data is represented at the bit level then don't use FITS - it wasn't designed for that. : Every extra line of code increases complexity and :chances for errors; this is of specific concern to those among us who :work on software that will produce basic archives: the number of :operations should be kept to an absolute minimum, on the one hand, :while the archived product should provide as good data access as :possible. Unsigned integers would help tremendously in that regard. Again the same argument can be presented from the opposite viewpoint. Adding new types increases the size and complexity of general FITS readers. You are requiring that FITS readers be slowed down so that your simpler, type-specific FITS writer can run faster. To make this point clearer, consider that in a general FITS reader every new type must be accompanied by at least three specialized sets of functions: 1. Machine-specific routines to convert between each FITS type and equivalent host-architecture data-types. This makes it hard to port to new architectures - more so if there are a lot of types. 2. Conversion functions between all other compatible types supported by the FITS library (eg. int to unsigned int, int to float...). For N types there are O(NxN) such conversions required. This has a significant impact on speed and complexity. 3. Functions to test for and set type-specific blank/null values. Obviously the more functions there are and the greater the number of possible routes through a FITS reader, the more difficult it becomes to write a general FITS library, the harder it is to test the result and the slower it becomes to read FITS. This is the impact on the rest of the community that I was referring to above, in turning your accusation of parochial thinking back on you. :[Text omitted] :On to the XTE case. Don Wells and Tom McGlynn have provided us with :some quick calculations that all of us (I hope) are capable of :carrying out but that, in their simplicity, completely miss the point. :The problem is not in the actual scaling itself, though I might point :out that we will decline Don's offer of his PC/XT clone, since we hope :to be able to process a day's worth of telemetry in considerably less :than 24 hours. Figuring out when, where, and how to scale, involves at :least the same amount of processing. This makes it sound as though XTE was designed without regard for how its output would be pipe-lined? Should the rest of the community have to shoulder the burden of such a design flaw? Can the satellite software not be changed to better match your archival requirements? :More important, though, is the added complexity. Of course, it can be :determined what needs to be scaled by what. And, of course, it can be :table driven. But we are talking about a significant increase in code :complexity. And should the rest of the community have to make their software more complex so that your's need not be? :[Text omitted] :that capability; strict encapsulation and design simplicity have made :this piece of software highly reliable - I would like to keep it that :way. I feel the same way about my general FITS library, and since it is much easier to write a specific form of FITS than to read the general case, shouldn't the burden be taken by the former? After all, during data processing, FITS files often have to be read multiple times, so speed and correctness during reading is very important to the end-user. :[More text omitted] :I sincerely hope that the community will move forward with this :expeditiously. As Bill Pence remarked, this problem is not going :away; therefore, the sooner we deal with it, the better. And to :require everybody to deal with the lack of unsigned integers in the :future, because "we have always managed to do without" in the past, :is, in my view, a dangerous position. Given that computers are getting faster all the time, and that storage facilities are getting more capacious, the argument for unsigned types may soon be outdated. I can't see how it could get more pressing with time. Unfortunately the same can't be said for 64 bit integers, but at least they only represent one extra type to add. : - Arnold Rots No offense intended. I just hope that you can find a simpler way to fix XTE without requiring radical changes to FITS. Martin Shepherd (mcs at phobos.caltech.edu) From fitsbits-request Mon Mar 21 04:31:49 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["460" "Mon" "21" "March" "1994" "09:12:35" "GMT" "John Boyer" "boyer at rd.eng.bbc.co.uk" "" "16" "Re: Is there a FITS image viewer for DOS?" "^From:" nil nil "3" "1994032109:12:35" "Is there a FITS image viewer for DOS?" (number " " mark " John Boyer Mar 21 16/460 " thread-indent "\"Re: Is there a FITS image viewer for DOS?\"\n") "<2mcmfc$g3d at usenet.INS.CWRU.Edu>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA24594; Mon, 21 Mar 94 04:31:49 EST Return-Path: Message-Id: Organization: British Broadcasting Corporation, UK Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!caen!sdd.hp.com!cs.utexas.edu!howland.reston.ans.net!pipex!bbc!ant!boyer References: <2mcmfc$g3d at usenet.INS.CWRU.Edu> From: boyer at rd.eng.bbc.co.uk (John Boyer) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Is there a FITS image viewer for DOS? Date: Mon, 21 Mar 1994 09:12:35 GMT Tom Lipovits (ac448 at cleveland.Freenet.Edu) wrote: : I am recently interested in the FITS images and am unable to find : any image viewers on local bulletin boards. Any information would : be helpful. Thank you : : _I_don't_have_an_employer_ : .Tom. Yes. it's called imdisp79. i think I got it from ames.arc.nasa.gov. You could also try explorer.arc.nasa.gov It will not do 32 bit floating point numbers. John B John.boyer at rd.eng.bbc.co.uk From fitsbits-request Mon Mar 21 09:29:00 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5486" "Mon" "21" "March" "1994" "09:28:52" "EST" "Arnold Rots" "arots at xebec.gsfc.nasa.gov" "<9403211428.AA12356 at xebec.gsfc.nasa.gov>" "112" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032114:28:52" "Unsigned Integers Proposal" (number " " mark " Arnold Rots Mar 21 112/5486 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA25770; Mon, 21 Mar 94 09:29:00 EST Return-Path: Message-Id: <9403211428.AA12356 at xebec.gsfc.nasa.gov> From: arots at xebec.gsfc.nasa.gov (Arnold Rots) Sender: fitsbits-request at fits.CV.NRAO.EDU To: mcs at goblin.caltech.edu Cc: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: Mon, 21 Mar 94 09:28:52 EST Martin Shepherd posted a detailed response to my last message. I won't replay to every point, but there are some issues that require clarification. My basic point was (and is) that the need for unsigned integer types is fairly general and will be increasing in the future. It's going to be easier to introduce them now than n years from today. The XTE project is not the reason for our wanting to have unsigned integers added to the list: I think FITS needs them and high-lighting their usefulness in the context of the XTE software was intended to illustrate how and why these data types are important and will continue to be so in other projects. The other side of the coins, though, is that I have no great desire to jump through all kinds of hoops now, if unsigned integers are going to be included in FITS anyway. > : I > :have no intention to pick on anyone in particular, but I understand > :very well where Don Wells is coming from: in radio astronomy floats, > :doubles, complex, signed integers and scaled integers have > :traditionally dominated the zoo of data types. > This is unreasonable. It seems far more likely that the chosen types > were based on those then available in FORTRAN-77 rather than a > conspiracy among radio astronomers. While I personally prefer C, I can > see the point of not supporting types that at least one language can't > handle. Admittedly, from this viewpoint, COMPLEX types should not have > been included, but it is undeniably too late to change that now. Sorry if this offended you. However, I was a lot closer to the birth of FITS than you were. And I wasn't saying it was a conspiracy; I said it was designed (with an honest effort to generalize) from a specific set of needs. That set did not include unsigned integers, but it did include complex floats (case in point). > :Again, if we *can* store original values in FITS tables, I feel it > :*should* be done. > In an ideal world perhaps, but in the real world it is pointless > because many architectures don't provide types with the same bit > representation as FITS decrees, and are thus unable to recover the > exact value that was written. Sorry, I wrote *values*, not *bit patterns*. > : Every extra line of code increases complexity and > :chances for errors; this is of specific concern to those among us who > :work on software that will produce basic archives: the number of > :operations should be kept to an absolute minimum, on the one hand, > :while the archived product should provide as good data access as > :possible. Unsigned integers would help tremendously in that regard. > Again the same argument can be presented from the opposite viewpoint. > Adding new types increases the size and complexity of general FITS > readers. You are requiring that FITS readers be slowed down so > that your simpler, type-specific FITS writer can run faster. To make > this point clearer, consider that in a general FITS reader every new > type must be accompanied by at least three specialized sets of > functions: > 1. Machine-specific routines to convert between each FITS type and > equivalent host-architecture data-types. This makes it hard to > port to new architectures - more so if there are a lot of types. > 2. Conversion functions between all other compatible types supported > by the FITS library (eg. int to unsigned int, int to float...). > For N types there are O(NxN) such conversions required. This has a > significant impact on speed and complexity. > 3. Functions to test for and set type-specific blank/null values. > Obviously the more functions there are and the greater the number of > possible routes through a FITS reader, the more difficult it becomes to > write a general FITS library, the harder it is to test the result and > the slower it becomes to read FITS. This is the impact on the rest of > the community that I was referring to above, in turning your accusation > of parochial thinking back on you. Bill Pence addressed these issues in an earlier post and concluded they weren't as significant as you make them appear. > And should the rest of the community have to make their software more > complex so that your's need not be? As I said in the introduction, I was not trying to argue that the whole community should change its standards because of XTE, only that many projects (XTE and the guys at FNAL are a case in point) would benefit tremendously from this proposal. > :[Text omitted] > :that capability; strict encapsulation and design simplicity have made > :this piece of software highly reliable - I would like to keep it that > :way. > I feel the same way about my general FITS library, and since it is > much easier to write a specific form of FITS than to read the general > case, shouldn't the burden be taken by the former? After all, during > data processing, FITS files often have to be read multiple times, so > speed and correctness during reading is very important to the > end-user. This raises an interesting question on one's fundamental design approach: should basic utility libraries (like FITS readers) be kept simple so that they are reliable, but requiring the application software that uses them to do the work and be more complex; or should those libraries be designed in such a way that the user-software can be simpler? Whatever case can be made against the introduction of unsigned integers, I don't think it could be supported by this argument. - Arnold Rots From fitsbits-request Mon Mar 21 12:22:09 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1234" "" "21" "March" "1994" "16:48:57" "GMT" "Tom Nicinski" "nicinski at fncasefnal.fnal.gov" "<2mkj5p$nno at fnnews.fnal.gov>" "25" "Re: Unsigned Integers Proposal" "^From:" nil nil "3" "1994032116:48:57" "Unsigned Integers Proposal" (number " " mark " Tom Nicinski Mar 21 25/1234 " thread-indent "\"Re: Unsigned Integers Proposal\"\n") "<9403211428.AA12356 at xebec.gsfc.nasa.gov>"] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA26131; Mon, 21 Mar 94 12:22:09 EST Return-Path: Message-Id: <2mkj5p$nno at fnnews.fnal.gov> Organization: FERMILAB, Batavia, IL Path: saips.cv.nrao.edu!hearst.acc.Virginia.EDU!concert!news.duke.edu!MathWorks.Com!europa.eng.gtefsd.com!howland.reston.ans.net!news.moneng.mei.com!uwm.edu!fnnews.fnal.gov!fncase!nicinski References: <9403211428.AA12356 at xebec.gsfc.nasa.gov> Reply-To: nicinski at fnal.fnal.gov From: nicinski at fncasefnal.fnal.gov (Tom Nicinski) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: Re: Unsigned Integers Proposal Date: 21 Mar 1994 16:48:57 GMT arots at xebec.gsfc.nasa.gov (Arnold Rots) writes: [deleted ...] |> My basic point was (and is) that the need for unsigned integer types |> is fairly general and will be increasing in the future. It's going to |> be easier to introduce them now than n years from today. I'd also like to see unsigned integers introduced soon as part of the BINTABLE extension, before it changes from a draft to a standard. Letting this issue sit and then providing another "standard" extension that is the BINTABLE extension plus unsigned integers (and signed 8-bit integers and possibly 64-bit integers (signed and unsigned)) would be a mistake. Talk about making FITS more complex: you'd have to worry about maintaining two extensions in synch with each other. Tom Nicinski +--- . .' ----+--------------------------------+----------------------------+ | /| | | | | Tom Nicinski / MS 234 | nicinski at fnal.fnal.gov | | +-+-+\ | | Fermi National Accelerator Lab | _ | | . | | | \| | P.O. Box 500 | [o] | | `--' | | | | Batavia, IL 60510-0500 | /|\ | +---- -' -----+--------------------------------+----------------------------+ From fitsbits-request Mon Mar 21 12:59:11 1994 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2905" "Mon" "21" "March" "1994" "18:56:25" "+0100" "Thierry Forveille" "forveill at gag.observ-gr.fr" "<9403211758.AA26156 at fits.cv.nrao.edu>" "53" "RE: Unsigned integer proposal" "^From:" nil nil "3" "1994032117:56:25" "Unsigned integer proposal" (number " " mark " Thierry Forveille Mar 21 53/2905 " thread-indent "\"RE: Unsigned integer proposal\"\n") nil] nil) Received: by fits.cv.nrao.edu (4.1/DDN-DLB/1.5) id AA26162; Mon, 21 Mar 94 12:59:11 EST Return-Path: Message-Id: <9403211758.AA26156 at fits.cv.nrao.edu> Posted-Date: Mon, 21 Mar 1994 18:56:25 +0100 From: forveill at gag.observ-gr.fr (Thierry Forveille) Sender: fitsbits-request at fits.CV.NRAO.EDU To: fitsbits at fits.CV.NRAO.EDU Subject: RE: Unsigned integer proposal Date: Mon, 21 Mar 1994 18:56:25 +0100 ------- Start of forwarded message ------- Posted-Date: Mon, 21 Mar 1994 18:32:42 +0100 In-Reply-To: <9403211428.AA12356 at xebec.gsfc.nasa.gov> References: <9403211428.AA12356 at xebec.gsfc.nasa.gov> From: forveill at gag.observ-gr.fr (Thierry Forveille) To: arots at xebec.gsfc.nasa.gov (Arnold Rots) Subject: Re: Unsigned Integers Proposal Date: Mon, 21 Mar 1994 18:32:42 +0100 Arnold Rots writes: > > My basic point was (and is) that the need for unsigned integer types > is fairly general and will be increasing in the future. It's going to > be easier to introduce them now than n years from today. The XTE > project is not the reason for our wanting to have unsigned integers > added to the list: I think FITS needs them and high-lighting their > usefulness in the context of the XTE software was intended to > illustrate how and why these data types are important and will > continue to be so in other projects. The other side of the coins, > though, is that I have no great desire to jump through all kinds of > hoops now, if unsigned integers are going to be included in FITS > anyway. > I happen to hold just the opposite opinion: the (perceived) need for unsigned integers is highly dated to a few years around 1993 and will fade away as it has arrived. The difference between signed and unsigned integers only matters when you don't fit on 15 bits, so nobody ever cared about FITS integers being signed before 16 bits ADC became common-place, and nobody will ever care again once 18 or 20 bits ADC will be the norm (though I could imagine some support gathered for 24 bit integers at that time...). Given typical evolution times in the electronic industry, my prediction is that this is going to happen before long, though the convenience of 16 bits as the half word of most computers may delay that evolution by a year or two. One can already see that need appearing in CCD manuals that state that the device is linear to better than 1% over the full range of the ADC: that just means the ADC cannot use the full range of the CCD and still sample the noise correctly. What you are buying with unsigned integers is just one bit (which you could also get more cheaply through a BZERO) and this isn't going to last you very long. A new data type in FITS should bring more than that (I for one would probably buy 64 bit integers). Thierry Forveille Observatoire de Grenoble PS: Since this is apparently becoming an issue in the discussion, I should probably state that I am a radioastronomer by training (is that a liability?), but now working on DENIS, a near IR survey of the sky which will produce about as many terabytes as the Sloan project