From stooke at vaxr.sscl.uwo.ca Wed Apr 1 10:53:50 1992 X-VM-VHeader: ("From:" "Sender:" "Resent-From" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:" "Resent-Date:") nil X-VM-Bookmark: 18 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["441" "" "1" "April" "92" "13:31:45" "GMT" "stooke at vaxr.sscl.uwo.ca" "stooke at vaxr.sscl.uwo.ca" nil "14" "Re: HST Io images & Scowen post" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: Social Science Computing Laboratory From: stooke at vaxr.sscl.uwo.ca Subject: Re: HST Io images & Scowen post Date: 1 Apr 92 13:31:45 GMT Just to clear up any possible confusion, the HST images described by Paul Scowen in his recent response to my original post - GIF images of Jupiter, Saturn and other objects, - these GIF images are NOT the same ones I was referring to in my post regarding images of Io. The Io images are from the Faint Object Camera, not WF/PC, and are at: ns3.hq.eso.org in directory: /pub/testimages/testimages-HST-restoration Phil 'sigless' Stooke From hanisch at stsci.edu Thu Apr 2 13:30:11 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2316" "" "2" "April" "92" "15:56:39" "GMT" "Bob Hanisch" "hanisch at stsci.edu " nil "38" "HST data in FITS format" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Keywords: HST, FITS Organization: Space Telescope Science Institute From: hanisch at stsci.edu (Bob Hanisch) Subject: HST data in FITS format Date: 2 Apr 92 15:56:39 GMT I just want to try to clarify a few points about HST data and FITS format. HST data is intended to fully conform to the FITS standard, and we have taken some pains to make the data fit into the simpler FITS structures. We use a variant of the FITS `random groups' format for our internal data structures, but map our data to standard FITS images for export. We do utilize the IEEE floating point format for the sake of efficiency and accuracy. Scaling the data to 8 bits can be dangerous, as data drop outs and other bad pixels (such as strong cosmic ray hits) can affect the scaling profoundly, putting all of the interesting data values into just a few scaled values. Concerning the WF/PC data formats: the data consists of four independent arrays of 800x800 pixels, not a single 1600x1600 array. In fact, the four subarrays cannot be simply patched together to form a contiguous 1600x1600 pixel image. There is some overlap of the images, and the four subimages are not perfectly aligned. To construct a single image you would have to resample 3 of the 4 frames. Observers may wish to do this for display purposes, but we do not advocate this or force it upon observers by providing a combined image. The four 800x800 images are provided to observers in one of two ways: (1) as a standard FITS image with three data dimensions, 800x800x4, or (2) as four independent FITS images, each simply 800x800. This is an option that users specify to us when they request data. Option (1) is preferred: you should bear in mind that a WFPC observation actually consists of a number of images (the actual data, both raw and calibrated, data quality files, etc.) and if all of these are broken up into four independent pieces it becomes very difficult to track what is what. However, the choice is up to the observer. The STSDAS FITS reading software will combine the data back together into our standard internal format. If anyone finds problems with HST FITS format data, I would appreciate hearing about it. -- Robert J. Hanisch Tel. (410)338-4910 Fax. (410)338-5090 Space Telescope Science Institute NSI: hanisch at stsci.edu [130.167.1.2] 3700 San Martin Drive DECNet: STSCIC::HANISCH [6549::] Baltimore, MD 21218 USA Bitnet: hanisch at stsci From pence at heawk1.gsfc.nasa.gov Thu Apr 2 16:26:16 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["923" "Thu" "2" "April" "1992" "19:30:30" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "16" "Blanks in ASCII columns" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Blanks in ASCII columns Date: Thu, 2 Apr 1992 19:30:30 GMT How should a FITS reader interpret trailing blanks in a numeric ASCII table field? For instance, if the TFORM keyword for the field is 'I4' and the 4 characters in the field are '12 ' (that is, the characters 1 and 2 followed by 2 blanks) then should this be read as the integer 12 or the integer 1200? This is analogous to the BN or BZ format options in a Fortran read, which either cause blanks to be ignored or to be interpreted as zeros. I looked in the NOST Implementation of FITS document but it provided no guidance on this question. It turns out that different compilers don't all have the same default interpretation, so I will need to force the FITSIO subroutines to be consistent, one way or the other. My own preference is to treat trailing blanks in the field as zeros (e.g., would read 1200 in the above example). Any other opinions? -Bill Pence, HEASARC pence at tetra.gsfc.nasa.gov or HEASRC::PENCE From cflatter at nrao.edu Thu Apr 2 16:26:25 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2279" "Thu" "2" "April" "1992" "20:48:18" "GMT" "Chris Flatters" "cflatter at nrao.edu " nil "40" "Re: Blanks in ASCII columns" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters) Subject: Re: Blanks in ASCII columns Date: Thu, 2 Apr 1992 20:48:18 GMT In article 702243030 at heawk1, pence at heawk1.gsfc.nasa.gov ( William Pence) writes: >How should a FITS reader interpret trailing blanks in a numeric ASCII >table field? For instance, if the TFORM keyword for the field is 'I4' >and the 4 characters in the field are '12 ' (that is, the characters >1 and 2 followed by 2 blanks) then should this be read as the integer 12 >or the integer 1200? This is analogous to the BN or BZ format options >in a Fortran read, which either cause blanks to be ignored or to be >interpreted as zeros. I looked in the NOST Implementation of FITS document >but it provided no guidance on this question. It turns out that different >compilers don't all have the same default interpretation, so I will need >to force the FITSIO subroutines to be consistent, one way or the other. >My own preference is to treat trailing blanks in the field as zeros >(e.g., would read 1200 in the above example). Any other opinions? I do not have the FORTRAN 77 standard handy but in Fortran 90 non-leading blank characters in numeric fields are treated according to the value of the BLANK= Specifier in the OPEN statement. The chapter and verse for the BLANK= specifier is 9.3.4.6 BLANK= Specifier in the OPEN Statement. The scalar-default-char-expr must evaluate to NULL or ZERO. The BLANK= specifier is permitted only for a file being connected for formatted input/output. If NULL is specified, all blank characters in numeric formatted input fields on the specified unit are ignored, except that a field of all blanks has a value of zero. If ZERO is specified, all blanks other than leading blanks are treated as zeros. If this specifier is omitted, the default value is NULL. In Fortran 90, the default behaviour is to read '12 ' as 12 using an I4 descriptor. I believe that the same is true of FORTRAN 77 (& the only text book I have handy concurs). I think that the NOST document should be revised to make the interpretations of the edit descriptors used to describe ASCII tables much more precise and to remove the necessity to make paper subroutine calls to the FORTRAN standard. As written the standard for ASCII tables makes life very difficult for people working in languages other than FORTRAN. Chris Flatters cflatter at nrao.edu From bschlesinger at nssdca.gsfc.nasa.gov Thu Apr 2 20:51:37 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1411" "Thu" "2" "April" "1992" "22:25:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdca.gsfc.nasa.gov " nil "31" "Re: Blanks in ASCII columns" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.41 Nntp-Posting-Host: nssdca.gsfc.nasa.gov Organization: NASA - Goddard Space Flight Center From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Subject: Re: Blanks in ASCII columns Date: Thu, 2 Apr 1992 22:25:00 GMT In article , pence at heawk1.gsfc.nasa.gov ( William Pence) writes... > How should a FITS reader interpret trailing blanks in a numeric ASCII > table field? For instance, if the TFORM keyword for the field is 'I4' > and the 4 characters in the field are '12 ' (that is, the characters > 1 and 2 followed by 2 blanks) then should this be read as the integer 12 > or the integer 1200? The relevant discussions appear in the FITS Tables paper (A&A Suppl. 73,365 for those not familiar with it): "To prevent trailing blanks from defaulting to trailing zeroes, the decoding of table extension data should be done as though the FORTRAN-77 OPEN statement specifier BLANK is set to NULL (i. e., blank characters should be ignored)." and in the NOST Standard, the last sentence of 8.1.5: "Blanks within the fields are not to be interpreted as zeroes; zeroes must be given explicitly." By these two prescriptions, the field would be read as 12. The ASCII Tables paper urges that "integers and floating point numbers should be right justified to prevent the problem of how trailing blanks are treated in different computers." So, FITS writers should not create fields like the one described above. Unfortunately, FITS readers have to be prepared to deal with issues like this that can arise from deficiencies in design elsewhere. Barry Schlesinger FITS Support Office From pence at heawk1.gsfc.nasa.gov Fri Apr 3 16:28:31 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["633" "" "3" "April" "92" "19:26:21" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "19" "Re: Blanks in ASCII columns" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: Goddard Space Flight Center Nntp-Posting-Host: heawk1.gsfc.nasa.gov From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Re: Blanks in ASCII columns Date: 3 Apr 92 19:26:21 GMT Barry Schlesinger wrote: >The relevant discussions appear in the FITS Tables paper (A&A Suppl. >73,365 for those not familiar with it): > >"To prevent trailing blanks from defaulting to trailing zeroes, the >decoding of table extension data should be done as though the >FORTRAN-77 OPEN statement specifier BLANK is set to NULL (i. e., blank >characters should be ignored)." > >and in the NOST Standard, the last sentence of 8.1.5: > >"Blanks within the fields are not to be interpreted as zeroes; zeroes >must be given explicitly." Thanks, Barry. That is pretty unambiguous, so I'll modify FITSIO accordingly. -Bill Pence From herzog at ux5.lbl.gov Sat Apr 4 08:38:56 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["503" "Fri" "3" "April" "1992" "23:44:45" "GMT" "Hanan Herzog" "herzog at ux5.lbl.gov " nil "20" "Image processing and conversion software" "^From:" nil nil "4"]) Newsgroups: alt.sci.astro.fits,sci.astro.fits Followup-To: alt.sci.astro.fits Keywords: fits, gif, ibmpc, unix Nntp-Posting-Host: ux5.lbl.gov Organization: Lawrence Berkeley Laboratory Distribution: na From: herzog at ux5.lbl.gov (Hanan Herzog) Subject: Image processing and conversion software Date: Fri, 3 Apr 1992 23:44:45 GMT Hello all. I am trying to locate software that will convert GIF <-> FITS formats. Source or binary, IBMPC (DOS) or UNIX. I would be interested in all and any FTP site addresses that carry image processing of this sort. Also, if somesone knows the address of some FTP sites that carry NIH image processing software, I would appreciate it greatly if you included them in your reply. Thanks in advance. Feel free to reply by email OR post. Hanan M. Herzog herzog at lbl.gov -- email: herzog at lbl.gov From thompson at stars.gsfc.nasa.gov Sun Apr 5 13:50:31 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1314" "" "4" "April" "92" "22:56:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "30" "Compression" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Compression Date: 4 Apr 92 22:56:00 GMT Has anyone come to any conclusions about the FITS compression proposal by Archie Warnock et al.? I'm interested in the subject of data compression in FITS files. Warnock et al. propose to compress whole FITS header and data units. The advantage of this approach is that any FITS data structure can be compressed, including ASCII and binary tables. However, as I see it, this scheme also has the following consequences: * Only lossless compression algorithms can be used, so that the compressed header can be reconstructed correctly. * Information about the data in the compressed header should also be reproduced in the compression extension header, so that the user can understand what the file contains. Since the same data is included twice, some loss of compression efficiency is engendered. * Compression schemes that work well on the data, may be inefficient in reducing the extension header. Thus, I have the following questions for the FITS community: * Is there currently support within the FITS community for adopting the Warnock et al. proposal as a standard? * Could the Warnock et al. proposal be changed to *not* compress the header by making the optional keywords LBITPIX, etc. mandatory instead, or would this make the proposal less flexible? Bill Thompson From warnock at nssdca.gsfc.nasa.gov Mon Apr 6 17:14:36 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3054" "" "6" "April" "92" "15:13:00" "GMT" "Archie Warnock" "warnock at nssdca.gsfc.nasa.gov " nil "61" "Re: Compression" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: Hughes STX - NASA/NCDS News-Software: VAX/VMS VNEWS 1.41 Nntp-Posting-Host: nssdca.gsfc.nasa.gov From: warnock at nssdca.gsfc.nasa.gov (Archie Warnock) Subject: Re: Compression Date: 6 Apr 92 15:13:00 GMT In article <4APR199218564977 at stars.gsfc.nasa.gov>, thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes... >Has anyone come to any conclusions about the FITS compression proposal by >Archie Warnock et al.? I'm interested in the subject of data compression in Well... I have . >including ASCII and binary tables. However, as I see it, this scheme also has >the following consequences: > * Only lossless compression algorithms can be used, so that the > compressed header can be reconstructed correctly. Yes - this is intentional. Remember that FITS is an interchange format, and as such, should represent the data stream accurately. You want to do lossy compression? Do it before putting the data in FITS... That's a partially facetious answer, and I'd welcome solutions which (a) retain the flexibility and (b) add the ability to use lossy compression. Part of the reason for releasing the proposal publicly was to generate such discussion _before_ the proposal became carved in stone (or at least carved in A&A Supp). > * Information about the data in the compressed header should also be > reproduced in the compression extension header, so that the user can > understand what the file contains. Since the same data is included > twice, some loss of compression efficiency is engendered. This redundancy is not mandatory, so if you're really concerned with efficiency, you can omit the "logical" keywords in the extension header. But see below the next point... > * Compression schemes that work well on the data, may be inefficient in > reducing the extension header. Yes, but we're only talking (typically) about a few thousand bytes of header. If that's a substantial fraction of the data volume, the compression may be inappropropriate. > * Could the Warnock et al. proposal be changed to *not* compress the > header by making the optional keywords LBITPIX, etc. mandatory > instead, or would this make the proposal less flexible? Of course it _could_ be changed. That's why we presented it at the AAS meeting in Albuquerque. An early iteration of the proposal did just such a thing, but we realized that, since there's no way to "nest" extensions, this would effectively exclude compressing any extensions - and tabular data tends to be highly compressible. You'd need to come up with some way to add keywords to document the presence of a compressed extension, and now you're talking about a (possibly) long and complicated set. There has been talk of some heirarchical schemes for extensions, and that may resolve some of these problems, but my sense is that those proposals aren't even as mature as the compression proposal - and it's still an infant, too. More discussion, more ideas... ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at nssdc.gsfc.nasa.gov -- Hughes STX SPAN: NSSDC::WARNOCK -- NASA/GSFC "Hail To The Redskins!" From gei at cfa.harvard.edu Fri Apr 10 19:56:28 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["185" "" "10" "April" "92" "14:31:53" "GMT" "Guenther Eichhorn" "gei at cfa.harvard.edu " nil "5" "FITS to GIF" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: Smithsonian Astrophysical Observatory, Cambridge, MA, USA From: gei at cfa.harvard.edu (Guenther Eichhorn) Subject: FITS to GIF Date: 10 Apr 92 14:31:53 GMT I am looking for a program that makes a GIF file from a FITS image for a Sun workstation. Has anybody heard of such a program? Please post or send mail to gei at cfa.harvard.edu Thanks From mehr at oskar.uchicago.edu Mon Apr 13 06:54:39 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["790" "" "13" "April" "92" "01:59:39" "GMT" "David Mehringer" "mehr at oskar.uchicago.edu " nil "20" "Re: FITS to GIF" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: Department of Astronomy and Astrophysics From: mehr at oskar.uchicago.edu (David Mehringer) Subject: Re: FITS to GIF Date: 13 Apr 92 01:59:39 GMT In article <1992Apr10.143153.3978 at cfa.harvard.edu> gei at cfa.harvard.edu (Guenther Eichhorn) writes: > >I am looking for a program that makes a GIF file from a FITS image for a Sun workstation. >Has anybody heard of such a program? Please post or send mail to gei at cfa.harvard.edu > >Thanks One way to do this is to use the snapshot utility to make a raster file of your image (assuming you have an AIPS version that can run on Xwindows). After snapping and saving the image, run xv and load the raster file. Save the xv-loaded image as a GIF file (or whatever format you want). -- Dave Mehringer | "... But the most amazing thing to me is, dmehring at zia.aoc.nrao.edu | I/get paid National Radio Astronomy | for doin' Observatory, Socorro, NM | this." -- Steve Martin From massengi at cloves.unx.sas.com Mon Apr 13 14:43:16 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["880" "Mon" "13" "April" "1992" "12:28:25" "GMT" "Darrell Massengill" "massengi at cloves.unx.sas.com " nil "21" "Re: FITS to GIF" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: cloves.unx.sas.com Organization: SAS Institute Inc. X-Bytes: 880 From: massengi at cloves.unx.sas.com (Darrell Massengill) Subject: Re: FITS to GIF Date: Mon, 13 Apr 1992 12:28:25 GMT In article <1992Apr10.143153.3978 at cfa.harvard.edu> gei at cfa.harvard.edu (Guenther Eichhorn) writes: > >I am looking for a program that makes a GIF file from a FITS image for a Sun workstation. >Has anybody heard of such a program? Please post or send mail to gei at cfa.harvard.edu > >Thanks The PBMPLUS toolkit can do that. It is public domain. Below is the information from the comp.graphics FAQ: PBMPLUS, by Jef Poskanzer. Comprehensive format conversion and image manipulation package. The latest version is always available via anonymous FTP as ftp.ee.lbl.gov:pbmplus*.tar.Z, wuarchive.wustl.edu:graphics/graphics/packages/pbmplus/pbmplus*.tar.Z, and export.lcs.mit.edu:contrib/pbmplus*.tar.Z. --- -- Darrell Massengill Manager of Image Processing SAS Institute Inc. massengi at unx.sas.com (919) 677-8000 x7658 SAS Campus Dr, Cary, NC From bschlesinger at nssdca.gsfc.nasa.gov Sat Apr 18 07:08:44 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5994" "Fri" "17" "April" "1992" "20:34:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdca.gsfc.nasa.gov " nil "123" "FITS Basics and Information" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.41 Keywords: FITS references, FITS description Nntp-Posting-Host: nssdca.gsfc.nasa.gov Organization: NASA - Goddard Space Flight Center From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Subject: FITS Basics and Information Date: Fri, 17 Apr 1992 20:34:00 GMT This basic FITS information will be posted and updated periodically for the benefit of new readers and the reference of old readers. FITS (Flexible Image Transport System) is a data format designed to allow convenient exchange of astronomical data between installations whose standard internal formats and hardware may be different. A FITS file consists of a sequence of Header Data Units (HDUs). Each is composed of a header made up of keyword=value type statements describing the format and organization of data that follow and the data so described. The data section of the HDU may contain a digital image, BUT IT DOESN'T HAVE TO. Other possible formats include tables and multidimensional matrices that are not images. The first HDU must contain a multidimensional matrix or no data at all; the data in subsequent HDUs, called extensions, may be of any type, consistent withe certain rules. The "Image" in the name comes from the original use of the format to transport digital images, but it's not just for images any more. FITS is not principally a graphics format designed for the transfer of pictures with standard support for many platforms included as part of the package for decoding the image. Separate software must be developed or obtained for converting the data from the FITS file into a form that can be readily displayed. As has been discussed in this newsgroup, and in alt.sci.astro.fits before it, the Extended Portable Bitmap Toolkit (pbm+) can be used for converting many FITS files to such a format, but support for all FITS files where the data are in the form of an image, in particular those where the data matrix members are in IEEE floating point format (BITPIX<0) or the matrix has more than two dimensions (NAXIS>2), is not guaranteed. Discussion of FITS - image converters appears in this newsgroup from time to time. The fundamental references on FITS are the following four papers, often referred to collectively as the "Four FITS Papers". These papers are the formal standard for FITS, endorsed by the International Astronomical Union (IAU). Wells, D. C., Greisen, E. W., and Harten, R. H., "FITS: a flexible image transport system," Astronomy and Astrophysics Supplement Series, 44, 363-370, 1981. Greisen, E. W. and Harten, R. H., "An extension of FITS for small arrays of data," Astronomy and Astrophysics Supplement Series, 44, 371-374, 1981. (NOTE: The format described in this paper has been used almost exclusively to transport radio interferometry and is likely to be replaced by other formats in the future. Writing data other than radio interferometry data using this format is not recommended.) Grosbol, P., Harten, R. H., Greisen, E. W., and Wells, D. C., "Generalized extensions and blocking factors for FITS," Astronomy and Astrophysics Supplement Series, 73, 359-364, 1988. Harten, R. H., Grosbol. P., Greisen, E. W., and Wells, D. C., "The FITS tables extension, Astronomy and Astrophysics Supplement Series, 73, 365-372, 1988. A User's Guide for FITS, commissioned by NASA Headquarters, is maintained by the NASA/OSSA Office of Standards and Technology (NOST). This Guide is intended to be a tutorial for new FITS users. In addition to presenting the rules of FITS, it provides some of the history and reasoning behind the choice of the rules, adds recommendations on good practices, and discusses current developments in FITS. This document is available only in hard copy form. NASA is sponsoring development of a formal standard for FITS. The goal of this process is a document consistent with FITS as endorsed by the IAU, eliminating some contradictions and ambiguities in the original FITS papers, that can be endorsed by the IAU FITS Working Group as the FITS standard. The document is being developed by a Technical Panel chaired by Dr. Robert J. Hanisch (STSci), with review by the astronomical community. Only minor revisions are expected to the current draft, version 0.3b, but the form of the standard is not final, and it does not supersede the four papers and Floating Point Agreement endorsed by the IAU as the official standard for FITS. The IAU has endorsed the Floating Point Agreement, which defines how floating point numbers are to be stored in the data matrix defined in the first of the four papers. The basic agreement appears verbatim in the User's Guide, and the substance is incorporated in the Draft NASA Standard. The NOST maintains a file of FITS information available by anonymous ftp from nssdca.gsfc.nasa.gov or DECnet copy from NSSDCA, in the directory FITS. It includes copies of the current draft NASA Standard in flat ASCII, PostScript, and LaTeX, and a list of the current extension type (structure) names registered with the IAU FITS Working Group. It also contains, in LaTeX form, the text of the proposal for one of these new extension types, IMAGE. A README. file describes the contents of the directory. There is a further subdirectory, SOFTWARE, described by README.FIRST, containing a program in C to read and list the headers of a FITS file and another file with information on publicly available FITS software packages. Paper copies of many of the documents listed above can be obtained from the NOST Librarian. Paper copies of the User's Guide and either paper or electronic copies of the Draft Implementation Standard, for those without ftp access, are available. Because of restrictions set by the copyright holder, NOST can send copies of the four FITS papers only to non-profit organizations. The addresses of the NOST are as follows: (Postal) NASA/OSSA Office of Standards and Technology Code 933 Goddard Space Flight Center Greenbelt MD 20771 USA (Internet) nost at nssdca.gsfc.nasa.gov (DECnet) NCF::NOST Please mention this posting in your request. Barry M. Schlesinger Coordinator, NASA/NSSDC NOST FITS Support Office (301) 513-1634 fits at nssdca.gsfc.nasa.gov NCF::FITS From jim at galileo.ifa.hawaii.edu Tue Apr 21 07:00:15 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["866" "Tue" "21" "April" "1992" "08:56:31" "GMT" "Bubba Mersault" "jim at galileo.ifa.hawaii.edu " nil "20" "FITS File Compression / CCD Image Compression" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits,alt.sci.astro.fits Summary: Request for info on image compression software Keywords: FITS,compression,CCD Nntp-Posting-Host: galileo.ifa.hawaii.edu Organization: Institute for Astronomy, Hawaii From: jim at galileo.ifa.hawaii.edu (Bubba Mersault) Subject: FITS File Compression / CCD Image Compression Date: Tue, 21 Apr 1992 08:56:31 GMT I am searching for suggestions on FITS file compression. I have experimented with the standard UNIX compress utility and found it not to be very effective. I have also used W. Press's FITSPRESS program, which is effective on some images but slow (19 minutes to compress a 2048x2048 image on a SPARC Server) for my purposes. It is, however, ineffective on on some images I have with only a couple of bright stars in the field. My needs dictate that a lossless compression system be used. Also, I am dealing with large format CCDs - 2048x2048 now, and larger ones in the near future. If anyone could suggest other software, or even a hardware solution, that I could evaluate, I would appreciate it. Kevin T.C. Jim Voice: (808) 956-9867 Institute for Astronomy FAX: (808) 988-2790 2680 Woodlawn Drive Email: jim at galileo.ifa.hawaii.edu From dwells at azalea.cx.nrao.edu Tue Apr 21 09:22:37 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5416" "Tue" "21" "April" "1992" "14:07:26" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "105" "Re: FITS File Compression / CCD Image Compression" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits,alt.sci.astro.fits In-Reply-To: jim at galileo.ifa.hawaii.edu's message of Tue, 21 Apr 1992 08: 56:31 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA. From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: Re: FITS File Compression / CCD Image Compression Date: Tue, 21 Apr 1992 14:07:26 GMT In article <1992Apr21.085631.20342 at news.Hawaii.Edu> jim at galileo.ifa.hawaii.edu (Bubba Mersault) writes: BM> Summary: Request for info on image compression software BM> Keywords: FITS,compression,CCD BM> "I am searching for suggestions on FITS file compression.. [have] BM> used W. Press's FITSPRESS program, which is effective on some BM> images but slow.. [need] a lossless compression system.. CCDs.. BM> 2048x2048..." This subject was discussed in the alt.sci.astro.fits newsgroup (also "fitsbits" exploder) in September-91 and January-92. The traffic is in the files: fits.cx.nrao.edu[192.33.115.8]:/FITS/fitsbits/fitsbits.{9109,9110,9201} I think that the discussion in the 9109 and 9110 files is quite complete; many, many people gave their opinions and ideas. Take a look at the announcement of program COMPFITS in the 9201 file. -*- You will find my opinions in several of the messages in the archives, but I will take this opportunity to state them once more: \begin{soapbox} Truly random noise is {\em incompressible}. Read that sentence again---it is fundamental! If any portion of your bitstream is random noise, then that portion is incompressible. Suppose you have sky-limited images with 16-bit pixels and sky level around 1000 (out of the 65536 total range) with negligible amplifier noise and dark value. Almost all of your pixels have approximately 10 non-zero bits (1000 ~= 2^10). The Poisson noise is sqrt(value), so the bottom 5 bits of each pixel are random noise. The top six bits are zero, so they will compress with high efficiency. The middle five bits are approximately constant (for a flattened image or a chip of good cosmetic quality) and so they too will compress well. The bottom five bits will not compress. In summary, this trivial back-of-the-envelope calculation tells us that an optimum non-destructive algorithm can reduce such an image from 16 bits/pixel to ~5 bits/pixel. This is a size reduction by a factor of ~3.2. Now you should ask yourself how many of those 5 bits you really need. I contend that for even the most demanding applications (e.g. detecting low surface brightness in the outermost parts of galaxies) you only need ~3 of them. Indeed, ~2 of them might be enough.\footnote{Use Monte Carlo trials of the demanding application(s) to decide the question.} THIS MEANS THAT ANOTHER FACTOR OF {\em TWO} IS AVAILABLE IF YOU ARE WILLING TO MASK EXTRA NOISE BITS! But wait---there's more. Probably the direct imagery is sbomewhat oversampled. By that I mean that probably the camera samples more than 2 pixels per seeing FWHM (this is the recommended practice in imaging). I.e., the power spectrum of the image probably extends beyond the power spectrum of the signal that came through the atmosphere and imaging system. (The cutoff of the signal must be evaluated in those portions of the image which have interesting signal composed of high spatial frequencies which the atmospheric turbulence has limited.) If a Wiener noise-limiting filter is applied to such an image the redundant noise at high spatial frequencies will be eliminated but the valid signal will remain. The effects on the signals produced by the linear filter operator are precisely calculable. Suppose that the effective cutoff of the signal is about 2/3 of the Nyquist frequency. THIS MEANS THAT ANOTHER FACTOR OF 1.5 IS AVAILABLE IF YOU ARE WILLING TO APPLY A LINEAR NOISE-LIMITING FILTER!\footnote{I believe that the reduction is actually 1.5^2 because the filter is applied in the y-direction too, but I will ignore this additional 1.5x for this discussion.} The two techniques I have just advocated are capable of making big gains in compression ratio without destroying the real signals in the data. What does the term "lossless" mean in light of this argument? If you are only a programmer and know nothing about the {\em meaning} of the bits, you will say that the specification "lossless" means that the de-compressed bitstream must be {\em identical} to the original bitstream. But if you are aware that the atmosphere, imaging system and random noise processes have already corrupted the signal in the original bitstream, you can decide that the specification "lossless" will have a different meaning in your compression application. For the example numbers (nbits=16, sky=1000, cutoff=0.67), the basic gain will eliminate about 2/3 of the bits, about 1/2 of the remaining 1/3 of the bits can be removed because they over-sample the noise, and finally about 1/3 of the remaining 1/6 of the bits can be removed because they over-sample the signal. IF ALL THREE TECHNIQUES ARE USED THE IMAGE CAN BE REPRESENTED BY ABOUT 1.7 BITS/PIXEL, OR A COMPRESSION RATIO OF ABOUT 9.5.\footnote{If the extra 1.5x Wiener filter factor is available then the numbers are ~1.1 bits/pixels, compression ~14x.} If compression really matters to you and your users it will be productive to think long and hard about using these techniques. In particular, masking the redundant (Poisson) noise bits is cheap and easy to implement and will make a {\em big} gain in many cases. \end{soapbox} -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From steve at cfht.hawaii.edu Tue Apr 21 16:47:08 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["13756" "Tue" "21" "April" "1992" "20:00:55" "GMT" "Steven S Smith" "steve at cfht.hawaii.edu " nil "410" "Re: FITS File Compression / CCD Image Compression" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits,alt.sci.astro.fits Keywords: FITS,compression,CCD Nntp-Posting-Host: quonset.cfht.hawaii.edu Organization: University of Hawaii From: steve at cfht.hawaii.edu (Steven S Smith) Subject: Re: FITS File Compression / CCD Image Compression Date: Tue, 21 Apr 1992 20:00:55 GMT jim at galileo.ifa.hawaii.edu (Bubba Mersault) writes: >I am searching for suggestions on FITS file compression. I ... FYI, I have enclosed the man page for some code we have here. I am not the author, but in the spirit of aloha... enjoy, steve COMPFITS(1) Public Domain COMPFITS(1) January 1992 NAME compfits - compress/uncompress 16 bit images (lossless or lossy) SYNOPSIS Compression: compfits [ -cinqvwL ] -F origfile [ -f compfile ] [ -P pathext ] [ -p argext ] [ -N nuser ] [ -e epsilon ] [ -m msbbuf ] [ -l lsbbuf ] [ -t tesbuf ] Decompression: compfits -d [ -inqv ] -f compfile [ -F decfile ] [ -P pathext ] [ -p argext ] [ -m msbbuf ] [ -l lsbbuf ] [ -t tesbuf ] DESCRIPTION Compfits is a utility program developed at Canada-France-Hawaii Telescope Corporation for lossless compression of astronomical 16 bits/pixel FITS (Flexible Image Transport System) format images. These images are usually very noisy and don't shrink well with the typical lossless compression algorithms. Options are available to compress non-FITS files and to perform lossy compression. Compfits doesn't compress anything by itself and therefore needs an external compression program to work. Compfits analyzes the bit planes of the original image to see which contain only noise (the low order planes) and which contain the actual information (the high order planes). These two sets of bit planes, or partitions, are respectively called the LSB image (containing the n least significant bits of the original image) and the MSB image (containing the 16-n most significant bits of the original image). The LSB image is supposed to contain only noise, therefore any attempt to compress it would actually increase its size. Compfits writes it as it is in the compressed file, just packing the bits one after the other. The MSB image contains the information and can usually compress fairly well. Compfits sends it to the chosen external compression program which must be a filter (i.e. read stdin and send compressed version to stdout) and writes the compressed MSB image in the compressed file. Of course, the best partition (i.e. the best value of n) depends a good deal on the image and slightly on the external compression program used. Compfits makes a guess for the optimal n trying to estimate the entropy of the different bit planes, unless the user specifies a partition. The compressed file has the following structure: HEADER LSB image Compressed MSB image The header is the FITS header if the original is a FITS format image or a blank 2880 bytes FITS header if it is not. - 1 - Formatted: April 21, 1992 COMPFITS(1) Public Domain COMPFITS(1) January 1992 For the decompression, it is necessary to know which partition was used during the compression. This information is written in the header as a card named COMPART. If the original file is a FITS file the value of the COMPART card is a long integer: n+17*size_diff where size_diff is the difference in bytes between the actual size of the file and the expected size according to the FITS header (size_diff should normally be zero). If the original file is not a FITS file the value of the COMPART card is a long integer: n+17*size_file where size_file is the size in bytes of the original file. Another card named COMPRESS is added to indicate that the file is compressed and to give the name of the compression algorithm. When Compfits is used, the value of the COMPRESS card is set to COMPFITS for a lossless compression and to LOSSFITS for a lossy compression. When it is not a FITS image, the added header contains both the COMPRESS and the COMPART card following a SIMPLE card with the logical value 'F' (false), to denote that although the header may be recognized as FITS, the data does not conform to the FITS standard. Because it needs to first analyse the data before starting the compression process, Compfits needs an input disk file. Because it needs to write the compressed MSB and LSB images at the same time to different locations in the compressed file, Compfits needs an output disk file. In other words, Compfits cannot be used as a filter. The recommended suffix for files compressed with Compfits is .CF. OPTIONS All numbers indicating sizes may be suffixed by b (512 bytes blocks), k (kilobytes), or m (megabytes). -c Compress data. -d Decompress data. -e epsilon Use this value of epsilon instead of the defined one. -f filename Name of the compressed file. For compression, the default is original.CF. For decompression, there is no default; this option must be given. -F filename Name of the original or reconstructed file. For compression, there is no default; this option must be given. For decompression, the default is compressed.REC. -i Ignore garbage at the end of file if any (this may happen if the file is blocked for storage). - 2 - Formatted: April 21, 1992 COMPFITS(1) Public Domain COMPFITS(1) January 1992 -L Lossy compression: the LSB image is not written which allows the program to achieve particularly good compression ratios. Reconstruction is not exact. -l lsbbuf Size of the buffer for the LSB image. Data is written to or read from the disk in blocks of this size. -m msbbuf Size of the buffer for the MSB image. Data is sent to or received from the external compression program in blocks of this size. -N nuser Force the partion value n to be nuser and not the one determined by the program. -n Indicates that the file to be compressed or uncompressed is not a FITS file. -P pathext Path to the external compression or decompression program. Default is /usr/local/bin/compact_1.0 for compression. Default is /usr/local/bin/uncompact_1.0 for decompression. -p argext Arguments of the external compression or decompression program. If multiple arguments with imbedded spaces are present, this must be a quoted string. Default is "-q" (quiet mode) for compact. -q Quiet mode: no compression or decompression statistics will be printed on stderr. -t tesbuf Size of the central test zone for determining the best partition. Default is 512*512*16 bits. -v Verbose: prints some information on the image to stderr. -w Print the entropy of all the bit planes. THEORY OF OPERATION For Compfits, the critical operation is to determine the optimal partition of LSB/MSB (i.e. determine the optimal n). To do so, it tries to compute the entropy of the different bit planes of a subimage. For homogeneous 16 bits/pixel images, experiments show that there is no advantage in sampling a subimage bigger than 512*512. First the image is translated from the binary code into the Gray code to give more coherence to the bit planes. Then for each bit plane i, starting with the zeroth order and increasing, the entropy is computed assuming that the bit plane is a binary discrete memoryless source. This continues until i-entropy(i)>epsilon. Thus the optimal n is chosen such that n-entropy(n)epsilon. This comes simply from the fact that the low order bit planes are supposed to contain almost only noise and therefore entropy(i) is very close to i, whereas high order bit planes mainly contain information - 3 - Formatted: April 21, 1992 COMPFITS(1) Public Domain COMPFITS(1) January 1992 so entropy(i) is much lower than i. With Compfits plus Compact 1.0, the experimental best value of epsilon is 0.01 (used as the default epsilon). Usually the function f(i)=i-entropy(i) is non-decreasing with a sharp transition for i=n, which makes the determination of the optimal partition quite reliable. Of course, considering each bit plane as a binary discrete memoryless source is a bit simplistic. However, as the computation is actually performed only on the low order bit planes (i.e. almost white noise most of the time), this model is quite accurate. PERFORMANCE The compression ratio achieved depends of course on the external compression program. As Compfits assumes a 16 bits/pixel image, the best results are obtained when using an external compression program which assumes a 16 bit word input stream too. An example of such a program is the public domain Compact 1.0, using the Lempel-Ziv algorithm. The default value of epsilon is set for use with this program. Compfits plus Compact 1.0 were tested on a number of 8 megabyte FITS images (2048*2048 pixels, 16 bits/pixels) taken at the Canada-France- Hawaii Telescope. Typically an object image (i.e. an actual sky image rather than a flat field or bias image) is compressed to 4 megabytes whereas usual lossless compression programs such as Unix Compress or Compact 1.0 rarely do better than 6 megabytes. The difference is even more drammatic when the images are very noisy (e.g. flat field images). Such images may not shrink at all with a usual lossless compression program whereas it is reduced to 6 megabytes with Compfits plus Compact 1.0. On these astronomical images, and with the default value of epsilon, the partition found by Compfits is almost always optimal; any other partition leads to worse results. Note that a partition with n=0 (0 LSB) is the same as using the external program alone. The more noise there is in the image, the higher n is and the greater the difference between the use of Compfits and the simple use of the external program is. The visual result of lossy compression is usually quite good as the LSB image which is removed is supposed to contain only noise. Moreover, 256 color workstation displays usually can't show the whole range of intensity which hides the defects. However a lossy compressed astronomical image can't possibly be considered as a scientific frame. WARNING: This progam will work well on any noisy 16 bits/word file, but may yield poor results on any other kind of file. DIAGNOSTIC Compfits prints error messages and warning messages on stderr. It differentiates between "error due to operating system", typically - 4 - Formatted: April 21, 1992 COMPFITS(1) Public Domain COMPFITS(1) January 1992 producing an errno code and "error due to program", (all the others). Compfits exits with a 0 status on normal completion. The exit status is negative on error: -1 for a system error; -2 for a program error. The exit status is positive on warning: 1 for bad arguments; 2 if the file is shorter than expected from examining the header; 3 if the file larger than expected from examining the header; 4 if the image is not 16 bits/pixel; 10 if the image is not a FITS file (unexpected). EXAMPLES compfits -F image will compress the file image, using /usr/local/bin/compact_1.0 as the external compression program and with all the default parameters. The compressed file will be image.CF. compfits -d -f image.CF will do the reverse, uncompressing image.CF using /usr/local/bin/uncompact_1.0 as the external program. The reconstructed file will be image.CF.REC and will be identical to image. compfits -F image -f compimage -P /usr/bin/compress -p -b16 will compress image to compimage, using /usr/bin/compress with options -b16 as the external compression program. All other parameters will be default ones. compfits -d -f compimage -F recimage -P /usr/bin/uncompress -p -b16 will do the reverse. recimage and image will be identical. compfits -F image -N 4 -w (reverse: compfits -d -f image.CF) will compress image to image.CF forcing the partition to be 4 LSB / 12 MSB, and displaying the entropy of the different bit planes. This is useful when testing if the default value of epsilon is optimal. compfits -F image -L (reverse: compfits -d -f image.CF) will compress image to image.CF. The compression will be lossy. Therefore after the decompression, image and image.CF.REC won't be identical. AUTHOR Jean-Pierre Veran Canada-France-Hawaii Telescope Corporation / Ecole Nationale Superieure des Telecommunications de Paris Copyright (C) 1992 Canada-France-Hawaii Telescope Corp. - 5 - Formatted: April 21, 1992 COMPFITS(1) Public Domain COMPFITS(1) January 1992 All the programs in the compfits package are distributed WITHOUT any warranty, and are under the terms of the GNU General Public License. Please, send any question or comment to "compfits at cfht.hawaii.edu". - 6 - Formatted: April 21, 1992 -- Steven S. Smith Canada-France-Hawaii Telescope Corp. steve at cfht.hawaii.edu (NeXT mail grudgingly accepted ;-) From pedelty at jansky.gsfc.nasa.gov Fri Apr 24 14:24:51 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["606" "Fri" "24" "April" "1992" "16:40:33" "GMT" "Jeff Pedelty" "pedelty at jansky.gsfc.nasa.gov " nil "16" "NIH Image and FITS" "^From:" nil nil "4"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: pedelty at jansky.gsfc.nasa.gov (Jeff Pedelty) Subject: NIH Image and FITS Date: Fri, 24 Apr 1992 16:40:33 GMT FITSers, There was some fairly recent discussion on the FITS reading capabilities of NIH Image. The macro supplied with version 1.41 only reads images which are 16 bit integers with NAXIS=2. I don't remember >from the discussion whether anyone has extended this macro to deal with more general images, or whether the consensus was that this would not be too difficult to do. Has anyone done this yet? Jeff Pedelty --- Jeffrey Pedelty | Code 932 | pedelty at jansky.gsfc.nasa.gov NASA / GSFC, Greenbelt, MD 20771 | Phone: 301-286-3065, Fax: x3221 From fitz at noao.edu Mon May 4 09:45:18 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1010" "Sun" "3" "May" "1992" "08:24:43" "GMT" "Mike Fitzpatrick" "fitz at noao.edu " nil "23" "Re: SAOimage and FITS" "^From:" nil nil "5"]) Newsgroups: sci.astro.fits Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: fitz at noao.edu (Mike Fitzpatrick) Subject: Re: SAOimage and FITS Date: Sun, 3 May 1992 08:24:43 GMT Hi Ryan, Basically the is no way to change the command's behavior using other flags/switches/whatevers. A quick tour of the code (I can't actually display an image from my terminal right now) shows that the image scale map is reset for the new image, but I'm not sure whether what you want it to do (or what you are seeing) is to rescale the display based on the new image min and max, or preserve the old transformation. You could try looping over each image as in: % foreach i (*.fts) > saoimage -fits $i:r.fts > end to display the images scaled individually (is this what you want?). If you've changed the scaling by moving the mouse and want to preserve that somehow you might try playing with the [r]min/[r]max switches to define a specific trans- formation for each image. In general the IRAF group is supporting SAOimage so questions may be directed to iraf at noao.edu, although we do monitor this group. I hope this helps. Mike Fitzpatrick NOAO/IRAF Group fitz at noao.edu or 5355::fitz