From hanisch at stsci.edu Thu Jun 4 12:09:37 1992 X-VM-VHeader: ("From:" "Sender:" "Resent-From" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:" "Resent-Date:") nil X-VM-Bookmark: 57 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["893" "Thu" "4" "June" "1992" "14:44:09" "GMT" "Bob Hanisch,N406,4910" "hanisch at stsci.edu " nil "19" "FITS proposals to be discussed at June 8 WGAS" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Reply-To: hanisch at stsci.edu Organization: Space Telescope Science Institute, Baltimore MD From: hanisch at stsci.edu (Bob Hanisch,N406,4910) Subject: FITS proposals to be discussed at June 8 WGAS Date: Thu, 4 Jun 1992 14:44:09 GMT Three pending FITS proposals will be discussed at the June 8 meeting of the AAS Working Group on Astronomical Software in Columbus, OH. These proposals concern blocking of FITS files on fixed-length block media, the binary tables (BINTABLE) extension, and the image extension. The text of these proposals can be retrieved using anonymous FTP to fits.cx.nrao.edu. Look for the following files: /FITS/Documents/FITS_blocking.txt /FITS/Documents/X_bintable.tex, .ps /FITS/Documents/X_image.tex Your comments on these proposals are welcome, either to this newsgroup or at the WGAS meeting. --- Robert J. Hanisch Tel. (410)338-4910 Fax. (410)338-5090 Space Telescope Science Institute NSI: hanisch at stsci.edu [130.167.1.2] 3700 San Martin Drive DECNet: STSCIC::HANISCH [6549::] Baltimore, MD 21218 USA Bitnet: hanisch at stsci From dirk at spacsun.rice.edu Thu Jun 4 14:55:59 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["769" "" "4" "June" "1992" "16:48:50" "GMT" "Dirk Valk" "dirk at spacsun.rice.edu " nil "13" "AAS WGAS meeting" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: Rice University, Houston, Texas From: dirk at spacsun.rice.edu (Dirk Valk) Subject: AAS WGAS meeting Date: 4 Jun 1992 16:48:50 GMT I notice in the advance release of the AAS meeting abstracts that W. Pence will be discussing the 'FTOOLS' package of IRAF software at the WGAS meeting (Sessions 14, talk 1). The abstract mentions that the planned release date will be discussed, as well as the suite of software that will comprise the package. I won't be attending the meeting, but I think it would be useful if anyone who does would post a brief summary to this newsgroup. Volunteers? -- : Dirk Valk |"And if it doesn't work out, : : Rice Univeristy | There'll never be any doubt, : : dirk at spacsun.rice.edu | That the pleasure was worth all the pain." : : Space Physics & Astro.| - Jimmy Buffett : From hanisch at stsci.edu Fri Jun 5 17:03:34 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2712" "Fri" "5" "June" "1992" "19:39:58" "GMT" "Bob Hanisch" "hanisch at stsci.edu " nil "50" "Re: AAS WGAS meeting" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Summary: more info on FTOOLS library Organization: Space Telescope Science Institute From: hanisch at stsci.edu (Bob Hanisch) Subject: Re: AAS WGAS meeting Date: Fri, 5 Jun 1992 19:39:58 GMT In article <1992Jun4.164850.26995 at rice.edu>, dirk at spacsun.rice.edu (Dirk Valk) writes: > I notice in the advance release of the AAS meeting abstracts that W. Pence > will be discussing the 'FTOOLS' package of IRAF software at the WGAS > meeting (Sessions 14, talk 1). The abstract mentions that the planned > release date will be discussed, as well as the suite of software that will > comprise the package. I won't be attending the meeting, but I think it > would be useful if anyone who does would post a brief summary to this > newsgroup. Volunteers? > > -- > : Dirk Valk |"And if it doesn't work out, : > : Rice Univeristy | There'll never be any doubt, : > : dirk at spacsun.rice.edu | That the pleasure was worth all the pain." : > : Space Physics & Astro.| - Jimmy Buffett : In a previous message Dirk Valk asked if someone would post a brief summary of my talk at the AAS meeting on Monday on the FTOOLS FITS file utility package. First, thank you for the free advertisement for my talk. I will be happy to post a summary of the talk here after I get back from the meeting. In the meantime, here is the abstract that I sent to the AAS: \begin{abstract} FTOOLS - A New Package of Programs to Manipulate and Process FITS Format Files The High Energy Astrophysics Science Archive Research Center (HEASARC) is developing a comprehensive set of programs to manipulate and analyze files in FITS (Flexible Image Transport System) format. One consequence of this project is to greatly expand the usage of FITS from simply a transport or interchange format to a convenient and versatile format to be used directly for data reduction and analysis. The FTOOLS utilities are specifically being written to process the data from the Astro-D X-ray satellite, but the tools themselves are very general and can be used to analyze any FITS format file. These utilities are built on top of the FITSIO subroutine library and are written in ANSI standard Fortran or C. The software is easily portable to different processing environments and will be available as an IRAF package as well as a set of stand-alone executable tasks on VMS or Unix systems. The current status of FITSIO and FTOOLS will be described along with plans for future enhancements. \end{abstract} Bill Pence, NASA/GSFC (c/o Bob Hanisch, ST ScI) -- Robert J. Hanisch Tel. (410)338-4910 Fax. (410)338-5090 Space Telescope Science Institute NSI: hanisch at stsci.edu [130.167.1.2] 3700 San Martin Drive DECNet: STSCIC::HANISCH [6549::] Baltimore, MD 21218 USA Bitnet: hanisch at stsci From pgrosbol at eso.org Wed Jun 10 19:54:12 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3205" "Mon" "8" "June" "1992" "11:13:02" "GMT" "pgrosbol at eso.org" "pgrosbol at eso.org" nil "76" "Annual report for IAU FITS Working Group 1991" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: pgrosbol at eso.org Subject: Annual report for IAU FITS Working Group 1991 Date: Mon, 8 Jun 1992 11:13:02 GMT The following report was submitted to BAAS for publication: IAU FITS Working Group Annual Report for 1991 Preben Grosbol European Southern Observatory This report covers main events and activities concerning the FITS standard for the period March 1991 to March 1992. 1. General ---------- 1.1 IAU FITS Working Group -------------------------- The FITS Working Group met during a meeting of Commission 5 at the IAU General Assembly XXI in Buenos Aires, Argentina. Due to the distant location, only a limited number of members participated. Main activities have included the acceptance of the Floating Point Agreement and discussions on new proposals. The Group is chaired by P. Grosbol (ESO) with D. Wells (NRAO) as vice-chairman. 1.2 NASA FITS Support Office ---------------------------- The office was given very substantial help to the FITS community by providing documentation and clarification on FITS issues. Beside its direct support activities, it has initiated the creation of a standard document for the Implementation of FITS (current version NOST 100-0.3b). This document gives a concise definition of FITS consistent with the IAU recommendations. The office is coordinated by B. Schlesinger. 1.3 Network News Group ---------------------- During the fall of 1991, D. Wells made a network News group for discussions of FITS matters. It was combined with an e-mail exploder because the USEnet News facility is not available at all sites. A vote on the creation of an official News group 'sci.astro.fits' was made in the spring of 1992. This News group has already been of great value by broadening the discussion of the FITS standard. It gives a natural forum for reviewing new proposals and suggestions. Further, it provides a convenient tool to obtain clarifications or solutions on FITS problems. 2. Proposals and Agreements --------------------------- Three proposal for extensions to the FITS standard were submitted for Binary Tables, Images and Blocking. The Binary Table extension provides higher efficiency by using binary storage of data instead of only ASCII representation as in the current Table format. Further, it allows table entries to be arrays, opening the possibility to transport spectra and images as single entries. The Image extension enables matrices of different size or data type to be saved in the same FITS file. It corresponds closely to basic FITS and is useful for auxiliary data such as PSF's or quality flags. The Blocking convention specifies the rules for recording FITS files on different media e.g. 8mm Exabyte, QIC and DDS/DAT cartridges. It also gives general recommendation for FITS on magnetic disks. 3. Availability --------------- Information on the FITS standard can be obtained via E-mail from the following Internet addresses: B. Schlesinger, NASA : bschlesinger at nssdca.gsfc.nasa.gov P. Grosbol, ESO : pgrosbol at eso.org D. Wells, NRAO : dwells at nrao.edu Documentation, general information and test data files are available on anonymous FTP to nssdca.gsfc.nasa.gov [128.183.36.23], ftphost.hq.eso.org [134.171.8.4] and fits.cx.nrao.edu [192.33.115.8]. From dwells at azalea.cx.nrao.edu Wed Jun 10 20:09:00 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5829" "Thu" "11" "June" "1992" "01:07:36" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "130" "Minutes of European FITS Committee Meeting, 1992 May 14-15" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory, Charlottesville, VA. From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: Minutes of European FITS Committee Meeting, 1992 May 14-15 Date: Thu, 11 Jun 1992 01:07:36 GMT [NOTE- the two messages below were transmitted to "fitsbits" yesterday, but did not explode due to a technical problem -Don] =-=-=-=-=-=-=-=-=-=-=-=-=-=-= From: pgrosbol at eso.org Subject: Minutes of European FITS Committee Meeting, 1992 May 14-15 Date: Tue, 09 Jun 92 13:27:28 +0200 Draft Minutes of 8th Meeting of European FITS Committee prepared by P.Grosbol ESO, 1992-June-09 The 8th meeting of the European FITS Committee was held in two sessions during the 4th Data Analysis Workshop in ESO, May 14-15, 1992. The meeting was attended by 10-15 people with many FITS Committee members e.g. M.Currie, P.Grosbol, P.Kalberla, F.Pasian, D.Ponz, E.Raimond, B.Peterson and D.Tody. The agenda was as follows: 1) Status report (Chairman) 2) Blocking Proposal 3) Binary Table Extension 4) Image Extension 5) World Coordinates in FITS 6) Any Other Business Besides the general status report the main discussion was concerned with the Blocking, BINTABLE and IMAGE Proposals. There were general agreement on that the Proposals were acceptable and could be puts forward to a vote in their present form. The more detailed issues were: 1) Status report: The annual report for 1991 of the IAU FITS Working Group was distributed and commented. The main issues were the IAU General Assembly, creation of the 'sci.astro.fits' news group by D.Wells, availability of anonymous ftp accounts with FITS documents, and the NASA FITS Support office including the draft of a standard document for 'Implementation of the FITS'. The main developments after this report are the official creation of the 'sci.astro.fits' news group, a final proposal for an IMAGE extension and the general discussion on the proposals through the news group. 2) Blocking: All expressed general agreement. The issue on whether or not 8mm Exabyte tapes should be defined as fixed or variable length was raised. There had not been extensive test of the fixed block QIC cartridges. 3) BINTABLE: The data types were discussed with reference to arrays of string instead of only characters. The main argument for arrays of strings was the usage for defining units and types of multi-dimensional arrays. Since this is not a main issue yet, it was recommended only to allow arrays of characters in the current proposal and review the issue when array definitions are more tested. Only the main proposal is up for recommendation NOT the appendices on Multi-dimensional and Variable Length Arrays. 4) IMAGE: All agreed to recommend the current proposal i.e. with the name 'IMAGE', PCOUNT=0 and GCOUNT=1. It was found that this extension 5) World-Coordinates: There were general agreement on that a firm definition was required even if it did not include all non-linear cases. 6) Any Other Business: A proposal from V.Lipovetsky (Russian 6m) was circulated. Many of the features could be encoded using the BINTABLE proposal which was not know to him at the time. A definition of data sequences in arrays was introduced. Compression of FITS data files were discussed. It was unclear if the better approach would be to have a special compress extension or just compress to total FITS file using standard tools. The introduction of hardware compress for several devices (e.g. 8mm and DAT) would solve some problems in transport, however, the techniques used are not yet standard. It was agreed that the three proposals listed below were acceptable in their present form and could be recommended: a) Binary Table Extension to FITS, W.D.Cotton and D.Tody, (20 September 1991). Note: only the main proposal is considered NOT the appendices! b) The FITS IMAGE Extension, A Proposal, J.D.Ponz, R.W.Thompson, J.R.Munoz (7 February 1992). c) Draft Proposal for Blocking of Fixed-block Sequential Media and bitstream Devices, P.Grosbol, D.Wells (1990-August-10). The texts are available through anonymous ftp at ESO, NRAO and GSFC. Since not all members of the Committee were present the final vote would be done in June through e-mail. Final testing, wording and possible clarification will be done through the review by the IAU FITS Working Group. =-=-=-=-=-=-=-=-=-=-=-=-=-=-= From: pgrosbol at eso.org Subject: Addendum to Minutes of European FITS Committee 8th Meeting Date: Tue, 09 Jun 92 17:10:15 +0200 Addendum to Draft Minutes of 8th Meeting of European FITS Committee prepared by P.Grosbol ESO, 1992-June-09 Due to an error I had during the editing, point 4) of the minutes was not complete. The full text is given below (sorry): 4) IMAGE: All agreed to recommend the current IMAGE proposal i.e. with the name 'IMAGE', PCOUNT=0 and GCOUNT=1. It was found that this extension is required to provide an easy way of append other images (e.g. flags, errors, or PSF's) to the prime matrix . Although the BINTABLE extension later may be extended with conventions to encode images, such conventions (see Appendices in BINTABLE proposal) are yet neither tested nor finalized. To ensure that the IMAGE extension remains simple the values PCOUNT=0 and GCOUNT=1 were recommended. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From opus at pioneer.unm.edu Sat Jun 13 17:28:07 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3713" "" "13" "June" "92" "19:46:14" "GMT" "Colby Kraybill" "opus at pioneer.unm.edu " nil "108" "Image/Data Archive unveiling ***" "^From:" nil nil "6"]) Newsgroups: sci.space,sci.astro,sci.astro.fits Organization: Space and Planetary Image Facility From: opus at pioneer.unm.edu (Colby Kraybill) Subject: Image/Data Archive unveiling *** Date: 13 Jun 92 19:46:14 GMT We are happy to announce the free availability of our database of over 150 cd-rom's. pioneer.unm.edu is the Space and Planetary Image Facility, located on the University of New Mexico campus. It is supported by the Computer and Information Resource Technology (CIRT) group on this campus, and also by the Space and Planetary Image Facility (SPIF). CIRT has provided the machine, diskspace, and cd-rom players. SPIF has provided the cd-roms and also the people maintaining the machine. This project is still in it's infancy, so you will have to bare with us. Right now you can only FTP into the machine and look around at what happens to be on the cd-rom drives, and also what software that is held on the machine. We are working on many various projects to increase the usefullness of the machine and it's database. Pioneer is a DECStation 5000/120 with 1.5 gigabytes of diskspace and three CD-ROM players. Budget cuts not withstanding, another 1.5 gigabyts of diskspace and a few more CD-ROM players around July. Here is the listing of the CD-ROM's that can be mounted on the players: ---------------------------------------------------------------------------- Title Subtitle Disks ---------------------------------------------- ----------------------- ------- Airborne Antarctic Ozone Experiment Aug/Sep '87 1/1 Airborne Artic Stratospheric Experiment Jan/Feb '89 1/1 DMSP SSM/I Brightness Temperature Grids Polar Regions 12 DMSP SSM/I Ice Grids 1 Einstein X-ray Data Slew Survey 1/1 High Resolution Imager 2/2 IPC Souces Catalog 3/3 Geological Remote Sensing Field Experiment 9/9 Gloria Sonar Data Atlantic Coast 2/2 Green Bank Sky Maps and Radio Source Catalog Radio Astronomy 1/1 High Resolution Bathymetry and Selected Geoscience Data for the Monterey Bay Region 1/1 Hubble Guide Star Catalog 2/2 International Halley Watch Sample Disk Tabular Data 1/1 IRAS star catalog 4/4 MacWorld Software Showcase 1/1 Magellan 1-22 & 30-42 34/52 Mars Digital Image Map Mosaics 6/6 National Energy Research Seismic Library 1/1 Nimbus 7 SMMR (disks 1-6 are N Pole radiances, 1-12 disks 8-12 are S Pole radiances, and disk #7 is NP radiances and seaice concentrations) PreMagellan Radar and Gravity Data 1/1 Selected Astronomical Catalogs 1/1 SIGCAT Sampler 1/2 SIGCAT Software Showcase 1/1 TOMS Gridded Ozone Data 1978-1991 2/2 Tropical Ocean Global Atmosphere 1/1 Viking Optical Data Raw Data 8/30 Viking Thermal Data 1/1 Voyager Optical Data 12/12 West Coast Time Series Coastal Zone Color Scanner Imagery 1/1 __________________________________________________________________________ | CD-ROM's in the possesion of the Map and Geographic Information Center | | (MAGIC) located in the Centennial Science and Engineering Library | -------------------------------------------------------------------------- Digital Line Graph Data 4/x Geology of Nevada 1/1 Orthophoto USDA SCS - USGS NMD Joint Project 1/1 Side Looking Airborne Radar 1/1 SIGCAT Sampler 1/1 Supermap Census Data 1/1 World Atlas 1/1 World Weather Disk 1/1 From carls at triton.unm.edu Sun Jun 14 08:10:13 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["346" "Sat" "13" "June" "92" "22:31:15" "GMT" "Mr.HolierThanThou" "carls at triton.unm.edu " nil "11" "Re: Image/Data Archive unveiling ***" "^From:" nil nil "6"]) Newsgroups: sci.space,sci.astro,sci.astro.fits Organization: University of New Mexico, Albuquerque From: carls at triton.unm.edu (Mr.HolierThanThou) Subject: Re: Image/Data Archive unveiling *** Date: Sat, 13 Jun 92 22:31:15 GMT Please note that the list of cd-roms includes a few disks that we may only make available to UNM as some of the data and software on the disks may not be 'public domain'. We will try to get permission >from our disk suppliers to put these disks on the net. Thank you. Bruce Carlson carls at pioneer.unm.edu carls at triton.unm.edu . From thompson at stars.gsfc.nasa.gov Fri Jun 19 06:50:53 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["764" "" "18" "June" "92" "17:53:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "15" "Keywords longer than eight characters" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Keywords longer than eight characters Date: 18 Jun 92 17:53:00 GMT Is there any kind of defacto way to store keywords longer than eight characters in a FITS header? Someone suggested using the COMMENT keyword followed by the non-standard keyword, although another possibility is using the null keyword (eight blanks). Is this what other people do? I'm not talking about keywords essential to reading and scientifically processing the data. I'm more interested in storing engineering information about the state of the instrument. It would be desirable to retain the original mnemonic names used by the engineers. Important information could also be replicated using more standard FITS keywords. Although it's obviously legal to put anything in a COMMENT field, I'm interested in knowing what other people do. Bill Thompson From forveill at gag.observ-gr.fr Fri Jun 19 06:51:01 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["652" "Fri" "19" "June" "1992" "09:14:43" "GMT" "Thierry Forveille" "forveill at gag.observ-gr.fr " nil "12" "Re: Keywords longer than eight characters" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: forveill at gag.observ-gr.fr (Thierry Forveille) Subject: Re: Keywords longer than eight characters Date: Fri, 19 Jun 1992 09:14:43 GMT Althought I never had to face that specific problem, I would suggest you store the information with a standard keyword and append the original mnemonic as an end of line comment (after the /). This is standard too, and the information will be easier to extract for a general FITS handling program if you later want to plot this engineering information. Extracting the information from a COMMENT or blank keyword would probably need ad hoc (though trivial) modifications of the program. Of course the standard keyword need not be particularly informative and something line ENGIN001 to ENGIN999 would work. Thierry Forveille Observatoire de Grenoble From dwells at azalea.cx.nrao.edu Fri Jun 19 12:53:16 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1136" "Fri" "19" "June" "1992" "17:50:26" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "21" "VLBA_format.ps available" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory, Charlottesville, VA. Distribution: sci From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: VLBA_format.ps available Date: Fri, 19 Jun 1992 17:50:26 GMT The anonymous-FTP archive on fits.cx.nrao.edu [192.33.115.8] now contains the 318_KB Postscipt file /FITS/Documents/VLBA_format.ps. This document describes the syntax and semantics of the set of BINTABLE extensionss which the VLBA will use to convey results from the Correlator to AIPS and to other data analysis ("post-processing" in radio-speak) packages. Radio astronomers may want to take a look at it to see how BINTABLE will be used to replace the older Random Groups extension to FITS. Please note that this document is a *draft*; it has been under development for more than a year, and is still being modified. It shows the current state of negotiation (and debugging) of the software interface between the VLBA Correlator and AIPS1. I think it would be a "good thing" if the archive contained analogous documents for other telescopes. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From warnock at Hypatia.gsfc.nasa.gov Fri Jun 19 17:45:35 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3051" "Fri" "19" "June" "1992" "19:21:34" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "53" "Thoughts from WGAS -- compression and nesting extensions " "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: NSSDC/Hughes STX From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Thoughts from WGAS -- compression and nesting extensions Date: Fri, 19 Jun 1992 19:21:34 GMT I've been thinking about our data compression proposal since the WGAS meeting last week, and talking to some people about possible modifications. Here are some semi-random thoughts for discussion... 1. Chris Biemesderfer recommends changing the optional "logical" keywords (LAXIS, LBITPIX, etc), so that the first character is a "U" (for uncompressed), and so that the "U" character is just prepended to each of the keywords which go in the header for the uncompressed data. This has the advantage of being extensible (i.e., there could conceivably be other characters used for other extensions and/or coding schemes) and mnemonic. I like it. 2. As I mentioned at the WGAS, Bill Thompson has pointed out that the proposal will no work for lossy compression schemes (which Immanual Freedman calls "approximate" compression schemes - I like that terminology), since the current proposal calls for compressing the entire original data stream, header and all. I'm not sure what to do about this. There's a certain beauty in uncompressing the bitstream and getting a fully-qualified FITS data stream out of it (hey - it's recursion... I always {\em knew} there had to be a use for it besides calculating factorials :-). The difficulty lies in figuring out a way to save the header(s) which go along with the resulting uncompressed data, but not necessarily required by the compressed stuff. The current proposal calls for compressing the headers with the data because that ensures that they're retrieved when the data is uncompressed, and that they're unambiguously associated with the uncompressed data. I claim it doesn't matter if the compression algorithm doesn't work well for the headers - if the data volume is such that the header is roughly the same size as the data, it's probably not a good candidate for compression anyway. Now... here's an idea which addresses some of these issues, but I warn you that I don't know how to cast it into FITS yet. What is probably sufficient, if not necessary, to allow a solution to the problem is to develop a scheme to allow extensions to be nested. Here's why - if we can do that, we can make the first extension indicate compressed data, and within that, nest the headers necessary for documenting the compressed data, tables, etc. The question is, how would such a syntax be built? Suppose the next data record after an extension header were another extension header (with due care taken in specifying the values of keywords related to the length of the data)? Tractable? ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From warnock at Hypatia.gsfc.nasa.gov Fri Jun 19 17:47:13 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5677" "Fri" "19" "June" "1992" "19:22:38" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "102" "Thoughts from WGAS -- IMAGE extension and data structures" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Thoughts from WGAS -- IMAGE extension and data structures Date: Fri, 19 Jun 1992 19:22:38 GMT Here are a couple of related issues which I got to thinking about at the WGAS meeting in Columbus last week. I want to spur a discussion on namings, in connection with the proposed IMAGE extension. While I don't have any particular problems with the proposal, as such, there is an issue I feel needs to be addressed by the FITS community, so we can achieve some sense of direction for the future. The word "IMAGE", as commonly used by astronomers, and as used in the IMAGE extension proposal actually has two meanings. The first is the interpretation of the word "IMAGE" as a high-level object, i.e., something which the scientist (or whoever) views, prints, processes, whatever. The second meaning is that of the underlying data structure, in which an image is stored as a 2-D raster, implied by the data representation described by the original FITS document and the IMAGE extension proposal. What I'm aiming for here is to point out that we place an undue burden on the end-user by confounding the two meanings. There is a natural division between data structures (which machines have to know about and end-users don't), and data interpretations (which end-users have to know about and machines don't). At some point, we (the astronomical community in general, and the FITS community in particular) need to address this issue. It would even be acceptable, though probably not desirable, to say that we understand the issue and will explicitly ignore it. But I think as FITS becomes more and more of a working format (via Pence's FITSIO and FTOOLS packages, for example), it becomes a crucial issue. Right now, it's simply the {\em name} IMAGE which bothers me. To the end-user with some computer science knowledge, there are any number of ways images can be stored besides the conventional pixel-by-pixel, row-by-row. Block storage, for example, can be efficient and fast. Yet, if we approve this extension with this particular name, no other data structure can ever have the name IMAGE, though images they may be. This imparts a burden on the end-user which can easily be off-loaded to the machine if we can develop a scheme to name underlying data structures independently from the high-level data interpretations. The result of such a scheme would be to allow end-users to deal with things they are familiar with (images, spectra, tables,..), without having to know about the specifics of data storage and structure. Let me give a brief example, from outside FITS, to illustrate the point a bit better. In the course of ingesting the International Halley Watch (IHW) data into the Planetary Data System (PDS), we had to develop a translation program to generate PDS-style labels from the FITS headers which IHW originally supplied. We found that there were spectra in the IHW archive, from different disciplines, which had been stored in a variety of ways - as ordered pairs (wavelength, flux), as pairs of vectors (all wavelengths, then all fluxes), as a starting wavelength, an increment and a vector of fluxes, even a 2-D "image" (an echelle). All of these use different data structures (within FITS!) to represent a single high-level object - a spectrum. Our initial contacts with the PDS had them suggesting that each of these things have a unique name, thus requiring the end-user to know all of the possible names a spectrum might have. Our response was that the PDS label syntax allowed for a nesting of these "object" definitions, and that a single high-level object called "SPECTRUM" would suffice, providing that the underlying data structures could be appropriately named. In this way, the software developer would be able to supply the I/O routines to read/write the data structures without having to know that the user was calling them spectra (and saving code - an echelle could be read with the same code that read an IMAGE). In addition, the end user doesn't have to know what data structures are being used - he/she gets to read and write "SPECTRUM" objects. Prior to this proposed extension, there really wasn't an instance where such confusion could arise. The "official" extensions were pretty much self-descriptive. Data which occurred in the primary data stream wasn't identified as to interpretation at all. The current IMAGE extension proposal is the first one to really blur that distinction, and I feel the issue should be addressed in one way or another. Do I have a proposal to make which will encode this kind of distinction? Nope - at least, not yet. Second, I'd like to point out the related issue, also raised by Immanuel Freedman, that we'd better start thinking about fitting advanced data structures into FITS. If we're going to use FITS as a working format, rather than just an interchange format, issues of efficiency become important. We will {\em need} to be able to put things like quadtrees and block-stored images into FITS. We will need to be able to store related structures - like keeping palettes with images - and to do so in a way which makes explicit the relations. Just putting a bunch of extensions into the same file doesn't do that. The floor is open for discussion... ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From bschlesinger at nssdca.gsfc.nasa.gov Fri Jun 19 17:47:34 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["6137" "" "19" "June" "92" "21:13:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdca.gsfc.nasa.gov " nil "125" "FITS basics and information sources (monthly posting)" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: NASA - Goddard Space Flight Center News-Software: VAX/VMS VNEWS 1.41 Nntp-Posting-Host: nssdca.gsfc.nasa.gov From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Subject: FITS basics and information sources (monthly posting) Date: 19 Jun 92 21:13:00 GMT This basic FITS information will be posted and updated periodically for the benefit of new readers and the reference of old readers. FITS (Flexible Image Transport System) is a data format designed to allow convenient exchange of astronomical data between installations whose standard internal formats and hardware may be different. A FITS file consists of a sequence of Header Data Units (HDUs). The header consists of keyword=value statements, which describine the format and organization of the data in the HDU, and may contain other information such as information about the instrument or the history of the data or data set. The data follow, structured as the header specifies. The data section of the HDU may contain a digital image, but, except for the first, IT DOESN'T HAVE TO. Other possible formats include tables and multidimensional matrices that are not images. The first HDU must contain a multidimensional matrix or no data at all; the data in subsequent HDUs, called extensions, may be of any type, consistent with certain rules. The "Image" in the name comes from the original use of the format to transport digital images, but it's not just for images any more. FITS is not principally a graphics format designed for the transfer of pictures that includes support for many platforms as part of the package for decoding the image. Separate software must be developed or obtained for converting the data from the FITS file into a form that can be readily displayed. As has been discussed in this newsgroup, and in alt.sci.astro.fits before it, the Extended Portable Bitmap Toolkit (pbm+) can be used for converting many FITS files to such a format, but support for all FITS files where the data are in the form of an image, in particular those where the data matrix members are in IEEE floating point format (BITPIX<0) or the matrix has more than two dimensions (NAXIS>2), is not guaranteed. Discussion of FITS - image converters appears in this newsgroup from time to time. The fundamental references on FITS are the following four papers, often referred to collectively as the "Four FITS Papers". These papers are the formal standard for FITS, endorsed by the International Astronomical Union (IAU). Wells, D. C., Greisen, E. W., and Harten, R. H., "FITS: a flexible image transport system," Astronomy and Astrophysics Supplement Series, 44, 363-370, 1981. Greisen, E. W. and Harten, R. H., "An extension of FITS for small arrays of data," Astronomy and Astrophysics Supplement Series, 44, 371-374, 1981. (NOTE: The format described in this paper has been used almost exclusively to transport radio interferometry and is likely to be replaced by other formats in the future. Writing data other than radio interferometry data using this format is not recommended.) Grosbol, P., Harten, R. H., Greisen, E. W., and Wells, D. C., "Generalized extensions and blocking factors for FITS," Astronomy and Astrophysics Supplement Series, 73, 359-364, 1988. Harten, R. H., Grosbol. P., Greisen, E. W., and Wells, D. C., "The FITS tables extension, Astronomy and Astrophysics Supplement Series, 73, 365-372, 1988. A User's Guide for FITS, commissioned by NASA Headquarters, is maintained by the NASA/OSSA Office of Standards and Technology (NOST). This Guide is intended to be a tutorial for new FITS users. In addition to presenting the rules of FITS, it provides some of the history and reasoning behind the choice of the rules, adds recommendations on good practices, and discusses current developments in FITS. This document is available only in hard copy form. NASA is sponsoring development of a formal standard for FITS. The goal of this process is a document consistent with FITS as endorsed by the IAU, eliminating some contradictions and ambiguities in the original FITS papers, that can be endorsed by the IAU FITS Working Group as the FITS standard. The document is being developed by a Technical Panel chaired by Dr. Robert J. Hanisch (STSci), with review by the astronomical community. Only minor revisions are expected to the current draft, version 0.3b, but the form of the standard is not final, and it does not supersede the four papers and Floating Point Agreement endorsed by the IAU as the official standard for FITS. The IAU has endorsed the Floating Point Agreement, which defines how floating point numbers are to be stored in the data matrix defined in the first of the four papers. The basic agreement appears verbatim in the User's Guide, and the substance is incorporated in the Draft NASA Standard. The NOST maintains a file of FITS information available by anonymous ftp from nssdca.gsfc.nasa.gov or DECnet copy from NSSDCA, in the directory FITS. It includes copies of the current draft NASA Standard in flat ASCII, PostScript, and LaTeX, and a list of the current extension type (structure) names registered with the IAU FITS Working Group. It also contains, in LaTeX form, the text of the proposal for one of these new extension types, IMAGE. A README. file describes the contents of the directory. There is a further subdirectory, SOFTWARE, described by README.FIRST, containing a program in C to read and list the headers of a FITS file and another file with information on publicly available FITS software packages. Paper copies of many of the documents listed above can be obtained from the NOST Librarian. Paper copies of the User's Guide and either paper or electronic copies of the Draft Implementation Standard, for those without ftp access, are available. Because of restrictions set by the copyright holder, NOST can send copies of the four FITS papers only to non-profit organizations. The addresses of the NOST are as follows: (Postal) NASA/OSSA Office of Standards and Technology Code 933 Goddard Space Flight Center Greenbelt MD 20771 USA (Internet) nost at nssdca.gsfc.nasa.gov (DECnet) NCF::NOST Please mention this posting in your request. Barry M. Schlesinger Coordinator, NASA/NSSDC NOST FITS Support Office (301) 513-1634 fits at nssdca.gsfc.nasa.gov NCF::FITS From pence at heawk1.gsfc.nasa.gov Fri Jun 19 17:48:36 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4205" "Fri" "19" "June" "1992" "20:13:10" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "78" "string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: string arrays Date: Fri, 19 Jun 1992 20:13:10 GMT Convention for Character String Arrays in FITS Binary tables. In the proposed FITS binary table format (Cotton and Tody, 20 Sept 1991) one can define a table column as an array of single characters (e.g., TFORMn = '20A' indicates that the column is 20 characters wide), but there is no strictly defined convention to indicate that a column contains an array of strings. To take the same example, suppose the 20-character field is composed of 5 4-character length strings (e.g.,'BearDuckBirdFishWorm'). The question then is what convention should be used within FITS for storing the information about the length of strings so that FITS readers and writers will know how to interpret the sequence of characters in a table column. One could use the convention proposed in Appendix A and add a keyword to the header of the form: TDIMn = '(4,5)' however I don't like this for a variety of reasons. Instead, I propose that the following convention, which is completely consistent with the proposed binary table standard, be appended to the binary table definition document. Any comments or suggestions are welcome. -Bill Pence, GSFC/HEASARC pence at tetra.gsfc.nasa.gov LHEAVX::PENCE -------------------------------------------------------------------------- Appendix C "Character String Array" Convention The binary table format allows any field of a table to be an array, but there is no simple convention for defining arrays of fixed-length strings within an ASCII character field. Since the use of fixed-length strings is arguably the simplest and most convenient way to represent arrays of strings within FITS (readers and writers can then directly compute the offset to any substring within the field) this merits having a special convention for this purpose. This convention uses the optional characters which are allowed to follow the datatype code character in the TFORMn keyword value (described in section 3.9 of the definition document) to specify the length of the character stings within the field. The value of the TFORM keyword then has the form 'rAw' where r is an integer specifying the total number of characters in the table field and w is an optional integer specifying the width of the fixed-length character strings within the field. FITS readers and writers which recognize this convention can then efficiently access individual strings within the field; other FITS readers and writers can simply ignore the optional w characters in the TFORMn keyword and treat the field as a single one-dimensional array of characters (unless some other convention for defining string lengths is recognized). It is recommended that the value of r should be an integer multiple of w, however, if this is not the case then any remaining characters after the last string in the field should be considered as undefined. For example if TFORMn = '23A4' then the field should be interpreted as containing 5 strings each 4 characters in length, followed by 3 undefined characters. If the value of w is greater than r, or if w is not an integer, then it does not conform to this convention and should be ignored (or interpreted under some other convention). The following examples illustrate the use of this convention: TFORMn Interpretation ------ -------------- '20A' a single string 20 characters in length '20A4' 5 strings each 4 characters in length '23A4' 5 strings each 4 characters in length followed by 3 undefined characters '20A100' a single string 20 characters in length (w is greater than r so should be ignored) '20A/000' a single string of 20 characters in length (the unrecognized character following the datatype code character should be ignored) This convention is optional and will not preclude other conventions. This convention is not part of the proposed binary table definition. From cflatter at nrao.edu Fri Jun 19 18:18:14 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3232" "Fri" "19" "June" "1992" "21:56:56" "GMT" "Chris Flatters" "cflatter at nrao.edu " nil "85" "Data structure in FITS (Was: Re: Thoughts from WGAS...)" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters) Subject: Data structure in FITS (Was: Re: Thoughts from WGAS...) Date: Fri, 19 Jun 1992 21:56:56 GMT In article 8132 at nsisrv.gsfc.nasa.gov, warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) writes: >Second, I'd like to point out the related issue, also raised by Immanuel >Freedman, that we'd better start thinking about fitting advanced data >structures into FITS. If we're going to use FITS as a working format, >rather than just an interchange format, issues of efficiency become >important. We will {\em need} to be able to put things like quadtrees >and block-stored images into FITS. We will need to be able to store >related structures - like keeping palettes with images - and to do so in >a way which makes explicit the relations. Just putting a bunch of >extensions into the same file doesn't do that. The FITS binary table extension already allows arbitrarily complex data structures to be stored. A table defines a data {\it type}. It is equivalent to a PASCAL record definition or a C struct definition (or a Fortran 90 TYPE definition). The columns in the table represent the components of that type. Each row in the table represents an {\it instance} of that type. Each instance is uniquely identifed by the name of the table it occurs in, the version number of the table and its row number. It is therefore possible to define structures that refer to instances of other structures by including columns which contain the name and version number of the table in which the referenced instance occurs and its row number. For example suppose that we have a binary tree of integers. In C we might declare the following struct to hold tree nodes. struct TreeNode { int value; /* Value */ TreeNode *left; /* Pointer to left subtree or null pointer */ TreeNode *right; /* Pointer to right subtree or null pointer */ }; We could store a binary tree of integers in a single FITS binary table with the following header. XTENSION='BINTABLE' BITPIX = 8 / Binary data NAXIS = 2 NAXIS1 = 12 / 3 integers at 4 bytes each NAXIS2 = ? / The number of nodes in the tree PCOUNT = 0 GCOUNT = 1 TFIELDS = 3 EXTNAME = 'BTREE ' / For example EXTVER = 1 TFORM1 = '1J ' TTYPE1 = 'VALUE ' TUNIT1 = ' ' TFORM2 = '1J ' TTYPE2 = 'LEFT SUBTREE ' TUNIT2 = ' ' TFORM3 = '1J ' TTYPE3 = 'RIGHT SUBTREE ' TUNIT3 = ' ' END In this case all of the node of the tree will be in the same table so we do not need table fields containing the names and version numbers of the tables containing the left and right subtrees. The tree 5 / \ 2 9 / \ \ 7 6 8 could be written using such a table as Row No. | VALUE | LEFT SUBTREE | RIGHT SUBTREE | 1 | 5 | 2 | 5 | 2 | 2 | 3 | 4 | 3 | 7 | 0 | 0 | 4 | 6 | 0 | 0 | 5 | 9 | 0 | 6 | 6 | 8 | 0 | 0 | Chris Flatters cflatter at nrao.edu From thompson at stars.gsfc.nasa.gov Fri Jun 19 19:26:49 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3169" "Fri" "19" "June" "1992" "22:25:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "57" "Re: Thoughts from WGAS -- compression and nesting extensions" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: Thoughts from WGAS -- compression and nesting extensions Date: Fri, 19 Jun 1992 22:25:00 GMT In article <1992Jun19.192134.8051 at nsisrv.gsfc.nasa.gov>, warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) writes... >I've been thinking about our data compression proposal since the WGAS >meeting last week, and talking to some people about possible >modifications. Here are some semi-random thoughts for discussion... > > ... (deleted stuff) ... > >2. As I mentioned at the WGAS, Bill Thompson has pointed out that the >proposal will no work for lossy compression schemes (which Immanual >Freedman calls "approximate" compression schemes - I like that >terminology), since the current proposal calls for compressing the >entire original data stream, header and all. I'm not sure what to do >about this. There's a certain beauty in uncompressing the bitstream and >getting a fully-qualified FITS data stream out of it (hey - it's >recursion... I always {\em knew} there had to be a use for it besides >calculating factorials :-). The difficulty lies in figuring out a way >to save the header(s) which go along with the resulting uncompressed >data, but not necessarily required by the compressed stuff. > > ... (more deleted stuff) ... > >Now... here's an idea which addresses some of these issues, but I warn >you that I don't know how to cast it into FITS yet. What is probably >sufficient, if not necessary, to allow a solution to the problem is to >develop a scheme to allow extensions to be nested. Here's why - if we >can do that, we can make the first extension indicate compressed data, >and within that, nest the headers necessary for documenting the >compressed data, tables, etc. The question is, how would such a syntax >be built? Suppose the next data record after an extension header were >another extension header (with due care taken in specifying the values >of keywords related to the length of the data)? Tractable? As far as I'm concerned, anything is allowed to be stored in the data records of a FITS extension, so long as the total number of records used can be calculated from the BITPIX, NAXIS*, PCOUNT and GCOUNT keywords. Software that doesn't know how to read that kind of extension should be able to simply skip over it. One simple way of doing this that I thought of, after reading this, would be to store the header after the compressed data array. The PCOUNT keyword could be used to give the number of bytes in the stored header. I haven't thought everything out yet, but there would have to be keywords signalling that there was or was not a stored header to go with the compressed data, and maybe a keyword signalling where the header started in the parameter area, similar to THEAP for binary tables. One might want to start the stored header on a 2880-byte record boundary. When I talked to Archie, I understood that he was thinking about using nested headers for other things besides the data compression. I don't want to get into what those could be, but at first blush it seems to me that one could store anything in the parameter field, including another complete FITS file if one wanted to. The FITS readers could get pretty hairy, but there's no reason in principal why it couldn't be done. Bill Thompson From tody at noao.edu Sun Jun 21 19:17:15 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3695" "" "21" "June" "92" "22:09:16" "GMT" "Doug Tody" "tody at noao.edu " nil "73" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Optical Astronomy Observatories, Tucson, AZ, USA Originator: tody at gemini.tuc.noao.edu Nntp-Posting-Host: gemini.tuc.noao.edu From: tody at noao.edu (Doug Tody) Subject: Re: string arrays Date: 21 Jun 92 22:09:16 GMT >From article , by pence at heawk1.gsfc.nasa.gov ( William Pence): > > ...however I don't like this for a variety of reasons. Instead, I propose > that the following convention, which is completely consistent with the > proposed binary table standard, be appended to the binary table > definition document. Any comments or suggestions are welcome. > > -Bill Pence, GSFC/HEASARC pence at tetra.gsfc.nasa.gov > LHEAVX::PENCE > > -------------------------------------------------------------------------- > Appendix C > > "Character String Array" Convention > > The binary table format allows any field of a table to be an array, but > there is no simple convention for defining arrays of fixed-length > strings within an ASCII character field. Since the use of fixed-length > strings is arguably the simplest and most convenient way to represent > arrays of strings within FITS... Bill and I discussed the general problem of arrays of strings in a get-together following the WGAS meeting at the Columbus AAS. My conclusion after this discussion was that the fixed length substring approach Bill proposes as a convention above is too restrictive to be put forward as the recommended way to solve this problem. To see why this might be so we can examine two alternative approaches to the problem of representing arrays of strings. 1) The fixed length substring approach. In this case we have something like the following for a "72A12" string array. "red green blue yellow turquoise aquamarine" 2) The delimited string approach. In this case the declaration might be something like "43A/040" and the stored data might look as follows. "red green blue yellow turquoise aquamarine" In both cases the array-of-strings is stored as a simple NUL terminated binary tables character array, and the convention used is compatible with the binary table definition. The fixed length substring approach is convenient for random access but is wasteful of space and imposes a rather severe limit on the length of each substring. The delimited substring approach can only be accessed sequentially, but is space efficient and removes the restriction on the length of a substring. If the delimiter character is parameterized as in the example declaration, common delimiters such as blank, comma, newline, or maybe even EOS (NUL) may be used, with obvious applications to various types of text data. There is a wealth of experience in C demonstrating how to access delimited strings efficiently, so I don't consider the sequential access limitation serious. There is a third approach which is a variant of the fixed length substring approach, but using TDIM to define the length of each substring. This is equivalent to what Bill suggests except for the declaration syntax. In my opinion the second approach suggested above is the better one, since it is space efficient (preferred for an archival storage format like FITS) and since it is much more general in the range of string data which can be stored. The main point I want to make though, is that the convention Bill suggests is by no means the only one, and is not necessarily even the best one. If we have to add a convention for representing arrays of strings, one which permits *either* fixed length or delimited substrings would be preferred. Doug Tody -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From warnock at Hypatia.gsfc.nasa.gov Mon Jun 22 10:54:31 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1669" "" "22" "June" "92" "14:14:24" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "44" "Re: Data structure in FITS (Was: Re: Thoughts from WGAS...)" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: Goddard Space Flight Center Nntp-Posting-Host: hypatia.gsfc.nasa.gov From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Re: Data structure in FITS (Was: Re: Thoughts from WGAS...) Date: 22 Jun 92 14:14:24 GMT In article <1992Jun19.215656.25564 at nrao.edu>, cflatter at nrao.edu (Chris Flatters) writes: |> For example suppose that we have a binary tree of integers. In C |> we might declare the following struct to hold tree nodes. |> |> struct TreeNode |> { |> int value; /* Value */ |> TreeNode *left; /* Pointer to left subtree or null pointer */ |> TreeNode *right; /* Pointer to right subtree or null pointer */ |> }; |> |> We could store a binary tree of integers in a single FITS binary table |> with the following header. |> |> XTENSION='BINTABLE' |> BITPIX = 8 / Binary data |> NAXIS = 2 |> NAXIS1 = 12 / 3 integers at 4 bytes each |> NAXIS2 = ? / The number of nodes in the tree |> PCOUNT = 0 |> GCOUNT = 1 |> TFIELDS = 3 |> EXTNAME = 'BTREE ' / For example |> EXTVER = 1 |> TFORM1 = '1J ' |> TTYPE1 = 'VALUE ' |> TUNIT1 = ' ' |> TFORM2 = '1J ' |> TTYPE2 = 'LEFT SUBTREE ' |> TUNIT2 = ' ' |> TFORM3 = '1J ' |> TTYPE3 = 'RIGHT SUBTREE ' |> TUNIT3 = ' ' |> END OK, so far I'm with you. Have we instituted some sort of control authority to handle the values of EXTNAME so applications can recognize the data structures there? Should we? -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From warnock at Hypatia.gsfc.nasa.gov Mon Jun 22 10:55:08 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1522" "Mon" "22" "June" "1992" "14:07:32" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "28" "Re: Thoughts from WGAS -- compression and nesting extensions" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Re: Thoughts from WGAS -- compression and nesting extensions Date: Mon, 22 Jun 1992 14:07:32 GMT In article <19JUN199218254599 at stars.gsfc.nasa.gov>, thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes: |> One simple way of doing this that I thought of, after reading this, would be to |> store the header after the compressed data array. The PCOUNT keyword could be |> used to give the number of bytes in the stored header. I haven't thought |> everything out yet, but there would have to be keywords signalling that there |> was or was not a stored header to go with the compressed data, and maybe a |> keyword signalling where the header started in the parameter area, similar to |> THEAP for binary tables. One might want to start the stored header on a |> 2880-byte record boundary. |> I sorta like this idea. Let's think on it a bit and see if anyone else leaps into the fray, too. It's been kinda quiet around here anyway... |> When I talked to Archie, I understood that he was thinking about using nested |> headers for other things besides the data compression. I don't want to get Well... it really wasn't that specific, but I felt that a general technique for nesting extensions would _allow_ using it for other things - in the best FITS tradition of extensibility. The first name in FITS is "flexible", after all... -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From thompson at stars.gsfc.nasa.gov Mon Jun 22 15:53:10 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2481" "Mon" "22" "June" "1992" "16:07:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "60" "Re: Data structure in FITS (Was: Re: Thoughts from WGAS...)" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: Data structure in FITS (Was: Re: Thoughts from WGAS...) Date: Mon, 22 Jun 1992 16:07:00 GMT In article <1992Jun22.141424.3147 at nsisrv.gsfc.nasa.gov>, warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) writes... >In article <1992Jun19.215656.25564 at nrao.edu>, cflatter at nrao.edu (Chris >Flatters) writes: >|> For example suppose that we have a binary tree of integers. In C >|> we might declare the following struct to hold tree nodes. >|> >|> struct TreeNode >|> { >|> int value; /* Value */ >|> TreeNode *left; /* Pointer to left subtree or null pointer */ >|> TreeNode *right; /* Pointer to right subtree or null pointer */ >|> }; >|> >|> We could store a binary tree of integers in a single FITS binary table >|> with the following header. >|> >|> XTENSION='BINTABLE' >|> BITPIX = 8 / Binary data >|> NAXIS = 2 >|> NAXIS1 = 12 / 3 integers at 4 bytes each >|> NAXIS2 = ? / The number of nodes in the tree >|> PCOUNT = 0 >|> GCOUNT = 1 >|> TFIELDS = 3 >|> EXTNAME = 'BTREE ' / For example >|> EXTVER = 1 >|> TFORM1 = '1J ' >|> TTYPE1 = 'VALUE ' >|> TUNIT1 = ' ' >|> TFORM2 = '1J ' >|> TTYPE2 = 'LEFT SUBTREE ' >|> TUNIT2 = ' ' >|> TFORM3 = '1J ' >|> TTYPE3 = 'RIGHT SUBTREE ' >|> TUNIT3 = ' ' >|> END > >OK, so far I'm with you. Have we instituted some sort of control >authority to handle the values of EXTNAME so applications can recognize >the data structures there? Should we? I believe that this is a misuse of the EXTNAME keyword. My understanding of EXTNAME is that it should identify what data is stored in the table, not the structure used for storing the data. Certainly, that is how I planned to use it for storing SOHO/CDS data: the EXTNAME record would contain the unique identification of the observation, which would then be linked to the master catalog. Binary tables have a tremendous capability of extending the way they can be used. The above is one example. Others may wish to follow the relational model. At the moment there doesn't seem to be any mechanism for codifying these subextensions. At one time, I suggested the additional keyword CONVENTN. Maybe its just too early to start codifying these conventions. There are so many ways that binary tables can be used, that maybe we should wait until we find out how people want to use binary tables. Bill Thompson From cflatter at nrao.edu Mon Jun 22 15:55:28 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2794" "Mon" "22" "June" "1992" "17:57:35" "GMT" "Chris Flatters" "cflatter at nrao.edu " nil "72" "Re: Data structure in FITS (Was: Re: Thoughts" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters) Subject: Re: Data structure in FITS (Was: Re: Thoughts Date: Mon, 22 Jun 1992 17:57:35 GMT In article 22JUN199212071356 at stars.gsfc.nasa.gov, thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes: >In article <1992Jun22.141424.3147 at nsisrv.gsfc.nasa.gov>, >warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) writes... > >>In article <1992Jun19.215656.25564 at nrao.edu>, cflatter at nrao.edu (Chris >>Flatters) writes: >>|> For example suppose that we have a binary tree of integers. In C >>|> we might declare the following struct to hold tree nodes. >>|> >>|> struct TreeNode >>|> { >>|> int value; /* Value */ >>|> TreeNode *left; /* Pointer to left subtree or null pointer */ >>|> TreeNode *right; /* Pointer to right subtree or null pointer */ >>|> }; >>|> >>|> We could store a binary tree of integers in a single FITS binary table >>|> with the following header. >>|> >>|> XTENSION='BINTABLE' >>|> BITPIX = 8 / Binary data >>|> NAXIS = 2 >>|> NAXIS1 = 12 / 3 integers at 4 bytes each >>|> NAXIS2 = ? / The number of nodes in the tree >>|> PCOUNT = 0 >>|> GCOUNT = 1 >>|> TFIELDS = 3 >>|> EXTNAME = 'BTREE ' / For example >>|> EXTVER = 1 >>|> TFORM1 = '1J ' >>|> TTYPE1 = 'VALUE ' >>|> TUNIT1 = ' ' >>|> TFORM2 = '1J ' >>|> TTYPE2 = 'LEFT SUBTREE ' >>|> TUNIT2 = ' ' >>|> TFORM3 = '1J ' >>|> TTYPE3 = 'RIGHT SUBTREE ' >>|> TUNIT3 = ' ' >>|> END >> >>OK, so far I'm with you. Have we instituted some sort of control >>authority to handle the values of EXTNAME so applications can recognize >>the data structures there? Should we? > >I believe that this is a misuse of the EXTNAME keyword. My understanding of >EXTNAME is that it should identify what data is stored in the table, not the >structure used for storing the data. Certainly, that is how I planned to use >it for storing SOHO/CDS data: the EXTNAME record would contain the unique >identification of the observation, which would then be linked to the master >catalog. This is correct. The data structure in a FITS binary table is defined by the TFORM/TTYPE keywords. > >Binary tables have a tremendous capability of extending the way they can be >used. The above is one example. Others may wish to follow the relational >model. The example is consistent with the use of Codd's relational model. In general FITS binary tables are not true R-tables since there is no restriction against duplicated rows. The simplest way to treat an arbitrary FITS binary table as an R-table is to use the row number as a `phantom' column in the R-table. The row number then become the primary key. Chris Flatters cflatter at nrao.edu From forveill at gag.observ-gr.fr Mon Jun 22 16:26:37 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3619" "Mon" "22" "June" "1992" "19:45:05" "GMT" "Thierry Forveille" "forveill at gag.observ-gr.fr " nil "67" "Re: Thoughts from WGAS -- compression and nesting extensions" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: forveill at gag.observ-gr.fr (Thierry Forveille) Subject: Re: Thoughts from WGAS -- compression and nesting extensions Date: Mon, 22 Jun 1992 19:45:05 GMT >Archie Warnock writes >In article <19JUN199218254599 at stars.gsfc.nasa.gov>, thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) writes: >|> One simple way of doing this that I thought of, after reading this, would be to >|> store the header after the compressed data array. The PCOUNT keyword could be >|> used to give the number of bytes in the stored header. I haven't thought >|> everything out yet, but there would have to be keywords signalling that there >|> was or was not a stored header to go with the compressed data, and maybe a >|> keyword signalling where the header started in the parameter area, similar to >|> THEAP for binary tables. One might want to start the stored header on a >|> 2880-byte record boundary. >|> > >I sorta like this idea. Let's think on it a bit and see if anyone else >leaps into the fray, too. It's been kinda quiet around here anyway... I certainly support the idea of only compressing the data, but I don't like the idea of dumping the header after the data. After all a header is supposed to be at the head... More seriously, a usefull feature of FITS is that one can type the file on just about any terminal, look at the header, and ctrl C before binary stuff starts appearing. This become impossible if one has to skip a few megabytes of binary data. Of course I can write a program to skip the data, but then why keep the header as formatted card images? I would thus rather advocate storing the header before the data (of course starting on a record boundary), and using a keyword to signal where the compressed data start. >|> When I talked to Archie, I understood that he was thinking about using nested >|> headers for other things besides the data compression. I don't want to get > >Well... it really wasn't that specific, but I felt that a general >technique for nesting extensions would _allow_ using it for other things >- in the best FITS tradition of extensibility. > >The first name in FITS is "flexible", after all... but the third one is "transport"... Flexibility should not be such that nobody (may be including the originator!) can read back the data. It should thus be a basic rule that extensions are only added if they are needed, and when they are needed. Compression doesn't seem to need nested extensions, so let's wait till something needs it. One point in the compression proposal which is unclear (at least to me) is what is supposed to be compressed: a FITS file or an extension? It looks to me as if the extension is the natural unit for compression, if only because different algorithms will probably be needed for different extensions (one wouldn't want to use previous pixel compression on a ASCII table, which could well be as large as the data). Another point which I would like to see discussed is semi random access to the compressed data. One application I have in mind would involve compressing binary tables which have about 200 rows, each of which has a size of roughly 1 (uncompressed) megabyte. Even though this is obviously a good candidate for compression (there would be a few thousand of them...), I would rather not have to decompress 100 megabytes each time I access one row. One possibility that I have considered is adding an index to the compressed stream before the data, probably as a separate binary table in the same fits file, but I would be interested in a standardized solution if that exists, and would otherwise like to learn about existing solutions to this problem (maybe at STScI for the digitized POSS??) Thierry Forveille Observatoire de Grenoble forveill at gag.observ-gr.fr From dwells at azalea.cx.nrao.edu Mon Jun 22 17:47:31 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1083" "Mon" "22" "June" "1992" "22:46:53" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "24" "WGAS exploder and sci.astro.fits" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory, Charlottesville, VA. Distribution: sci From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: WGAS exploder and sci.astro.fits Date: Mon, 22 Jun 1992 22:46:53 GMT During the past few days there have been a number of postings to both the WGAS exploder and the sci.astro.fits newsgroup (which is also the fitsbits exploder) with titles like: Data structure in FITS (Was: Re: Thoughts from WGAS...) Thoughts from WGAS -- compression and nesting extensions Data structure in FITS (Was: Re: Thoughts Thoughts from WGAS (1) IMAGE extension The initial postings appeared in both places, and then followups and new threads of discussion occurred in both places independently. The purpose of this message is to alert the people who are listening to only one of the two places that important discussions have been happening in the other place. The number of these postings is large enough that I don't think it will be practical to hand-forward the traffic. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From leb at Hypatia.gsfc.nasa.gov Mon Jun 22 21:04:47 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2307" "Mon" "22" "June" "1992" "20:57:46" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "47" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: Re: string arrays Date: Mon, 22 Jun 1992 20:57:46 GMT tody at noao.edu (Doug Tody) writes: >To see why this might be so we can examine two alternative approaches to the >problem of representing arrays of strings. > >1) The fixed length substring approach. In this case we have something >like the following for a "72A12" string array. > > "red green blue yellow turquoise aquamarine" > >2) The delimited string approach. In this case the declaration might be >something like "43A/040" and the stored data might look as follows. > > "red green blue yellow turquoise aquamarine" I really don't see how the second approach is going to save much space because of the restriction that all rows in binary tables must be of the same length. Suppose one row has the colors listed as above but the same field in the following row is "aquamarine turquoise yellow tangerine chartreuse avacado" The field's TFORM must now be "58A/040". In general, the maximum possible width for the field must be reserved and zero-filled when the string is shorter than the maximum. The only alternative is to use the variable array feature, which I think is outside of the discussion at hand. Personally, if the fixed-field way is chosen (I'm currently undecided about *which* is best), rather than having the TFORM for example (1) above be "72A12", I would much prefer "6A12". The interpretation being that "A12" denotes a character object 12 bytes in length and there are an array of 6 of them. I like this much better than the interpretation of 72 objects of type 'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number following the 'A' be a modifier of just the data type, not of the data type and array length. In fact the same notational conventions can be applied across the board to the other data types cleanly, e.g. "16I5" would be interpreted as an array of 16 elements where each element is a 5-tuple of 16-bit integers. This would give FITS at least marginal object orientation. Nothing as useful as ASN.1 or ODL, but you have to start somewhere. -- Lee -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center From pence at heawk1.gsfc.nasa.gov Tue Jun 23 16:48:14 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5348" "Tue" "23" "June" "1992" "17:49:23" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "131" "WGAS talk" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: WGAS talk Date: Tue, 23 Jun 1992 17:49:23 GMT As was requested prior to the meeting, here is a summary of the talk that I gave at the WGAS session of the AAS meeting in Columbus earlier this month. -Bill Pence pence at tetra.gsfc.nasa.gov LHEAVX::PENCE ----------------------------------------------------------------------------- FITS RELATED ACTIVITIES AT THE HIGH ENERGY ASTROPHYSICS SCIENCE ARCHIVE RESEARCH CENTER (HEASARC) AT GODDARD SPACE FLIGHT CENTER I. FITSIO STATUS UPDATE - FITSIO is a Fortran subroutine interface for reading or writing FITS files. - Supports all common types of FITS files: primary arrays, IMAGE extensions, ASCII and Binary table extensions (including variable length arrays) - Currently supported on: VAX/VMS, SUNs, DECstations, IBM mainframes, IBM PCs, Amigas, NeXTs, Silcon Graphics IRIS, Convex - Latest release: V3.20 was released in April 1992; - No new major enhancements are planned, however minor releases will be made periodically to fix bugs and add a few enhancements - Available via anonymous ftp from tetra.gsfc.nasa.gov in directory pub/fitsio3/ - Next release (V3.21) will be available in the last week of June 1992. - Subroutine bindings for C and possibly for SPP are being developed for FITSIO - These consist of thin layers of C or SPP subroutines which sit on top of the Fortran FITSIO subroutines - These binding will allow C or SPP programmers to simply call the FITSIO library without having to worry about the machine specific details of calling Fortran subroutines from another language, or the conversion of data types (e.g., converting SPP strings into Fortran strings). - The C bindings use the CFORTRAN library (available from zebra.desy.de [131.169.2.244]) which provides a machine independent interface for calling Fortran from C. - In the SPP version the low level file I/O routines will also be replaced with calls to the IRAF VOS for complete IRAF compatibility. - the C binding should be available in July and the SSP binding later in July or August. II. FTOOLS - FITS TOOLS PACKAGE DESCRIPTION - FTOOLS is a set of utility programs for manipulating FITS files using FITSIO. - modeled after the STSDAS TTOOLS package - Specifically developed to support the Astro-D mission but most tasks are very general. - FTOOLS will form the core HEASARC software for processing data from any past or future high-energy astronomy mission. - FTOOLS directly processes FITS files without first having to convert them into a local data format. (FITS is not just a transport format). - the software and the data are both machine independent - can simply ftp FITS data files between users on different machines - All input and output files to the FTOOLS tasks are in FITS format (with the occasionally ASCII file as well) . - Each FTOOLS task performs one simple operation, e.g., sort, print, plot, select, or copy information in a FITS file. - Most of the tasks operate on tables (ASCII or Binary) however they also operate on images wherever it make sense to do so. III. FTOOLS DESIGN PHILOSOPHY - The FTOOLS code is designed with portability as a prime requirement. - written in ANSI-standard Fortran or C - all data I/O and user I/O is isolated through standard interface subroutines - data I/O: FITSIO used to access FITS files - user I/O: uses IRAF F77 parameter interface calling sequences - FTOOLS is currently implemented in 2 different environments: - as an IRAF package - uses .par files just like any other IRAF task - as stand-alone tasks using the SAO-developed Parameter Interface which mimics the way IRAF reads parameters from a .par file, but has no dependency on the IRAF software or libraries. - It should be relatively easy to port the code to any other environment. - First public release of FTOOLS is planned for July 1992. Should run on SUNS, and DECstations, and probably VAX/VMS. [watch this space for further announcements when the package is available] - 2.5 programmers and 2 (part time) astronomers are currently working on the FTOOLS project. Work on developing more FTOOLS tasks will continue into 1993. IV. List of Selected FTOOLS Tasks The following is a list of just some of the tasks that are planned for the FTOOLS package. FDUMP - format the header and/or data of a FITS table into an ASCII file FCREATE - create a new FITS table extension from an input ASCII template file FMODHEAD - modify the header keywords in a FITS file FLCOL - list the names and datatypes of all columns in a table FPROJECT - create a new table from columns extracted from an existing table FSELECT - extract selected rows from a table and write to a new table FAPPEND - combine 2 or more separate extensions into one FITS file FEXTRACT - extract an entire FITS extension from one file to a new file FHIST - bin the data in a table column into a histogram F2DHIST - bin the data in 2 table columns into a rectangular FITS image FSTAT - calculate statistics (mean, variance, etc) on a column of numbers FSORT - sort a table based on the entries in a given column FCALC - perform arithmetic operations on a table of numbers PLOTFITS - plot one or more columns in a table extension From dwells at azalea.cx.nrao.edu Fri Jun 26 17:06:16 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["563" "Fri" "26" "June" "1992" "22:05:39" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "12" "File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory, Charlottesville, VA. Distribution: sci From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: File-type extension code? Date: Fri, 26 Jun 1992 22:05:39 GMT Should we agree on a recommended file type extension code for the names of FITS files written on disks? If the answer is "yes", then I suggest that 'fit' would be a good choice (i.e., the recommended practice would be that FITS files should have names of the form *.fit). -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From thompson at stars.gsfc.nasa.gov Fri Jun 26 19:52:47 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1001" "Fri" "26" "June" "1992" "22:29:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "20" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics Distribution: sci From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: File-type extension code? Date: Fri, 26 Jun 1992 22:29:00 GMT In article , dwells at azalea.cx.nrao.edu (Don Wells) writes... >Should we agree on a recommended file type extension code for the >names of FITS files written on disks? > >If the answer is "yes", then I suggest that 'fit' would be a good >choice (i.e., the recommended practice would be that FITS files should >have names of the form *.fit). I personally use '.fits'. Data I get from Big Bear Solar Observatory has the extension '.fts'. Data I get off of tape, I put whatever extension onto the name I want. I guess I don't feel too strongly about it. The only place I can see it being important is when data is ftp'd from some remote location (like my Big Bear example above). Although I haven't done it so far, I could pretty easily write a reader that would try to open a file with the extensions '.fits', '.fit' and '.fts' in turn, or to list all files with those three extensions. Or, if I wanted to, I could simply rename the file. Bill Thompson From leb at Hypatia.gsfc.nasa.gov Fri Jun 26 22:26:38 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1420" "Sat" "27" "June" "1992" "01:15:04" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "27" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center Distribution: sci From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: Re: File-type extension code? Date: Sat, 27 Jun 1992 01:15:04 GMT dwells at azalea.cx.nrao.edu (Don Wells) writes: >Should we agree on a recommended file type extension code for the >names of FITS files written on disks? >If the answer is "yes", then I suggest that 'fit' would be a good >choice (i.e., the recommended practice would be that FITS files should >have names of the form *.fit). I strongly support ".fit" mainly because the CD-ROMs we have produced at the Astronomical Data Center use this convention and we plan to continue with the same practice. Readers should remember that ISO 9660 Level 1 limits filenames to MS-DOS-like 8.3, e.g. FILENAME.EXT as a maximum. I have found that in FTP downloads to MS-DOS machines, files with a ".fits" extension, the local ftp program truncates the extension to ".fit". Given the practice in standards of "sinking to the lowest common denominator", an extension of ".fit" is the most portable. >-- >Donald C. Wells Associate Scientist dwells at nrao.edu >National Radio Astronomy Observatory +1-804-296-0277 >520 Edgemont Road Fax= +1-804-296-0278 >Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center From leb at Hypatia.gsfc.nasa.gov Sat Jun 27 09:13:22 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2632" "" "27" "June" "92" "02:13:06" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "48" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: Goddard Space Flight Center Nntp-Posting-Host: hypatia.gsfc.nasa.gov From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: Re: string arrays Date: 27 Jun 92 02:13:06 GMT leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >Personally, if the fixed-field way is chosen (I'm currently undecided about >*which* is best), rather than having the TFORM for example (1) above be >"72A12", I would much prefer "6A12". The interpretation being that "A12" >denotes a character object 12 bytes in length and there are an array of 6 of >them. I like this much better than the interpretation of 72 objects of type >'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number >following the 'A' be a modifier of just the data type, not of the data type >and array length. > >In fact the same notational conventions can be applied across the board to >the other data types cleanly, e.g. "16I5" would be interpreted as an array of >16 elements where each element is a 5-tuple of 16-bit integers. This would >give FITS at least marginal object orientation. Nothing as useful as ASN.1 or >ODL, but you have to start somewhere. I'm a little surprised that my previous message has yet to elicit a response. The notational convention I mentioned is sufficiently powerful that at least I expected some opposition to it. Therefore I'll press my case.... As a standing member of the NASA technical committee on the NOST FITS standard, I will reserve my vote on the addition of the Binary Tables extension until the matter of this more powerful notation convention is resolved. I am not hard-headed about it, I just would like to see some debate. What are the implications for FITS readers that may have to deal with structures that are of this form? What possible applications can benefit from this notation? From the top of my head, some data objects that are natural for this are coordinates -- they are RA/Dec, Latitude/Longitude, Altitude/Alzimuth pairs. A structure that defines an area could be an array of these two-dimensional values, e.g. '120E2' could describe a set of 120 points in 2-dimensional space that encloses a constellation or some other irregular area of the sky. It can also describe 120 points in a flux/wavelength spectrum. The uses seem endless. Let's talk about it. -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center From tody at noao.edu Sat Jun 27 09:18:23 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5324" "Sat" "27" "June" "1992" "05:13:30" "GMT" "Doug Tody" "tody at noao.edu " nil "86" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Originator: tody at gemini.tuc.noao.edu Nntp-Posting-Host: gemini.tuc.noao.edu Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: tody at noao.edu (Doug Tody) Subject: Re: string arrays Date: Sat, 27 Jun 1992 05:13:30 GMT >From article , by leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman): > leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >>Personally, if the fixed-field way is chosen (I'm currently undecided about >>*which* is best), rather than having the TFORM for example (1) above be >>"72A12", I would much prefer "6A12". The interpretation being that "A12" >>denotes a character object 12 bytes in length and there are an array of 6 of >>them. I like this much better than the interpretation of 72 objects of type >>'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number >>following the 'A' be a modifier of just the data type, not of the data type >>and array length. >> >>In fact the same notational conventions can be applied across the board to >>the other data types cleanly, e.g. "16I5" would be interpreted as an array of >>16 elements where each element is a 5-tuple of 16-bit integers. This would >>give FITS at least marginal object orientation. Nothing as useful as ASN.1 or >>ODL, but you have to start somewhere. > > I'm a little surprised that my previous message has yet to elicit a response. > The notational convention I mentioned is sufficiently powerful that at least I > expected some opposition to it. Therefore I'll press my case.... Ok Lee - as you probably expected, I at least don't like your suggestion. What you suggest seems like just a gee whiz feature to me. There is no case for it, we don't know what it might be useful for. One doesn't just stick features in a specification (especially a standard) because they *might* be useful. We already have a draft binary tables specification out there which took years to get to where it is, and a pretty good case has to be made to make such a significant change. The suggestion proposed by Bill Pence, which I repeated in my earlier posting, e.g., "72A12", has the advantage of being compatible with the current binary table specification. The basic binary tables specification supports only 1 dimensional arrays; in the example 72 is the array length. Your suggestion would be a major change, but all it does is add a restricted 2 dimensional array facility. If we are going to go to the trouble to support arrays of greater than 1 dimension then we should do something more general. The TDIM convention is better, at least it supports arrays of arbitrary dimension. The basic binary tables field type specification, as in "72A", purposely states that characters may follow the type character, and that the meaning of these characters is not defined by the basic binary tables specification. This was done to allow experimentation with things like array specification in the type specifier. For example, it is possible that some sort of syntax can be defined using the type specifier which would eventually prove better than TDIM. In the meantime naive software can treat the stored object as a simple one-dimensional array, or "blob" - this is desirable in any case, as many table access operations do not require knowledge of the contents of a "blob" or array. Bill's original suggestion is consistent with this approach. Your's isn't, and would make it more difficult to reserve space in the type specifier to experiment with layered conventions. I already don't like fixed size arrays for character data; as I have argued previously, even in the case of strings there may be better ways to represent the data. Delimited strings are much more flexible in the range of data which can be represented (e.g. a sequence of lines of text, like this message), and are simple and efficient to use for simple sequences of strings as well ("red green blue" etc.). You don't have to buy this argument, the point is that since it can be debated so easily, we should not lock any one approach into the standard. You are suggesting we not only standarize on an array of fixed length strings for character string storage, but define the same awkward construct for all data types as well. One dimensional arrays are fundamentally different than multidimensional arrays, because any arbitrary data object can be stored in a one dimensional array, and the NxM-etc. multidimensional array is only one such object. An array of strings is another, as is a sequence of delimited strings, or a data structure encoded as a byte stream. There are many types of objects which could be stored in an array. The one dimensional array, or "blob", is a general method for storing objects in a database. It can be treated in a well defined fashion by generic programs without need to understand the nature of the object stored in the array. One dimensional arrays (including variable length arrays in Appendix B) are defined by the standard because they are fundamentally important and reasonably well understood. As soon as we start looking at more complex objects we get into discussions like this one - that is why the current binary tables specification stops where it does. Experimentation with layered conventions is required to learn how to store more complex objects. Doug Tody -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From dbriggs at zia.aoc.nrao.edu Sat Jun 27 09:55:48 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1767" "" "27" "June" "92" "02:01:20" "GMT" "Daniel Briggs" "dbriggs at zia.aoc.nrao.edu " nil "32" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Distribution: sci Organization: National Radio Astronomy Observatory, Socorro NM From: dbriggs at zia.aoc.nrao.edu (Daniel Briggs) Subject: Re: File-type extension code? Date: 27 Jun 92 02:01:20 GMT In article dwells at azalea.cx.nrao.edu (Don Wells) writes: >Should we agree on a recommended file type extension code for the >names of FITS files written on disks? > >If the answer is "yes", then I suggest that 'fit' would be a good >choice (i.e., the recommended practice would be that FITS files should >have names of the form *.fit). It's probably a good idea, in the cases where the data format is made explicit in an extension. The local R&D imaging package, SDE, has been using disk fits as the default data storage format for several years. It knows to recognize *.FTS as a fits file and *.SDE as a machine dependent memory image. Neither of the above is the default format, which is again FITS. I suspect that I'm not alone in often encoding quite a lot of information in the filename, like 3C84-X-BIG-CLN.XFR.20 An additional recommended or manditory field on the end of every such name quickly becomes a nuisance. (Not that such things are easy to type in the first place!) However, in the case where a FITS file is stored as *., I vote for the characters 'FTS'. Try to say it, and you'll see that it is really much more mnemonic than 'FIT'. Sad to say, we should probably stick to 3 character extensions, for the usual lowest common denominator reasons. Would we be specifying a case for the characters? If we have to pick one or the other, 'FTS' is probably a better move than 'fts'. -- | Daniel Briggs (dbriggs at nrao.edu) | USPA B-14993 | New Mexico Tech / National Radio Astronomy Observatory | DoD #387 | P.O. Box O / Socorro, NM 87801 (505) 835-7391 | Support the League for Programming Freedom (info from league at prep.ai.mit.edu) From mcs at goblin.caltech.edu Sat Jun 27 09:56:15 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["314" "Sat" "27" "June" "1992" "03:42:59" "GMT" "Martin Shepherd" "mcs at goblin.caltech.edu " nil "9" "VLBA UV-data FITS test files" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: goblin.caltech.edu Organization: California Institute of Technology From: mcs at goblin.caltech.edu (Martin Shepherd) Subject: VLBA UV-data FITS test files Date: Sat, 27 Jun 1992 03:42:59 GMT I have just been reading the draft proposal for the new FITS format for interferometer UV data exchange. Does anybody know if there are any FITS test files in this format - available by anonymous ftp? There didn't seem to be anything relevant at fits.cx.nra.edu. Thanks. Martin Shepherd (mcs at phobos.caltech.edu) From dwells at azalea.cx.nrao.edu Sat Jun 27 10:13:51 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1067" "Sat" "27" "June" "1992" "15:11:28" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "20" "Re: VLBA UV-data FITS test files" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits In-Reply-To: mcs at goblin.caltech.edu's message of Sat, 27 Jun 1992 03: 42:59 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA. From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: Re: VLBA UV-data FITS test files Date: Sat, 27 Jun 1992 15:11:28 GMT In article <1992Jun27.034259.18877 at cco.caltech.edu> mcs at goblin.caltech.edu (Martin Shepherd) writes: ".. [re] draft proposal for.. new FITS format for interferometer UV data exchange.. [are there] any FITS test files in this format..?..." The VLBA Correlator produces this format on disk in real-time. The test files are readable by current "TEST" versions of AIPS. Perhaps the Correlator team will be able to select some existing test data and make it available in the archive; I will check into that on Monday. Probably we won't be able to make any more test files for at least a month -- the disassembly of the actual Correlator began on Friday, and half of it is already in the hallway in wooden crates. The whole complex will be shipped to New Mexico 13 days from now... -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From warnock at Hypatia.gsfc.nasa.gov Sat Jun 27 16:03:54 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["830" "Sat" "27" "June" "1992" "18:18:58" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "19" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Re: string arrays Date: Sat, 27 Jun 1992 18:18:58 GMT leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: > I'm a little surprised that my previous message has yet to elicit a response. >The notational convention I mentioned is sufficiently powerful that at least I >expected some opposition to it. Therefore I'll press my case.... Maybe it was so intuitively obvious that no one thought it wasn't part of the proposal. > Let's talk about it. Other than reservations about it being too "Fortran-like", it's an excellent addition. I'm in favor of it. I'm not even biased just because you're across the hall from me... -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From warnock at Hypatia.gsfc.nasa.gov Sat Jun 27 16:04:40 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["704" "Sat" "27" "June" "1992" "18:17:35" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "17" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center Distribution: sci From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Re: File-type extension code? Date: Sat, 27 Jun 1992 18:17:35 GMT dwells at azalea.cx.nrao.edu (Don Wells) writes: >Should we agree on a recommended file type extension code for the >names of FITS files written on disks? A resounding "yes". >If the answer is "yes", then I suggest that 'fit' would be a good >choice (i.e., the recommended practice would be that FITS files should >have names of the form *.fit). Agreed - it's preferable to 'fts' for reasons of truncation, as Lee points out. Let's go for it... -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From thompson at stars.gsfc.nasa.gov Sat Jun 27 20:57:58 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4548" "" "27" "June" "92" "23:51:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "82" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: string arrays Date: 27 Jun 92 23:51:00 GMT In article , leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes... >leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >>Personally, if the fixed-field way is chosen (I'm currently undecided about >>*which* is best), rather than having the TFORM for example (1) above be >>"72A12", I would much prefer "6A12". The interpretation being that "A12" >>denotes a character object 12 bytes in length and there are an array of 6 of >>them. I like this much better than the interpretation of 72 objects of type >>'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number >>following the 'A' be a modifier of just the data type, not of the data type >>and array length. >> >>In fact the same notational conventions can be applied across the board to >>the other data types cleanly, e.g. "16I5" would be interpreted as an array of >>16 elements where each element is a 5-tuple of 16-bit integers. This would >>give FITS at least marginal object orientation. Nothing as useful as ASN.1 or >>ODL, but you have to start somewhere. > > I'm a little surprised that my previous message has yet to elicit a response. >The notational convention I mentioned is sufficiently powerful that at least I >expected some opposition to it. Therefore I'll press my case.... > > As a standing member of the NASA technical committee on the NOST FITS >standard, I will reserve my vote on the addition of the Binary Tables >extension until the matter of this more powerful notation convention is >resolved. > > I am not hard-headed about it, I just would like to see some debate. What >are the implications for FITS readers that may have to deal with structures >that are of this form? What possible applications can benefit from this >notation? From the top of my head, some data objects that are natural for this >are coordinates -- they are RA/Dec, Latitude/Longitude, Altitude/Alzimuth >pairs. A structure that defines an area could be an array of these >two-dimensional values, e.g. '120E2' could describe a set of 120 points in >2-dimensional space that encloses a constellation or some other irregular area >of the sky. It can also describe 120 points in a flux/wavelength spectrum. >The uses seem endless. > > Let's talk about it. All right. First of all, let's consider 72A12 vs 6A12 for a series of six character strings of 12 characters each. Although I agree that the latter is more elegant, the former preserves the property that the first number and the character code gives the number of bytes taken up by that particular column. One could still theoretically read the file even if one's reader did not understand the part following the datatype code character. As for the rest, I believe that the proposal is so restrictive as to be almost useless. The effect of this proposal is to allow for two-dimensional arrays, although you describe it somewhat differently. I believe that N-dimension capabilities are needed, such as the TDIM or Green Bank conventions provide for. (Some astronomical data I've worked with have has as much as six dimensions.) Character string arrays are something of an exception, since the number of characters in each string is a property quite distinct from the dimensions that the characters may be organized into. It has been quite correctly pointed out that there are at least two ways of delimiting character strings, by a preset number, or by a delimiter character. One also has to consider both fixed and variable length arrays. At first I was receptive to the 'rAw' proposal. However, I now feel that it would be a mistake to expect to be able to store this information in the TFORM keyword, encompassing all reasonable possibilities. I believe that the best approach is to leave the current binary table proposal untouched, and to work on enhancing its capabilities through the addition of keywords such as TDIM. For instance: TCLENnnn Character string length for column "nnn" TCDELnnn Character string delimiter (byte value) for column "nnn" (The above are intended more as examples then as serious proposals, although they could be.) On a related issue, an earlier post used the notation "\012" to denote an octal value. While this is standard C and UNIX usage, I believe that its use in FITS files would be new and unprecedented. I don't necessary say that it's a bad thing. A standard for expressing values in octal (or even hexadecimal) notation would be valuable. It would have to be carefully looked at, though. Bill Thompson From thompson at stars.gsfc.nasa.gov Sat Jun 27 20:58:48 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1794" "Sat" "27" "June" "1992" "23:56:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "33" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics Distribution: sci From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: File-type extension code? Date: Sat, 27 Jun 1992 23:56:00 GMT In article <1992Jun27.020120.670 at zia.aoc.nrao.edu>, dbriggs at zia.aoc.nrao.edu (Daniel Briggs) writes... >In article > dwells at azalea.cx.nrao.edu (Don Wells) writes: >>Should we agree on a recommended file type extension code for the >>names of FITS files written on disks? >> >>If the answer is "yes", then I suggest that 'fit' would be a good >>choice (i.e., the recommended practice would be that FITS files should >>have names of the form *.fit). > >It's probably a good idea, in the cases where the data format is made >explicit in an extension. The local R&D imaging package, SDE, has been >using disk fits as the default data storage format for several years. >It knows to recognize *.FTS as a fits file and *.SDE as a machine dependent >memory image. Neither of the above is the default format, which is again >FITS. I suspect that I'm not alone in often encoding quite a lot of >information in the filename, like 3C84-X-BIG-CLN.XFR.20 An additional >recommended or manditory field on the end of every such name quickly >becomes a nuisance. (Not that such things are easy to type in the first >place!) > >However, in the case where a FITS file is stored as *., I >vote for the characters 'FTS'. Try to say it, and you'll see that it is >really much more mnemonic than 'FIT'. Sad to say, we should probably stick >to 3 character extensions, for the usual lowest common denominator reasons. >Would we be specifying a case for the characters? If we have to pick one >or the other, 'FTS' is probably a better move than 'fts'. When I ftp a file from a VAX machine (which is case insensitive) to a UNIX machine, the filename always comes over as lowercase. For that reason '.fts' is much better than '.FTS'. Bill Thompson From lucio at ifctr.mi.cnr.it Mon Jun 29 08:25:13 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3156" "Mon" "29" "June" "1992" "15:47:02" "GMT" "Lucio Chiappetti - IFCTR Milano" "lucio at ifctr.mi.cnr.it" nil "62" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: Lucio Chiappetti - IFCTR Milano Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 15:47:02 GMT >Donald C. Wells Associate Scientist dwells at nrao.edu writes : >Should we agree on a recommended file type extension code for the ----------- >names of FITS files written on disks? > >If the answer is "yes", then I suggest that 'fit' would be a good >choice (i.e., the recommended practice would be that FITS files should ----------- >have names of the form *.fit). My *personal* inclination would favour '.fits' more than '.fit' (in some context one might think this is reserved or used by some fitting programs), however I appreciate the issue of "minimum compatibility" which may indi- cate usage of 3-char extensions (BUT do we really think realistic to use FITS files on PCs ? Are there any other systems nowadays requiring this limitation ?). In general I like "verbose" filetypes ('image' and not 'img', 'spectrum' and not 'spx', 'rate' and not 'rat' etc.) However I believe there is one more important issue, which makes me underline the word "recommended" above. That is, will the major packages (IRAF, MIDAS, IDL ... should add AIPS, but we do not use it at this site, sorry Don) comply with the recom- mendation ? One of the major uses of keeping FITS data on disk should be to store them in a machine and package independent format (suppose you want to interchange data between IRAF and MIDAS, or between a Sun and a DECstation sharing NFS disks). Of course another one could be to retrieve stuff from ftp archives, in which case a self-documenting extension certainly helps. Coming back to the major package issue, at the moment they are still excessively linked to the mag tape model (this does NOT imply that I favour using FITS other than a 'T'ransport format,I am just talking of the names) : e.g. IRAF (unless one does it file by file) saves FITS files on disk as xxxx001 xxxx002 xxxx003 (with a 3-figure ordinal and NO extension) MIDAS instead uses xxxx0001.mt xxxx0002.mt etc. (with a 4-figure ordinal and .mt extension) IF they would save their files with the same name they have in the native format (e.g. pinco.imh pinco.pix pinco.bdf etc.) and a standard FITS filetype (pinco.fits or pinco.fit or pinco.fts) that would save the user to do the renaming all the times (or similar workarounds) I believe that, until the major packages will support the recommendation, it will remain just that. In this respect I have no strong preference for any of the proposed names. ----------------------------------------------------------------------------- A member of G.ASS : Group for Astronomical Software Support ----------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR Milano | U N I C U I Q U E via Bassini 15 - I-20133 Milano - Italy | System Manager Internet: LUCIO at IFCTR.MI.CNR.IT | FORTUNAE SUAE Decnet: IFCTR::LUCIO (39610::LUCIO) | Bitnet: EXOSAT at IMISIAM | ----------------------------------------------------------------------------- Acknowledge-To: From dwells at azalea.cx.nrao.edu Mon Jun 29 08:40:44 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1746" "Mon" "29" "June" "1992" "13:39:07" "GMT" "Don Wells" "dwells at azalea.cx.nrao.edu " nil "34" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits In-Reply-To: Lucio Chiappetti - IFCTR Milano's message of Mon, 29 Jun 1992 15: 47:02 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA. From: dwells at azalea.cx.nrao.edu (Don Wells) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 13:39:07 GMT In article <9206290904.AA06192 at fits> Lucio Chiappetti - IFCTR Milano writes: LC> "... My *personal* inclination would favour '.fits' more than LC> '.fit' (in some context one might think this is reserved or used LC> by some fitting programs), however I appreciate the issue of LC> "minimum compatibility" which may indi- cate usage of 3-char LC> extensions (BUT do we really think realistic to use FITS files on LC> PCs ? Are there any other systems nowadays requiring this LC> limitation ?). FITS files are used on PCs by many people. Also, CD-ROMs also use the DOS caseless 8+3 naming convention. LC> "... will the major packages (IRAF, MIDAS, IDL ... AIPS.. comply LC> with the recom- mendation ?".. LC> IF they would save their files with the same name they have in LC> the native format (e.g. pinco.imh pinco.pix pinco.bdf etc.) and a LC> standard FITS filetype (pinco.fits or pinco.fit or pinco.fts) LC> that would save the user to do the renaming all the times (or LC> similar workarounds) This is exactly my motivation for asking the question. I noticed in a recent newsletter that the ST-SDAS team have added support for FITS as an *internal* (in addition to interchange) disk format, with names *.fit. I expect that this will start a trend. It would be convenient if the various packages could share an internal file format as well as an interchange format; a naming convention would facilitate this. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From ahenden at magnus.acs.ohio-state.edu Mon Jun 29 10:21:34 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1248" "Mon" "29" "June" "1992" "13:21:39" "GMT" "Arne A Henden" "ahenden at magnus.acs.ohio-state.edu " nil "21" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: top.magnus.acs.ohio-state.edu Organization: The Ohio State University From: ahenden at magnus.acs.ohio-state.edu (Arne A Henden) Subject: Re: string arrays Date: Mon, 29 Jun 1992 13:21:39 GMT >leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >>Personally, if the fixed-field way is chosen (I'm currently undecided about >>*which* is best), rather than having the TFORM for example (1) above be >>"72A12", I would much prefer "6A12". The interpretation being that "A12" >>denotes a character object 12 bytes in length and there are an array of 6 of >>them. I like this much better than the interpretation of 72 objects of type >>'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number >>following the 'A' be a modifier of just the data type, not of the data type >>and array length. > > I'm a little surprised that my previous message has yet to elicit a response. >The notational convention I mentioned is sufficiently powerful that at least I >expected some opposition to it. Therefore I'll press my case.... > I agree with Doug in this case. TDIM for fixed-size items plus the variable length array extension handles all the cases you've mentioned. The major benefit of the simplistic element count for TFORM is that a simpler file reader can be written that ignores columns. If you are really concerned about a Fortran-like interface, why not something like 72A(6A12), which fits the proposed format. From ahenden at magnus.acs.ohio-state.edu Mon Jun 29 10:22:14 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1097" "Mon" "29" "June" "1992" "13:34:41" "GMT" "Arne A Henden" "ahenden at magnus.acs.ohio-state.edu " nil "25" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: top.magnus.acs.ohio-state.edu Organization: The Ohio State University From: ahenden at magnus.acs.ohio-state.edu (Arne A Henden) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 13:34:41 GMT > warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) writes: >>dwells at azalea.cx.nrao.edu (Don Wells) writes: >>Should we agree on a recommended file type extension code for the >>names of FITS files written on disks? > >A resounding "yes". > >>If the answer is "yes", then I suggest that 'fit' would be a good >>choice (i.e., the recommended practice would be that FITS files should >>have names of the form *.fit). > >Agreed - it's preferable to 'fts' for reasons of truncation, as Lee >points out. Let's go for it... >-- For those cases where the file type must be specified, I'd prefer .fits for those systems that can handle it and .fit for the lowest common denominator (and lower case for Bill's reason). The 8x3 format of PCs and ISO9660 is really limiting in this day and age, and quite often my files end up looking like 891206.005 for raw data, but stored in fits format. Therefore, I'd stess that the .fit/.fits extension is a *recommended* extension only and not a required one. You can always build a reader that has a set of default file extensions in use by your organization. From ewing-martin at CS.YALE.EDU Mon Jun 29 11:16:47 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["452" "Mon" "29" "June" "1992" "14:18:40" "GMT" "Martin Ewing" "ewing-martin at CS.YALE.EDU " nil "12" "FITS -> Macintosh?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: minerva.cis.yale.edu Organization: Yale Univ. Sci & Eng Comp Facility, New Haven CT 06520 From: ewing-martin at CS.YALE.EDU (Martin Ewing) Subject: FITS -> Macintosh? Date: Mon, 29 Jun 1992 14:18:40 GMT [Obligatory header: Sorry if this is an FAQ, but...] What can you recommend for handling FITS data on a Macintosh? Are there public domain programs that read FITS on the Mac? (I would like to use a system like NCSA Image.) A FITS to HDF conversion would be a possibility. FITS to GIF or TIFF conversion (Unix or Mac based) would also be of interest. Thanks for any clues or pointers. Martin Ewing (ewing at yale.edu) "Omnia Vincit Celeritas" From thompson at stars.gsfc.nasa.gov Mon Jun 29 12:34:48 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1364" "Mon" "29" "June" "1992" "14:38:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "27" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 14:38:00 GMT In article <9206290904.AA06192 at fits>, lucio at ifctr.mi.cnr.it (Lucio Chiappetti - IFCTR Milano) writes... > Coming back to the major package issue, at the moment they are still > excessively linked to the mag tape model (this does NOT imply that I > favour using FITS other than a 'T'ransport format,I am just talking of > the names) : > > e.g. IRAF (unless one does it file by file) saves FITS files on disk > as xxxx001 xxxx002 xxxx003 (with a 3-figure ordinal and NO extension) > MIDAS instead uses xxxx0001.mt xxxx0002.mt etc. (with a 4-figure > ordinal and .mt extension) > > IF they would save their files with the same name they have in the native > format (e.g. pinco.imh pinco.pix pinco.bdf etc.) and a standard FITS > filetype (pinco.fits or pinco.fit or pinco.fts) that would save the user > to do the renaming all the times (or similar workarounds) I would like to point out that IDL does do exactly this. The FITS tape reader we have will allow the user to specify a keyword which contains the filename. Then, for each file it reads, it will check the header for this keyword. It used to be that the IDL FITS tape reader that we had converted the file to SDAS format when it wrote it to disk. However, we now have a reader which leaves it a FITS file--and it automatically puts on the extension '.fits'. Bill Thompson From zarate at scivax.stsci.edu Mon Jun 29 12:36:55 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1310" "Mon" "29" "June" "1992" "16:33:57" "GMT" "zarate at scivax.stsci.edu" "zarate at scivax.stsci.edu" nil "33" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: Space Telescope Science Institute Distribution: na From: zarate at scivax.stsci.edu Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 16:33:57 GMT In article , dwells at azalea.cx.nrao.edu (Don Wells) writes... > Should we agree on a recommended file type extension code for the > names of FITS files written on disks? > > If the answer is "yes", then I suggest that 'fit' would be a good > choice (i.e., the recommended practice would be that FITS files should > have names of the form *.fit). I have developed a library under IRAF to access FITS files by any IRAF tasks and so far the default file type extension is ".fit". So I cannot agree more with this recommended practice for FITS on disk. But I must say that having one extension is somewhat restrictive and sometimes impossible to applied. For example here at the Institute we generate data sets with filenames that differ only on the extension (3 characters); for example "z0ad010ar.c3h". One alternative in replacing "c3h" for "fit" is to apppend it to the original name, like "z0ad010ar.c3h.fit". To avoid the previous unwanted filename; a more general FITS file type extension is to have "??f" were "?" is any character. I would like to agree on a minimum number of possible FITS extension that any FITS reader can handle as defaults. Nelson Zarate (zarate at stsci.edu) Space Telescope Science Institute Baltimore, Maryland 21218 (410) 516-8697 From pence at heawk1.gsfc.nasa.gov Mon Jun 29 13:40:40 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["524" "Mon" "29" "June" "1992" "15:47:19" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "10" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 15:47:19 GMT Having read the various reponses to the suggested file type extension code, I would vote for using .fits, (truncated to .fit on machines that only support 3 letter extensions). I fully agree with one of the posters that this should only be a recommendation, and not a requirement. We have recently issued many fits files on CDROM with the 8.3 file name restriction, where we had to use the extension characters to uniquely identify the file; it was too wasteful of filename space to name all the files .fit. -Bill Pence From lucio at ifctr.mi.cnr.it Mon Jun 29 13:41:15 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1329" "Mon" "29" "June" "1992" "23:48:25" "GMT" "Lucio Chiappetti - IFCTR Milano" "lucio at ifctr.mi.cnr.it" nil "25" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Organization: National Radio Astronomy Observatory From: Lucio Chiappetti - IFCTR Milano Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 23:48:25 GMT >Nelson Zarate (zarate at stsci.edu) writes > >But I must say that having one extension is somewhat restrictive and >sometimes impossible to applied. For example here at the Institute we >generate data sets with filenames that differ only on the extension >(3 characters); for example "z0ad010ar.c3h". One alternative in replacing >"c3h" for "fit" is to apppend it to the original name, like >"z0ad010ar.c3h.fit". Names containing more than one dot in the full filename should be avoided. Any non-Unix operating system will complain. One possibility (ussed quite frequently by various pieces of s/w) is to replace the dots with underscores (but the last, new one), e.g. z0ad010ar_c3h.fit ----------------------------------------------------------------------------- A member of G.ASS : Group for Astronomical Software Support ----------------------------------------------------------------------------- Lucio Chiappetti - IFCTR/CNR Milano | U N I C U I Q U E via Bassini 15 - I-20133 Milano - Italy | System Manager Internet: LUCIO at IFCTR.MI.CNR.IT | FORTUNAE SUAE Decnet: IFCTR::LUCIO (39610::LUCIO) | Bitnet: EXOSAT at IMISIAM | ----------------------------------------------------------------------------- Acknowledge-To: From tody at noao.edu Mon Jun 29 15:06:03 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2701" "Mon" "29" "June" "1992" "18:26:42" "GMT" "Doug Tody NOAO/IRAF" "tody at noao.edu " nil "45" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Originator: tody at gemini.tuc.noao.edu Nntp-Posting-Host: gemini.tuc.noao.edu Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: tody at noao.edu (Doug Tody NOAO/IRAF) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 18:26:42 GMT As Nelson, Don and others have mentioned we are adding a FITS image kernel to IRAF - this will allow FITS files on disk or CD-ROM to be directly accessed with IRAF image tasks. One of the motivations for this image kernel will be to make it easier to share images between different analysis systems (IRAF and AIPS, IDL, MIDAS etc.). It will also be nice to be able to directly browse FITS archives such as CD-ROMs, without having to convert the image to a different format. I agree that, particularly for data sharing, it would be nice to be able to agree upon a standard file extension. Reviewing the situation with IRAF I find that we have been using ".fits". Files with the extension ".fits" can be included in IRAF directory trees and will be treated as exportable, machine independent files by IRAF programs like rmbin, wtar, "mkpkg " and so on. The current IRAF image kernel interface (IKI) supports file extensions of only three characters or less, which is probably why Nelson used ".fit" in his development version of the FITS image kernel. This three character limit can easily be changed however, it is internal to the interface and the limit exists only because of the set of supported disk image formats in existence at the time IKI was initially developed. We can change this when the FITS image kernel is integrated into IRAF. As Bill Pence mentions CD-ROMs have been written which contain FITS files with no file extension, due to limitations on file names imposed by the ISO 9660 standard, which is DOS oriented (recently I heard of something called the Rock Ridge extensions which addresses this problem, but I think it may only be for CD-ROMS meant to be read on UNIX or other non-DOS systems). If a file extension were used on such a CD-ROM it would have to be limited to three characters. It appears that we cannot count on the file extension to specify the file type if we are trying to read data written externally. Given our interest in archival data and CD-ROMS I think there is some justification for the ".fit" extension, however I suspect that the DOS and CD-ROM problem is going to cause problems no matter what we do. For data sharing using FITS-on-disk the most natural file extension is probably ".fits", and this is what IRAF currently supports. I could live with either choice. I don't like ".fts" (sounds like fourier transform spectrometer data to me) and I don't like "??f" ("??h" as for SOGS data was bad enough). Doug Tody -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From sla at fast.ucsc.edu Mon Jun 29 15:29:18 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1027" "" "29" "June" "92" "18:50:20" "GMT" "Steve Allen" "sla at fast.ucsc.edu " nil "15" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Summary: .FITS or .fits, truncated or ignored where necessary Organization: UCO/Lick Observatory From: sla at fast.ucsc.edu (Steve Allen) Subject: Re: File-type extension code? Date: 29 Jun 92 18:50:20 GMT In article pence at heawk1.gsfc.nasa.gov ( William Pence) writes: >I would vote for using .fits, (truncated to .fit on machines that only >support 3 letter extensions). I fully agree with one of the posters that this should >only be a recommendation, and not a requirement. I concur with the suggestion that ".fits" should be the *recommended* default. However, given the differences between various filesystems and the severe constraints imposed by some, this will have to stand as a recommendation. I do not see this as issue as being something to get dogmatic about, and if given the chance to vote, I would vote against the adoption of any particular type/extension as a requirement. _______________________________________________________________________________ Steve Allen | That was the equation! | sla at helios.ucsc.edu UCO/Lick Observatory | Existence!...Survival must | If the UC were opining, Santa Cruz, CA 95064 | cancel out programming! -- Ruk | it wouldn't tell me. From cflatter at nrao.edu Mon Jun 29 16:21:00 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["365" "Mon" "29" "June" "1992" "20:21:36" "GMT" "Chris Flatters" "cflatter at nrao.edu " nil "11" "Re: File-type extension code?" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters) Subject: Re: File-type extension code? Date: Mon, 29 Jun 1992 20:21:36 GMT Since it is possible to determine whether a file its a FITS file or not by checking for the magic string "SIMPLE = T" at the beginning of a file, I don't see any necessity to mandate or to recommend the use of any particular extension for FITS files on disk. Why not leave the choice of file name up to the user? Chris Flatters cflatter at nrao.edu From pence at heawk1.gsfc.nasa.gov Mon Jun 29 16:28:29 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["7214" "Mon" "29" "June" "1992" "19:28:25" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " nil "123" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Re: string arrays Date: Mon, 29 Jun 1992 19:28:25 GMT leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >Personally, if the fixed-field way is chosen (I'm currently undecided about >*which* is best), rather than having the TFORM for example (1) above be >"72A12", I would much prefer "6A12". The interpretation being that "A12" >denotes a character object 12 bytes in length and there are an array of 6 of >them. I like this much better than the interpretation of 72 objects of type >'A' of which 12 are "subgrouped", as it were. It's cleaner to have the number >following the 'A' be a modifier of just the data type, not of the data type >and array length. I had originally proposed using exactly this convention (e.g., "6A12") last year in my comments about the bintable format and have even implemented it in the current FITSIO package. However, since it appears that this suggestion is about to be shot down (the Europeans apparently favor sticking with the original wording of the bintable format) I proposed the new way of representing fixed length strings in the tform keyword (e.g., "72A12"). Since this new proposal is entirely consistent with the proposed binary table format, people cannot object to it for that reason (although they may have other objections) and it should be easier to gain acceptance for the proposal. As the implementor of FITSIO, I really don't care which form ("6A12" or "72A12") is chosen, since they are equally easy to implement. All I ask is that something like this be adopted so that we don't have to go through some other clumsy procedure (like looking for TDIMn keywords) just to support a simple one dimensional array of strings. The TFORMn keyword is all that is needed to support one dimensional arrays of every other possible datatype (logical, bit, byte, short integer, long integer, reals, double precision, complex, double complex) so why can't one dimension arrays of stings be supported as well?? The answer is that they can be supported by using the simple syntax that I have proposed. As Doug Tody wrote: "One dimensional arrays are fundamentally different than multidimensional arrays, because any arbitrary data object can be stored in a one dimensional array... [stuff deleted]... The one dimensional array, or "blob", is a general method for storing objects in a database. It can be treated in a well defined fashion by generic programs without need to understand the nature of the object stored in the array." I agree with this and am simply trying to find a way within the proposed bintable format to support one dimensional arrays of strings. Note that I also have no objections to using the TDIMn keyword to express the higher dimensionality of arrays of ANY datatype. I just object to requiring that the STRING datatype be the ONLY datatype which requires the use of the TDIM keyword to express one-dimensional arrays. This objection is based on a very practical consideration: the FITSIO package only deals with binary table columns as one-dimensional arrays. Thus, except for string arrays, the FITSIO code does not have to worry about reading or writing TDIM keywords; It is left up to the higher level calling program to worry about any higher dimensionality, if it is appropriate. Doug Tody has objected to my proposal on the grounds that there may be other ways to express arrays of strings rather than just as fixed length arrays (e.g., null terminated strings). To this I would reply that my proposal in no way excludes alternative ways of encoding arrays of strings (this is explicitly stated at the end of my proposal). All I am proposing is that IF someone wants to write an array of fixed length strings into a column, then he/she MAY use the proposed 'rAw' TFORM format to express this. Just as an example of alternative ways of writting strings, one could write TFORMn = '72A/000' to signify a 72 character field containing strings terminated by nulls (octal 000). I do not advocate using delimited strings withing FITS files because I think it could be a nightmare to implement, but still, my proposal does not preclude such conventions from being used. As a final comment, I don't think that using delimited strings is a very good idea in FITS files. First, one must allocate a fixed width in the binary table column anyway, which seriously restricts the usefulness of varable length strings (delimited strings) within the field. The only advantage to using delimited strings is that one can use some of the unused space from one string array element and use it for a longer element somewhere else in the array. But practically speaking I think one would end up wasting more space in the table due to having to have delimiter characters between each string element, than if one just had fixed length strings in the first place. The really major disadvantage that I see to using delimited strings in FITS files is that one has no idea how many elements of the string array are in a given row of an ASCII column making it impossible to directly offset to any particular element within the string array (unlike every other datatype which has a fixed element size). For example, if someone calls one of my FITSIO subroutines to write an array of 20 strings into a TFORM='36A' column using delimited strings then FITSIO won't know how many strings should be packed into each row of the table. Should it try to force all 20 stings into one 36 character row? Or at the other extreme, should FITSIO write one string into 20 consecutive rows of the table?? If, on the other hand we use fixed length strings then the meaning is very clear. If for example TFORM = '36A18' then FITSIO would know that the 20 strings should be written as 2 18-character strings on 10 consecutive lines of the table. leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) also writes: >In fact the same notational conventions can be applied across the board to >the other data types cleanly, e.g. "16I5" would be interpreted as an array of >16 elements where each element is a 5-tuple of 16-bit integers. This would >give FITS at least marginal object orientation. Nothing as useful as ASN.1 or >ODL, but you have to start somewhere. I agree with Doug here in that generalizing this TFORM convention to other datatypes offers too little new functionality to be of much general use. This basically is just a way to specify 2-dimensional arrays, but I think is would be better to have a single general way to specify arrays of any number of dimensions (e.g., the TDIMn keyword). (Some might argue that following this same logic, one should consider an arrays of fixed length strings as a 2 dimensional array of characters, and hence one should stick with the TDIM convention for specifying the string length. I don't buy this arguement however because I regard a string as a fundamental datatype in its own right, quite different from an array of characters. In the real world people want to be able to deal with a string as a single entity. The fact that the string can be broken down into an array of characters is no more relevant than the fact that an integer can be broken down into an array of bits.) -Bill Pence pence at tetra.gsfc.nasa.gov LHEAVX::PENCE From thompson at stars.gsfc.nasa.gov Mon Jun 29 21:41:07 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2266" "Mon" "29" "June" "1992" "21:34:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "42" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: string arrays Date: Mon, 29 Jun 1992 21:34:00 GMT In article , pence at heawk1.gsfc.nasa.gov ( William Pence) writes... (message deleted to save space) I don't really see anything wrong with William Pence's 'rAw' proposal, as such. I agree that the length of a string is fundamentally different than the dimension of an array. The trouble starts when trying to extend this idea to other ways of storing string arrays. One can either store fixed-length strings, or delimited strings. It's been suggested that the latter case be identified by "rA/ooo" where "ooo" is the octal representation of the delimiter character. However, what would you do about character strings stored as variable length arrays? I also agree that TDIMn, while a valuable resource, should not be necessary if all one wants to do is specify where one string ends, and the next begins. That's why I suggested that the separate keyword TCLENn, or something like it, be used to specify the number of characters in each string. The beauty of this approach is that it works for both fixed and variable length arrays. Alternatively, one could use the keyword TCDELn, which takes an integer value between 0 and 255, to represent the character that delimits each string. The idea of having "A" followed by a number whose meaning changes if it is given in decimal or octal notation bothers me. I agree that it can be properly interpreted by the computer, but I disagree with it on philosophical grounds. If we use it this way now, we severely restict the use of octal and hexadecimal constants in the future. Character strings are fundamentally different from other data types, and there's probably no completely satisfying way around that fact. Bill Thompson P.S. A separate issue comes up when one does want to use the TDIM keyword, or some other way of specifying dimensions, together with either Pence's 'rAw' or my TCLEN proposal. Does one include the string length in both TDIM and TFORM, which would be consistent with the way that Appendix A of the Binary Tables Extension proposal currently reads, or would one only put the regular dimensions in TDIM, which would be more consistent with allowing both fixed character length and character delimiter possibilities. From leb at Hypatia.gsfc.nasa.gov Tue Jun 30 13:01:21 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1582" "Tue" "30" "June" "1992" "16:01:59" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "26" "string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: string arrays Date: Tue, 30 Jun 1992 16:01:59 GMT I guess I should have quoted more of my original message. My original intent in proposing a general convention for a suffix was to introduce something approaching an "object" into FITS. The objections like "that's just a 2-d array" strengthen my feelings that the FITS implementors are too absorbed in "bit" descriptions and not "data" descriptions. It's not a 2-dimensional array, it's a one dimensional array of objects consisting of multiple members. Granted, the notatation I offered is very shallow compared to the much more advanced data description languages already in exisitance, but it would be at least a humble start and about all that FITS could handle at present. Oh well, all I wanted was some discussion, and it was quite one-sided against the idea, so I'll drop it. However, the FITS community should really start to consider the fact that there are a growing set of powerful, object-oriented data description languages out there, Object Description Language (ODL) and Abstract Syntax Notation 1 (ASN.1), just to name two. Perhaps the scientists that the software developers are supposed to be supporting would be better served if they had access to data description tools that allowed them to concentrate on what the data is more than what it looks like internally. Just a thought. I'll get off the soapbox, now. -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center From warnock at Hypatia.gsfc.nasa.gov Tue Jun 30 16:43:41 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1380" "Tue" "30" "June" "1992" "18:36:52" "GMT" "Archie Warnock" "warnock at Hypatia.gsfc.nasa.gov " nil "23" "Re: string arrays" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: warnock at Hypatia.gsfc.nasa.gov (Archie Warnock) Subject: Re: string arrays Date: Tue, 30 Jun 1992 18:36:52 GMT In article , leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: |> approaching an "object" into FITS. The objections like "that's just a 2-d |> array" strengthen my feelings that the FITS implementors are too absorbed |> in "bit" descriptions and not "data" descriptions. It's not a 2-dimensional |> array, it's a one dimensional array of objects consisting of multiple members. |> Oh well, all I wanted was some discussion, and it was quite one-sided against |> the idea, so I'll drop it. However, the FITS community should really start to Well, it wasn't all one-sided. Objects are something worth working towards. I never cease to be amazed at how the astronomical community seems to believe science only moves forward in astronomy. Someday, we'll have to realize that useful things have happened in other fields in the last 20 years. Computer science has lots of improvements to offer. How many astronomers are still using Bevington as their primary statistics reference, pretending that nothing useful has been contributed to the literature in the 25 years since he wrote his book? -- ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at hypatia.gsfc.nasa.gov -- Hughes STX "Unix --- JCL For The 90s" -- NASA/GSFC From bschlesinger at nssdca.gsfc.nasa.gov Tue Jun 30 16:44:48 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1549" "Tue" "30" "June" "1992" "18:47:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdca.gsfc.nasa.gov " nil "32" "File validation software prototype for testing" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits News-Software: VAX/VMS VNEWS 1.41 Keywords: FITS tester, conformance Nntp-Posting-Host: nssdca.gsfc.nasa.gov Organization: NASA - Goddard Space Flight Center From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger) Subject: File validation software prototype for testing Date: Tue, 30 Jun 1992 18:47:00 GMT A prototype of the developing NOST FITS Support Office software package to validate FITS files is now available for testing. It is written in ANSI C. The current version checks for the presence of the required keywords in the primary header and whether or not they are in the proper order. It will find them if they are out of order but will issue warnings. Syntax is also checked. Also, if the user chooses, it creates a file of selected members of integer data arrays, as specified by the user, unless there are major header errors. This version cannot handle floating point. If there are minor header errors, it will attempt to derive information to obtain the data array but will warn the user that the data may not be extracted properly. The current version does not check extensions. This very simple prototype is being made available now in order to identify errors and portability problems before the package becomes too complex. The software has been developed on a VAX cluster. It can handle twos complement integers with either most significant byte or least significant byte first, but it cannot handle ones complement. ASCII-EBCDIC conversion also has not been implemented. Sun users should be warned that because the software is ANSI C, they will not be able to use SUN C1.0, the standard compiler for SPARCstations. To obtain a copy of the package and the instructions for use, send electronic mail to (Internet) fits at nssdca.gsfc.nasa.gov (NSI-DECnet) NCF::FITS Barry Schlesinger NOST FITS Support Office From leb at Hypatia.gsfc.nasa.gov Wed Jul 1 07:16:41 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5421" "Tue" "30" "June" "1992" "21:58:57" "GMT" "Lee E. Brotzman" "leb at Hypatia.gsfc.nasa.gov " nil "113" "ODL and ASN.1" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Keywords: Data interchange, standards Nntp-Posting-Host: hypatia.gsfc.nasa.gov Organization: Goddard Space Flight Center From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) Subject: ODL and ASN.1 Date: Tue, 30 Jun 1992 21:58:57 GMT Bob Hill asked me a question about ODL and ASN.1 and the answer got just preachy enough that I think I might just as well post it here as well. :-) > In article , you write... > >data description languages out there, Object Description Language (ODL) and > >Abstract Syntax Notation 1 (ASN.1), just to name two. Perhaps the scientists > > Lee, > > Do you have any references for these? Maybe one reference that discusses > both concisely? > > - Bob Hill > ODL is used by the Planetary Data System to describe all of the data they are distributing on CD-ROM. (I think they were calling it 'PDL' for a while. I can't remember what the 'P' stood for but I don't think it was planetary.) ODL is also an integral part of the Standard Formatted Data Unit (SFDU) Red Book standards put out by NASA and a bunch of other international space agencies. You should be able to get at least the Red Book standards from the NASA Office of Standards and Technology (NOST) at NSSDC. Try sending E-mail to ncf::nost. ASN.1 is an Open System Interconnection (OSI) standard, document ISO 8824. I learned about it my last networking class at Johns Hopkins. It is used to precisely describe the formats of the Protocol Data Units (i.e. packets) shipped around in OSI networks. Even the TCP/IP guys have started using it to define the structures of the TCP and IP packets. Basically you have an "abstract syntax" which describes the data stream in a notation vaguely like Pascal. Then the abstract syntax is "interpreted" into a "transfer syntax" which specifies the bit stream to be transmitted. An example from my text book goes like this... Given a structure like: < Pascal > < C > type dinosaur = record struct dinosaur { name: array [1..12] of character; char name[12]; length: integer; int length; carnivorous: boolean; int carnivorous; bones: integer; int bones; discovery: integer; int discovery; end; } The ASN.1 abstract syntax would be: Dinosaur ::= SEQUENCE { name OCTET STRING, -- 12 characters length INTEGER, carnivorous BOOLEAN, bones INTEGER, discovery INTEGER } Primitive data types are indicated with all upper case characters. It is also possible to define your own data types which are formed from primitive types. Note that the comment following the "--" is just that, a comment. The length of the name of the dinosaur is not encoded (if you think about it, there's just no need to have to specify the maximum length of the dinosaurs name, and there is quite a bit of variation anyway, so it's best to leave it open.) When the ASN.1 abstract syntax is encoded for transmission, each of the fields is translated into a type-length-value bytestream. The type is a single octet which is split into: (a) 2 bits to denote whether the data type is universal (i.e. predefined), application, context-specific, or private; (b) 1 bit to denote whether the type is constructed or primitive (all the types in the example above are primitive); and (c) 5 bits for the type code. Provisions are made for type fields larger than one octet. The length is however many bytes it takes to give the length of the field, and the data for the field immediately follow the length. There is an optional end-of-contents flag if the length of the data object is unknown. The above example, when given the value { "stegosaurus", 10, FALSE, 300, 1877 } encodes into the following byte stream (all numbers are in hexidecimal): ASN.1 Type Length Value SEQUENCE 40 1B (The value is the object describing a dinosaur) OCTET STRING 04 0B 73 74 65 67 6F 73 61 75 72 75 73 "stegosaurus" in ISO 646 (ASCII) INTEGER 02 01 0A = 10 decimal BOOLEAN 01 01 00 = FALSE INTEGER 02 02 2C 01 = 300 decimal (little-endian) INTEGER 02 02 55 07 = 1877 decimal Or, all together: 40 1B 04 0B 73 74 65 67 6F 73 61 75 72 75 73 02 01 0A 01 01 00 02 02 2C 01 02 02 55 07 I may have a few bits wrong here or there -- that's why God gave us computers to figure this crap out! I think that ODL is probably the most used abstract object description language in the NASA environment, given the scads of data from the planetary, space and earth science folks (note that no discipline other than astronomy uses FITS, we are truly alone). When OSI is thrust down our throats in the form of GOSIP, I think we'll be seeing a lot about ASN.1 (or ASN.2, whatever is coming down the pike). The main thing about ASN.1 is that it has a whole pile of money and politics pushing it along. Now, was that answer long-winded enough? :-) -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center -- -- Lee E. Brotzman Internet: leb at hypatia.gsfc.nasa.gov -- Hughes STX DECNET: NDADSA::BROTZMAN -- National Space Science Data Center BITNET: ZMLEB at SCFMVS -- NASA Goddard Space Flight Center From thompson at stars.gsfc.nasa.gov Wed Jul 1 07:17:55 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2612" "" "30" "June" "92" "22:46:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " nil "54" "Re: ODL and ASN.1" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Keywords: Data interchange, standards Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: ODL and ASN.1 Date: 30 Jun 92 22:46:00 GMT In article , leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes... >Bob Hill asked me a question about ODL and ASN.1 and the answer got just >preachy enough that I think I might just as well post it here as well. :-) > > >> In article , you write... >> >data description languages out there, Object Description Language (ODL) and >> >Abstract Syntax Notation 1 (ASN.1), just to name two. Perhaps the scientists >> >> Lee, >> >> Do you have any references for these? Maybe one reference that discusses >> both concisely? >> >> - Bob Hill >> > >ODL is used by the Planetary Data System to describe all of the data they are >distributing on CD-ROM. (I think they were calling it 'PDL' for a while. I can't >remember what the 'P' stood for but I don't think it was planetary.) ODL is >also an integral part of the Standard Formatted Data Unit (SFDU) Red Book >standards put out by NASA and a bunch of other international space agencies. >You should be able to get at least the Red Book standards from the NASA Office >of Standards and Technology (NOST) at NSSDC. Try sending E-mail to ncf::nost. > >ASN.1 is an Open System Interconnection (OSI) standard, document ISO 8824. I >learned about it my last networking class at Johns Hopkins. It is used to >precisely describe the formats of the Protocol Data Units (i.e. packets) >shipped around in OSI networks. Even the TCP/IP guys have started using it to >define the structures of the TCP and IP packets. (rest of message deleted to save space) I think there may be some subtle distinctions between ODL as used by the PDS and PVL (parameter-value language) as incorporated into SFDUs. But they are related; I believe PVL was inspired by ODL. The "Object" in the name may be slightly misleading. Nowadays the term "objects" is used to refer not only the structure of the data, but also the structure of the operations that can be performed on the data. The description of ASN.1 is described as resembling C or Pascal structures. I think that where the ASN.1 most naturally fits into FITS binary tables is to consider that an entire binary table defines a structure, and that the records (rows) of that database are instances of that structure. The object-oriented software for this structure would have to sit on top of the primitives that directly interact with the binary table. Right now we seem to be most concerned with getting the binary representation of the data correct so that we can most efficiently build the higher level software that understands complex objects. Bill Thompson From cflatter at nrao.edu Wed Jul 1 07:18:35 1992 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1783" "Tue" "30" "June" "1992" "23:45:47" "GMT" "Chris Flatters,208,7209,homephone" "cflatter at nrao.edu " nil "33" "Re: ODL and ASN.1" "^From:" nil nil "6"]) Newsgroups: sci.astro.fits Reply-To: cflatter at nrao.edu Organization: NRAO From: cflatter at nrao.edu (Chris Flatters,208,7209,homephone) Subject: Re: ODL and ASN.1 Date: Tue, 30 Jun 1992 23:45:47 GMT In article 709941537 at Hypatia, leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman) writes: >ASN.1 is an Open System Interconnection (OSI) standard, document ISO 8824. I >learned about it my last networking class at Johns Hopkins. It is used to >precisely describe the formats of the Protocol Data Units (i.e. packets) >shipped around in OSI networks. Even the TCP/IP guys have started using it to >define the structures of the TCP and IP packets. > > ... > >I think that ODL is probably the most used abstract object description language >in the NASA environment, given the scads of data from the planetary, space and >earth science folks (note that no discipline other than astronomy uses FITS, we >are truly alone). When OSI is thrust down our throats in the form of GOSIP, I >think we'll be seeing a lot about ASN.1 (or ASN.2, whatever is coming down the >pike). The main thing about ASN.1 is that it has a whole pile of money and >politics pushing it along. ASN.1 is intended to be used for the encoding of data for transmission over a network connecting computers of different architectures (hence the stream orientation). It is similar to XDR save that information in an ASN.1 data stream contains type tags while information in an XDR stream does not. In both cases a program reading the data needs to know what types of data to expect in what order to be able to process the data stream (but an ASN.1 reader can detect that it isn't getting the data it expects while an XDR reader will just read garbage). FITS files describe both the format of the data contained in them and supply hints on how it is to be interpreted. ASN.1 streams describe the data contained in them but give no information on how that data is to be interpreted. Chris Flatters cflatter at nrao.edu