From conrad at jupiter.ucsc.edu Thu Aug 1 17:25:22 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["317" "" "1" "August" "91" "16:28:26" "GMT" "Al Conrad, x3208" "conrad at jupiter.ucsc.edu " nil "9" "fitsio lib -> STS" "^From:" nil nil "8"]) Newsgroups: alt.sci.astro.fits Organization: University of California, Santa Cruz From: conrad at jupiter.ucsc.edu (Al Conrad, x3208) Subject: fitsio lib -> STS Date: 1 Aug 91 16:28:26 GMT I am using the fitsio library (HEASARC, GSFC, version 2.0) for a spectograph data taking system and have been asked to extend the system to produce the two-file fits-on-disk format used by IRAF-STSdAS. Are there plans to extend fitsio to produce this format? Thank you in advance, Al Conrad conrad at helios.ucsc.edu From dwells at fits.cx.nrao.edu Fri Aug 2 17:25:53 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5058" "Fri" "2" "August" "1991" "19:51:38" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " nil "90" "DynaFITS (LONG. was: fitsio lib -> STS)" "^From:" nil nil "8"]) Newsgroups: alt.sci.astro.fits In-Reply-To: jwright at cfht.hawaii.edu's message of 2 Aug 91 08: 11:56 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: DynaFITS (LONG. was: fitsio lib -> STS) Date: Fri, 2 Aug 1991 19:51:38 GMT In article jwright at cfht.hawaii.edu (Jim Wright) writes: JW> ...How do you support adding an arbritary number of cards to a JW> FITS header? The obvious solution (two separate files, JW> concatenated when finished) is less attractive as the size of the JW> data portion grows. Right now we're at 8MB files (2Kx2K) with JW> the prospect of 32MB shortly (4Kx4K). Any clever way to prepend JW> the header to the data without having to move megabytes of data? I assume that you are wanting to add more cards to an already-written header in order to record ancillary information about the observation. Many practical cases like yours are well-represented in tabular form. If your ancillary data looks like a table, you should consider using the FITS ASCII-table extension for this purpose. Voluminous tables can be represented more efficiently in binary format; there is good reason to expect that most astronomical data systems will be enhanced to support the proposed BINTABLE extension during the next few years (see file /FITS/doc/bintable.ps on fits.cx.nrao.edu [192.33.115.8]). However! The table technique does not really address the basic problem which you point out. The problem arises in two contexts: (1) RT data acquisition systems in which header and data need to be created _asynchronously_ (the case which concerns you) and (2) use of FITS as a machine-independent _random-access_, _re-writeable_, _shareable_ disk format in data analysis systems. In August 1990, at an NRAO workshop discussing what is now called the aips++ project, item (2) was suggested as a desirable concept for the disk file format of aips++, and Bob Hanisch (STScI) and I were asked to bring back a proposal. The concept which we suggested was to define an index file, a table, which would tabulate pointers to a sequence of byte-strings which, when concatenated, would constitute a standard FITS file. This would permit not only the content but also the _length_ of any segment in the FITS file to be updated dynamically, an essential feature in a data analysis system. Accesses to the index table for updates would be a "critical region" and would have to be guarded by lock mechanism(s), of course. This idea for dynamically updating FITS bytestreams was promptly dubbed "DynaFITS" by someone else at the workshop (Geoff Croes?). The original concept was that the index table would have three columns, local filename (including path), byte offset and number of bytes. Over the past year I have thought a lot about this idea, and I am just about convinced that it is a "good thing". I now think that the filename column should use some sort of syntactic convention to indicate remote hosts and also a convention to indicate (remote) _sockets_. My current thinking is that the index file should be formatted as a naked FITS binary table (no Basic-FITS header). I have not decided on column names, but 'PATH', 'OFFSET' and 'LENGTH' sound like good choices. Presumably the name would be EXTNAME='DYNAFITS'. Perhaps one or two more columns should be added to specify whether a segment is the beginning of the Basic-FITS data, or an extension header of a certain type, or the extension data, or... Incidentally, such a table could have other, system-dependent, proprietary columns which could be used by a data system to keep track of such things as whether a binary copy of the header segment is cached in the I/O system, whether a machine-dependent translation of the data segment is in a scratch file, whether an index exists for a column of a table, etc... The idea of being able to specify a _socket_ instead of a path to a file is potentially a very powerful feature of DynaFITS, because it would enable an arbitrary server process on an arbitrary host to _dynamically_ create a segment of a FITS file by an arbitrary computational algorithm. It would be possible to compute the various segments of the file with different processes at different times. This feature of the DynaFITS idea is particularly important for RT data acquisition systems. In the case being cited in the original posting, an auxiliary table could be computed and written to a file while the CCD image is being copied to another file, and while yet another process is generating the Basic-FITS header in a third file. The result would be a virtual FITS file which could be copied to external media later on by a trivial utility operation which reads the index file and does the obvious thing. This posting to alt.sci.astro.fits about the DynaFITS idea is not really a proposal, it is merely a trial balloon. I may or may not build a prototype implementation of DynaFITS myself for the VLBA Correlator. I would be happy to discuss the idea with anyone... -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From warnock at nssdca.gsfc.nasa.gov Thu Aug 8 17:46:14 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2031" "" "2" "August" "91" "22:32:00" "GMT" "Archie Warnock" "warnock at nssdca.gsfc.nasa.gov " nil "35" "Re: fitsio lib -> STS" "^From:" nil nil "8"]) Newsgroups: alt.sci.astro.fits Organization: ST Systems Corp. - NASA/NCDS News-Software: VAX/VMS VNEWS 1.4-b1 From: warnock at nssdca.gsfc.nasa.gov (Archie Warnock) Subject: Re: fitsio lib -> STS Date: 2 Aug 91 22:32:00 GMT In article , jwright at cfht.hawaii.edu (Jim Wright) writes... >This sort of brings up a problem we've been having with FITS. How do you >support adding an arbritary number of cards to a FITS header? The obvious >solution (two separate files, concatenated when finished) is less attractive >as the size of the data portion grows. Right now we're at 8MB files (2Kx2K) Well, the problem isn't so much that using two files is unattractive - rather that the FITS reader you use insists on having a single file. The approach taken by IRAF/SDAS/etc, while not FITS (see, Don? I _have_ learned that much), is close enough to allow virtually trivial conversion. There is also code in the IDL User's Library to handle this flavor. Perhaps the solution is in modifying the software to read what you can easily manipulate, rather than modifying the data to fit what the software can read. FITS isn't all that flexible, at least, not in that sense. The separate file approach has also been used by the International Halley Watch, so solve the problem of how you get the darn headers out of the way of tabular data so you can read it with something useful, like a database program. Final solution? If you're absolutely tied to reading FITS (single file, as it were :-), try putting in some blank 80-byte records (don't you dare call them "cards"!!) before the final END statement. Since everything is fixed length, 80-bytes within the 2880 byte logical structure, you can do random-access writes into the blanks. Be sure to fill those records with genuine, ASCII 32s, though. Anything else may cause your FITS reader (or someone else's) to puke. FITS isn't flexible in _that_ sense, either. ---------------------------------------------------------------------------- -- Archie Warnock Internet: warnock at nssdc.gsfc.nasa.gov -- ST Systems Corp. SPAN: NSSDC::WARNOCK -- NASA/GSFC "Unix - JCL for the 90s" From dwells at fits.cx.nrao.edu Sun Aug 11 18:16:35 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1015" "Sun" "11" "August" "1991" "02:29:05" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " nil "22" "Re: FITS standards" "^From:" nil nil "8"]) Newsgroups: alt.sci.astro.fits In-Reply-To: leonard at qiclab.scn.rain.com's message of 9 Aug 91 23: 37:02 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Re: FITS standards Date: Sun, 11 Aug 1991 02:29:05 GMT In article <1991Aug9.233702.5948 at qiclab.scn.rain.com> leonard at qiclab.scn.rain.com (Leonard Erickson) writes: I have a somewhat naive question. Where can I get "the" description of the FITS format? I've got a couple of FITS files from sci.astro, and if I have to write a display program, I'd like it to be smart enough to handle any image my display can. (note that I'm using Messy-DOS and have no FTP capability...) FITS documents are available in various anonymous-FTP servers, e.g. fits.cx.nrao.edu [192.33.115.8] (directory /FITS/docs) and nssdca.gsfc.nasa.gov. I don't know of any mechanism by which a UUCP user can fetch files from anonFTP servers. I will Email the documents to you if you wish. -Don -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From dbriggs at zia.aoc.nrao.edu Thu Aug 22 14:29:32 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["5932" "" "22" "August" "91" "16:30:59" "GMT" "Daniel Briggs" "dbriggs at zia.aoc.nrao.edu " nil "108" "Re: Second summary of Image Standard for Space Shuttle Program" "^From:" nil nil "8"]) Newsgroups: comp.graphics,sci.space.shuttle,alt.sci.astro.fits Distribution: usa Organization: National Radio Astronomy Observatory, Socorro NM From: dbriggs at zia.aoc.nrao.edu (Daniel Briggs) Subject: Re: Second summary of Image Standard for Space Shuttle Program Date: 22 Aug 91 16:30:59 GMT In article <1991Aug20.050605.324 at aio.jsc.nasa.gov> kjenks at galileo.uucp (Kenneth C. Jenks [GM2] 483-4368) writes: >The story so far: > >I'm in the Space Shuttle Program Office (Level II), in the >Management Integration/Information Systems Office at JSC. My >office sets information systems standards for the Space Shuttle >Program. My boss asked me to put together a draft of a photo >transmission and storage standard. > >I proposed to the Internet that I was going to make the Shuttle program >standardize on GIF as the image file storage and interchange format. >Everybody said no, so I changed by mind and proposed TIFF 5.0. The >plan is to impose an interim standard now, then move all our files to >the official US Gov't standard when it comes out. I don't flame in public that often, but this is above and beyond.... I didn't reply to your original post because *I THOUGHT IT WAS A JOKE*! I simply couldn't believe that a) someone would possibly suggest using GIF as a serious format for interchange of even semi-scientific data and that b) someone would, in the same message, suggest that this decision would have serious implications across the entire US space program, ask questions of a level indicating that essentially *no* background research has been done ("What are the advantages and disadvantages of GIF vs. the other color image standards?") and propose as the standard the single format about which he had (incomplete) information. Look, Ken, I'm happy that you've decided to let the net in on an important decision, and Usenet is a valuable resource for something like this. But we are NOT here to do your job for you. If you look at your original post, it has more in common with the posts that begin, "I'm doing a report for my high school science class. Please send me all the information you have on the space shuttle." than it does with a serious scientific proposal. The next thing you should realize is that while the network is a useful way to gather large numbers of opinions, it is probably not a representative sample of the people you need to ask. If you ask a teenage hacker with a C64 what image standard he wants you to use, he will likely tell you GIF or TIFF. If you ask a researcher at a scientific institution, she will likely tell you something else. So the first question on your list should be "what is the user population for my standard, and what are their needs?" I don't know the answer to that question, so it's hard to suggest an answer to your original post. I strongly suspect that you will have at least two fundamentally different populations of users, and that their desires will be different and possibly even incompatible. The scientific community *must* have an imaging standard with header information. At a minimum, it must specify such information as how the axes of the image correspond to real world coordinates on the sky or on the earth. It must specify the observing wavelength(s). It should specify when the data was taken, by whom, and with what instrument. It might include history information on how the data was processed, (eg. clipping, transformations and deconvolution). It might include auxiliary tabular information such as filter characteristics or gain information. I'll admit that I'm personally partial to FITS as a common standard that does all this and more, but essentially any well thought out scientific standard that preserves this information would do. If you go with GIF or TIFF, you'll have to have some sort of auxiliary header file which will of course need its own standard. Two files instead of one then carries the risk that they can become separated or modified relative to each other. Better just to do it right in the first place. On the other hand, the more complete formats are complicated. FITS readers are just now starting to appear for PC class machines, and aren't exactly common. (It's probably even worse for the other scientific imaging standards.) FITS itself isn't really an ideal format for the transmission of simple color images. It essentially deals with the problem of getting arbitrary numbers into arbitrary shaped arrays on the target machine, and leaves the problem of display to the target. I don't believe that there is a generally accepted substandard on color table mappings, for instance. (If I am wrong on this, someone please tell me?) While it would certainly be trivial to put the three RGB planes on the third axis of the data cube, I'm not even sure if that is standardized. You might want to get in touch with STScI, to see if they've adopted any standard for "general release" images. Our local FITS guru is Don Wells, (dwells at nrao.edu), and I'm sure he'd be willing to answer questions if you explained your situation. Also, it's true that FITS is not a compressed format. To some extent, this can be alleviated with external compressors, but that's still a pain. A compressed FITS standard is in the works, but don't expect it anytime soon. It really looks to me like you're going to be stuck with two standards: an internal one for people who need to work with the images, and an external one for the people who simply want to look at the pictures. If you've got to have one single format, I think the external users lose, since they can go to the more complicated formats, while the scientists simply cannot put back data that has been thrown away. >I'm writing a position paper this week. I'll post a draft to the net >for comments. While I question that you're really ready to write a position paper this week, please do post what ever you come up with on the net. You might seriously think about using that week in email and phone discussions with the people who are most likely to use your images, though. -- Daniel Briggs (dbriggs at nrao.edu) New Mexico Tech / National Radio Astronomy Observatory P.O. Box O / Socorro, NM 87801 (505) 835-7391 From dwells at fits.cx.nrao.edu Sun Aug 25 18:30:00 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2428" "Sun" "25" "August" "1991" "23:11:53" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " nil "47" "Re: Office Document Architecture ISO 8613" "^From:" nil nil "8"]) Newsgroups: alt.sci.astro.fits In-Reply-To: sla at helios.ucsc.edu's message of 23 Aug 91 16: 26:37 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Re: Office Document Architecture ISO 8613 Date: Sun, 25 Aug 1991 23:11:53 GMT In article <19879 at darkstar.ucsc.edu> sla at helios.ucsc.edu (Steve Allen) writes: SA> Summary: FITS become obsolete? I doubt it. SA> Keywords: Commercial Image Standards SA> SA> [In] Message-ID: <18929 at venera.isi.edu> rogers at wlf.isi.edu (Craig SA> Milo Rogers) [says:] SA> CMR> To a large extent, TIFF, FITS, and other such standards will SA> CMR> be obsolete if one of the newer Office Document Architecture SA> CMR> standards, such as ISO 8613, becomes commercially successful. SA> SA> I wonder if the author of the above knows anything about FITS. His words suggests that he thinks FITS is only a syntax for transmitting screendumps of 8-bit bytes. Also, the quoted remark doesn't address the issues of (1) ancillary information and (2) semantics of scientific data. SA> Does anyone in the astro community know anything about ISO 8613? I recently read a discussion of ODA by somebody at DEC. It sounded like discussions of SGML to some extent. It also sounded like discussions of X.400 and other "multi-media [E]mail" techniques. There are commercial products in this area, but it is unclear whether any of these things are truly "commercially successful" yet. They are schemes for encapsulating non-textual data objects inside a "document", in a more graceful fashion than, e.g., injecting a "Fig" drawing into a LaTeX document with a \special to give instruction to dvi2ps. In the sense that such documents can encapsulate a spreadsheet (table) as well as an image (matrix) plus arbitrary text (ancillary information), there is an element of truth in the assertion by Rogers, but it appears to me that document architecture folk will need to do a lot more R&D before they can capture the market for archival representation of scientific data. Because such "document" architectures can encapsulate a variety of data structures, and are extensible, they could encapsulate FITS bytestreams within documents. BUT!! Because the FITS architecture can encapsulate a variety of data structures as generalized extensions, and is extensible, FITS could encapsulate any other types of bytestreams (e.g., ODA) within FITS files. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N