From GROTH at pupgg.princeton.edu Thu Oct 10 23:23:04 1991 X-VM-Message-Order: (18 19 21 20 10 11 15 22 23 24 25 27 28 29 30 9 1 2 5 6 7 8 13 14 16 3 4 12 17 26) X-VM-Summary-Format: "%n %*%a %-17.17F %-3.3m %2d %4l/%-5c %I\"%s\"\n" X-VM-Labels: nil Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["866" "" "8" "October" "91" "21:35:47" "GMT" "Edward J. Groth 609-258-4361" "GROTH at pupgg.princeton.edu " "<2539367 at toto.iv>" "23" "Legal line in header???" "^From:" nil nil "10" "1991100821:35:47" "Legal line in header???" (number " " mark " Edward J. Groth Oct 8 23/866 " thread-indent "\"Legal line in header???\"\n") nil] nil) X-VM-VHeader: ("Resent-" "From:" "Sender:" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:") nil X-VM-Bookmark: 9 Newsgroups: alt.sci.astro.fits Reply-To: groth at pupgg.princeton.edu Organization: Physics Department, Princeton University Nntp-Posting-Host: pupggg.princeton.edu From: GROTH at pupgg.princeton.edu (Edward J. Groth 609-258-4361) Subject: Legal line in header??? Date: 8 Oct 91 21:35:47 GMT I've just come across a FITS header with a line of the following form: IDENT4 = R.A 00H -19 DEC +89 /title of map I thought what could follow an equals sign was T, F, a legal integer, a legal floating point number with exponent field if desired, or alphanumeric data enclosed in ' '. If so, the above line is illegal. So, is the line legal or not. If it's legal, what are the syntax rules for header lines. - Ed /----------------------------------------------------------------------\ | Edward J. Groth | Phone: 609-258-4361 | | Physics Dept., Jadwin Hall | Fax: 609-258-1124 | | Princeton University | SPAN/HEPNET: PUPGG::GROTH=44117::GROTH | | Princeton, NJ 08544 | Internet: groth at pupgg.princeton.edu | \----------------------------------------------------------------------/ From bschlesinger at nssdcb.gsfc.nasa.gov Thu Oct 10 23:23:15 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1966" "" "9" "October" "91" "17:11:00" "GMT" "Barry Schlesinger" "bschlesinger at nssdcb.gsfc.nasa.gov " "<1100956 at toto.iv>" "39" "Re: Legal line in header???" "^From:" nil nil "10" "1991100917:11:00" "Legal line in header???" (number " " mark " Barry Schlesinger Oct 9 39/1966 " thread-indent "\"Re: Legal line in header???\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Organization: NASA - Goddard Space Flight Center News-Software: VAX/VMS VNEWS 1.41 Nntp-Posting-Host: nssdcb.gsfc.nasa.gov From: bschlesinger at nssdcb.gsfc.nasa.gov (Barry Schlesinger) Subject: Re: Legal line in header??? Date: 9 Oct 91 17:11:00 GMT In article <15029 at princeton.Princeton.EDU>, groth at pupgg.princeton.edu writes... >I've just come across a FITS header with a line of the following >form: > >IDENT4 = R.A 00H -19 DEC +89 /title of map > >I thought what could follow an equals sign was T, F, a legal >integer, a legal floating point number with exponent field if >desired, or alphanumeric data enclosed in ' '. > >If so, the above line is illegal. > >So, is the line legal or not. If it's legal, what are the syntax >rules for header lines. The fixed format for keyword value, defined in the original FITS paper, is strictly required only for mandatory keywords. Such keywords would include the required keywords for the primary header, the keywords required by the agreement on generalized extensions, and the keywords required for individual extension types. For other keywords, the fixed format is not required but strongly recommended. As a matter of practice, I would urge that the fixed format be used except in cases where it is clearly impossible to do so, such as a double precision floating point number in exponential form that requires more that 20 columns. For the case in question, strictly speaking, the line is permissible. It appears to be intended to be a character string. In that case, I would urge the writers to reorganize the value in a way that allows them to surround it with quotes. (Another, somewhat obscure, interpretation is that the entire field starting in column 9 is a comment. If the keyword has a value, than column 9 must contain an equal sign. However, the presence of an equal sign in column 9 does not require that the keyword have a value! I would strongly recommend against putting an equal sign in column 9 when there is no value, because it will lead to confusion.) So in summary, I would say that the line is "legal". But not everything that is legal is good practice. Barry Schlesinger NSSDC/NOST FITS Support Office From dwells at fits.cx.nrao.edu Thu Oct 10 23:23:19 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2340" "Wed" "9" "October" "1991" "22:13:40" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " "<6548567 at toto.iv>" "44" "Re: Legal line in header???" "^From:" nil nil "10" "1991100922:13:40" "Legal line in header???" (number " " mark " Don Wells Oct 9 44/2340 " thread-indent "\"Re: Legal line in header???\"\n") nil] nil) Newsgroups: alt.sci.astro.fits In-Reply-To: bschlesinger at nssdcb.gsfc.nasa.gov's message of 9 Oct 91 17: 11:00 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Re: Legal line in header??? Date: Wed, 9 Oct 1991 22:13:40 GMT In article <9OCT199112110371 at nssdcb.gsfc.nasa.gov> bschlesinger at nssdcb.gsfc.nasa.gov (Barry Schlesinger) writes: BS> ... the case in question... appears to be intended to be a BS> character string.... the presence of an equal sign in column 9 ... BS> [FITS] does not require that the keyword have a value ... in summary, BS> I would say that the line is "legal". I myself would say that it should be illegal for a header line to have an equal sign in column 9 with a value field in an invalid format. Reasonable people can easily differ about this (as Barry and I are doing), because the FITS papers are ambiguous on this subtle point, as on many others. I just checked NOST-100-0.2f (14May91), the NASA draft standard for FITS. Its wording is also somewhat ambiguous. In 5.1.1 (Syntax) it says "if a value is present, column 9 shall contain an equal sign... column 10 ... [a] blank, and columns 11-80 ... as specified in Section [5.3]. If no value is present... 9-80 may contain any ASCII text." (actually 5.1.1 says the "remainder" of 5.2, but I think it really means 5.3 --- this should be checked). While 5.1.1 does not explicitly say that an = in 9 *requires* a valid value, it does seem to me that that is the most plausible reading of the wording. The wording of 5.1.2.2 (Value Indicator (bytes 9-10)) is consistent with this interpretation. Section 5.2.3.1 (Additional Keywords, Requirements) also appears to be consistent, but is somewhat more ambiguous. The wording of 5.3 (Value), interpreted in the context of the whole of Section 5 (Headers), is clearly restricting value fields to 5 possible formats with syntactic rules taken from Fortran-77 list-directed READ: string ('-quoted), logical, integer, float and complex (two floats), with /-delimited comments. I cannot think of any future expansion need that would make it useful to allow arbitrary strings behind an equal sign in column 9. I recommend that the NOST Technical Panel review this technical point, and provide an explicit rule in Sections 5.x of NOST-100-x.x. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From dwells at fits.cx.nrao.edu Thu Oct 10 23:23:22 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1107" "Thu" "10" "October" "1991" "19:48:57" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " "<1169533 at toto.iv>" "25" "Re: Legal line in header???" "^From:" nil nil "10" "1991101019:48:57" "Legal line in header???" (number " " mark " Don Wells Oct 10 25/1107 " thread-indent "\"Re: Legal line in header???\"\n") nil] nil) Newsgroups: alt.sci.astro.fits In-Reply-To: dwells at fits.cx.nrao.edu's message of Wed, 9 Oct 1991 22: 13:40 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Re: Legal line in header??? Date: Thu, 10 Oct 1991 19:48:57 GMT In article dwells at fits.cx.nrao.edu (Don Wells) writes: DW> I myself would say that it should be illegal for a header line to DW> have an equal sign in column 9 with a value field in an invalid DW> format. Barry Schlesinger has informed me that Preben Grosbol has pointed out an exception to the interpretaton which I suggested: Section 5.2.2.4 ("Commentary Keywords") of the NOST document specifies the following rule for keywords COMMENT, HISTORY and blank: "this keyword shall have no associated value; columns 9-80 may contain any ASCII text". This rule allows an "=" in column 9 with columns 10-80 containing an invalid format. I recommend that the NOST Technical Panel revise the text so that these three cases will be an explicit exception to an explicit rule. -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From pence at heawk1.gsfc.nasa.gov Wed Oct 23 10:25:37 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["7919" "Thu" "3" "October" "1991" "21:06:21" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " "<1619644 at toto.iv>" "133" "Comments on 'Binary Table Extension' draft" "^From:" nil nil "10" "1991100321:06:21" "Comments on 'Binary Table Extension' draft" (number " " mark " William Pence Oct 3 133/7919 " thread-indent "\"Comments on 'Binary Table Extension' draft\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Keywords: fits Nntp-Posting-Host: heawk1.gsfc.nasa.gov Organization: Goddard Space Flight Center From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Comments on 'Binary Table Extension' draft Date: Thu, 3 Oct 1991 21:06:21 GMT Comments on the "Binary Table Extension to FITS" Draft by Cotton and Tody dated 20 September 1991: For the most part, I have found that the binary table format proposed in this document to be unambiguously defined and straight forward to implement. It appears general enough to be able to handle virtually any type or format of data. I was able to support all the binary table features in the FITSIO software package except for the 'Variable length array facility' which will be added in the near future. I have just a couple suggestion for changes to this draft: 1) Sections 5.7 and 5.8 states: 'The IEEE NaN (not a number) values are used to indicate an invalid number; ... All IEEE special values are recognized.' I have 2 comments on this. First, why we do we need more than one value to represent an undefined pixel? An IEEE NaN value is defined to have all the exponent bits set to 1 and a non-zero set of fractional bits which is a very large set of different possible values. This makes it more difficult for FITS readers, especially those written in Fortran, to test if a pixel is undefined. It would be very much simpler if FITS were to adopt a single NaN value to represent undefined values; an obvious choice would be to adopt a value with ALL bits = 1 (which is equivalent to a value of -1 in 2's complement integer notation). This would have the advantage that the value has the same representation on every type of computer (you don't have to worry about where the exponent bits are on machines with swapped bytes) and it would make it much more efficient for FITS readers to test for undefined values. (I realize that the FITS definition for primary array states that any NaN value may be used to represent an undefined value and it may be too late to change this now. But this does not imply that we have to adopt the same convention for the binary table extension). Secondly, what does it mean that 'All IEEE special values are recognized'? If a pixel has the IEEE value representing + or - infinity, for instance, what is a FITS reader supposed to do with it? These special values are not supported on all machines, so the FITS reader can't simply pass them on without possible loss of information. 2) The first paragraph of section 5 states: 'The last 2880-byte record should be zero byte filled past the end of valid data.' I believe that this is an overly strict requirement, and is in fact unenforceable. Are FITS reading programs supposed to read these bytes to be sure they are zero, and if not, refuse to read the rest of the FITS file?? I would suggest modifying this sentence to state: 'The value of any bytes in the last 2880-byte record past the end of valid data are undefined and FITS reading software should make no assumptions as to their values.' This is a rather trivial objection for static FITS files, but is more important in applications where a FITS file is used in a dynamic real time environment where the contents are frequently modified (e.g., in a database type application). It might impose needless overhead on the software if it has to explicitly reset these bytes to zero any time the length of the table is shortened. 3) The biggest problem I have with the current Binary Table draft has to do with the way character fields are handled. Basically this standard treats these as arrays of single characters rather than as strings. I believe that this is a too primitive level for most FITS applications which would prefer to treat a string of characters as a single unit. This becomes very important when one has to deal with arrays of strings (and not single characters) in a table. It is perhaps clearer to illustrate the problem with an example: Suppose a table contains a column with TFORM = '20A', i.e, each row of the column contains 20 characters. Now suppose a user calls a fortran subroutine (in the FITSIO package for example) to read values from this column. The user passes a character string array to the subroutine (for the sake of argument, suppose the user has declared the character array as CHARACTER*10 CARRAY(50)) and the user expects the subroutine to fill up the array with the first 50 strings contained in the FITS table column. But the subroutine cannot do this because it only knows that there are a total of 20 characters in each row, but has no idea if these should be interpreted as a single 20-character string, 2 10-character strings, 4 5-character strings, etc. Thus, the subroutine cannot pack characters from the FITS column into the user's Fortran string array because it does not know the length of each FITS string. The draft document proposes a way to solve this problem by defining another keyword in the header, e.g., TDIM = '(5,4)', to signify that the 20A column is to be interpreted as 4 substrings of 5 characters each. But I object to this on 2 grounds: first, this proposed convention for the use of TDIM is only defined in Appendix A and is not part of the formal binary table definition and thus is not binding on the creators of FITS files. FITS readers then cannot be sure that the users will always adopt this convention. The second, and to me more compelling objection is that a subroutine interface like FITSIO does not need to look for a TDIM keyword for any other column data type except for ASCII characters. For all other datatypes the TFORM keyword itself is sufficient for the FITS reading subroutine to read the data. For example, if the user wants to read a column of double precision numbers, the reading program only needs to look at the TFORM keyword (e.g, TFORM = '10D') to know how many values are contained in each row of the column. It is only in the case of ASCII strings that the subroutine has to go hunting around for more information in order to be able to interpret the data. (This is analogous to the situation if you were asked to read a column of integer values, which contained 4 bytes per row, but you did not know whether the the column contained 4 1-byte integers, 2 2-byte integers, or 1 4-byte integer in each row). Note in passing that in a subroutine interface like FITSIO, one does not need to worry about the higher dimensionality of the data in the FITS files; as far as the FITSIO interface is concerned, every FITS array or table column can be interpreted as a one-dimensional array of data; it is the responsibility of the calling program to group the data into higher dimensions, if required. As a better solution to this problem, I propose that the TFORM value for ASCII columns be changed to the form: TFORM = 'rAw' where 'w' is the length of a unit string, and 'r' is a repeat count. For instance, '3A5' would indicate that the field contains 3 strings each 5 characters in length. An exception to this general rule would be the case where no width is specified, as in TFORM = '20A'. This would be interpreted as if it were written 'A20', i.e., that this is a single string 20 characters in width and not 20 1-character strings. This modified form for TFORM is also the same is is used in FITS ASCII table extensions, except for the fact that ASCII tables columns cannot have a repeat count. This new convention for the TFORM keyword has already been implemented in the version 3.xx FITSIO software for anyone who wants to try it out. This is of course completely unofficial, so users should use this feature sparingly for any real data that is to be distributed to other users. 4) I do not care much for the ability to terminate a character string before its explicit length by an ASCII NULL character. This feature is useless in Fortran implementations, and just requires more overhead to test for NULL characters, and then pad them out with blanks. Using a NULL in the first character to represent an undefined string is fine though. -Bill Pence, HEASARC pence at tetra.gsfc.nasa.gov or LHEAVX::PENCE From dwells at fits.cx.nrao.edu Wed Oct 23 10:25:43 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4366" "Fri" "11" "October" "1991" "05:35:26" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " "<672734 at toto.iv>" "82" "Re: Comments on 'Binary Table Extension' draft" "^From:" nil nil "10" "1991101105:35:26" "Comments on 'Binary Table Extension' draft" (number " " mark " Don Wells Oct 11 82/4366 " thread-indent "\"Re: Comments on 'Binary Table Extension' draft\"\n") nil] nil) Newsgroups: alt.sci.astro.fits In-Reply-To: pence at heawk1.gsfc.nasa.gov's message of Thu, 3 Oct 1991 21: 06:21 GMT Organization: National Radio Astronomy Observatory, Charlottesville, VA From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Re: Comments on 'Binary Table Extension' draft Date: Fri, 11 Oct 1991 05:35:26 GMT In article pence at heawk1.gsfc.nasa.gov (William Pence) writes: WP> Date: Thu, 3 Oct 1991 21:06:21 GMT WP> Comments on the "Binary Table Extension to FITS" Draft by Cotton WP> and Tody dated 20 September 1991: WP> ... Sections 5.7 and 5.8 states: 'The IEEE NaN (not a WP> number) values are used to indicate an invalid number; ... All WP> IEEE special values are recognized.' ... An IEEE NaN value is WP> defined to have all the exponent bits set to 1 and a non-zero set WP> of fractional bits ... difficult for FITS readers ... written in WP> Fortran, to test if a pixel is undefined... simpler ... adopt a WP> single NaN value ... ALL bits = 1 ... equivalent to -1 in 2's WP> complement ... This BINTABLE NaN rule is consistent with the NaN rule of the Floating Point Agreement which became effective 01-January-1990. The motivation of the NaN rule was to allow IEEE machines to use their hardware support to propagate NaNs automatically. The majority of computers used in astronomical computing today support IEEE FP. The usual solution to the problem that standard Fortran is unsuitable for systems programming is to resort to a set of machine-dependent utility subroutines ("Z" routines in the parlance of IRAF and AIPS). This problem does not arise for C programmers because the standard language includes shift and mask operators. WP> ... what does it mean that 'All IEEE special values are WP> recognized'? If a pixel has the IEEE value representing + or - WP> infinity, for instance, what is a FITS reader supposed to do with WP> it? These special values are not supported on all machines, so WP> the FITS reader can't simply pass them on without possible loss WP> of information. IEEE machines support +/-Inf in their hardware and libraries(?). Most data analysis systems do not exploit this fact, but I contend that they should. My opinion is not just theoretical, I put it into practice in the IPPS [Interactive Picture Processing System] which I and others built at KPNO during ~1974 to ~1980, and which ran in production until May 1985. I operated the CDC-6000-series CPU in the mode in which its hardware and its library routines would calculate and propagate infinities by the formal mathematical rules. In the IPPS when you took the logarithm of an image the zero pixels produced -Infs, and when you exponentiated that image you got your zeroes back! The arctangent of +Inf was indeed pi/2, etc. When the IPPS had to convert floating to integer for output it would interpret Infs as NaNs. This is just what a FITS writer should do today when writing floating numbers in an integer format. When reading floating FITS data on a non-IEEE machine I recommend that the Infs and NaNs be construed as "blanks". If your datasystem does not support the concept of "blanks" you will have to make some arbitrary decision. Incidentally, Crays support NaNs and Infs, even though they are are not IEEE machines. WP> 2) The first paragraph of section 5 states: 'The last 2880-byte WP> record should be zero byte filled past the end of valid data.' I WP> believe that this is an overly strict requirement, and is in fact WP> unenforceable. ... This is consistent with the zero-padding rule of the Basic FITS Agreement of 1979. I view the rule as a matter of neatness, of craftsmanship. I can imagine some programmer new to FITS examining a hex dump of a file and getting very confused when the data values in the last logical record do not end at the expected place, and indeed appear random -- or worse, display some pattern left over in the memory of the writing program. The zeroes will reassure hir... The draft NOST standard document specifies the zero-padding rule in Sect.4.3.2 (Primary Data Array) and specifies ASCII-blank-padding for the TABLE extension in 8.1.3 (Data Sequence). I myself am reluctant to relax these rules because of my neatness motivation, but I do concede that you have a point when you say that they are "overly strict requirement[s], and [are] in fact unenforceable". -- Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N From olson at anchor.esd.sgi.com Wed Oct 23 10:25:57 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["975" "" "11" "October" "91" "04:45:09" "GMT" "Dave Olson" "olson at anchor.esd.sgi.com " "<6475947 at toto.iv>" "22" "Re: Exabyte tape marks (was Re: Maximum blocking factor)" "^From:" nil nil "10" "1991101104:45:09" "Exabyte tape marks (was Re: Maximum blocking factor)" (number " " mark " Dave Olson Oct 11 22/975 " thread-indent "\"Re: Exabyte tape marks (was Re: Maximum blocking factor)\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Organization: Silicon Graphics, Inc. Mountain View, CA From: olson at anchor.esd.sgi.com (Dave Olson) Subject: Re: Exabyte tape marks (was Re: Maximum blocking factor) Date: 11 Oct 91 04:45:09 GMT In <1991Oct5.075855.2333 at noao.edu> tody at noao.edu (Doug Tody) writes: ... | Whether or not one makes use of short filemarks depends upon the driver | used. The Sun SCSI tape driver, at least to date, writes only long | filemarks. Most people with Suns use the Sun st driver since it comes with | SunOS. Better drivers can be found from third party vendors. ApUNIX sells | such a driver, although we have not tried it here (we do use their DAT | driver). Features such as short filemarks for Exabyte or fast file | seeks for DAT may not be available with a poor driver. | | If I recall correctly DAT filemarks are 130 Kb or so... The ARDAT Python uses 4 bytes (not even Kbytes) for a FM. Obviously if no data is written, it can still take up an entire frame, which is ~122Kbytes if I remember right. Most other DAT drives conforming to DDS are the same. -- Dave Olson ----============---- Life would be so much easier if we could just look at the source code. From vicki at quake.Stanford.EDU Wed Oct 23 10:26:09 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["406" "" "15" "October" "91" "22:33:21" "GMT" "Vicki Johnson" "vicki at quake.Stanford.EDU " "<7805135 at toto.iv>" "7" "keyword with multiple values" "^From:" nil nil "10" "1991101522:33:21" "keyword with multiple values" (number " " mark " Vicki Johnson Oct 15 7/406 " thread-indent "\"keyword with multiple values\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Reply-To: vicki at quake.Stanford.EDU (Vicki Johnson) Organization: Stanford From: vicki at quake.Stanford.EDU (Vicki Johnson) Subject: keyword with multiple values Date: 15 Oct 91 22:33:21 GMT What is the recommended keyword approach for describing a parameter with several (say < 100) values? Assume a FITS file is generated from a program run with PARMS=(1.1, 2.2, 3.3). One approach would be FITS keywords of the form PARMS1=1.1, PARMS2=2.2, etc., similar to the NAXIS usage. I've seen mention of hierarchical keywords, but I gather that is discouraged. Are there other approaches? Vicki From thompson at stars.gsfc.nasa.gov Wed Oct 23 10:26:14 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["436" "" "21" "October" "91" "17:06:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " "<4325296 at toto.iv>" "9" "Extensions" "^From:" nil nil "10" "1991102117:06:00" "Extensions" (number " " mark " William Thompson, Oct 21 9/436 " thread-indent "\"Extensions\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Extensions Date: 21 Oct 91 17:06:00 GMT I was told that I could get a list of all registered FITS extensions by sending a mail message to Preben Grosbol at pgrosbol at eso.org. I've been trying this for some time now, and it doesn't work. Does anybody know how to make contact with this person, or how to get this information in another way? I'm particularly interested in getting a description of something called the 'IMAGE' (or possibly 'MATRIX') extension. Bill Thompson From thompson at stars.gsfc.nasa.gov Wed Oct 23 10:26:31 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["804" "" "22" "October" "91" "14:51:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " "<4168418 at toto.iv>" "16" "Re: Extensions" "^From:" nil nil "10" "1991102214:51:00" "Extensions" (number " " mark " William Thompson, Oct 22 16/804 " thread-indent "\"Re: Extensions\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: Re: Extensions Date: 22 Oct 91 14:51:00 GMT In article <21OCT199113063901 at stars.gsfc.nasa.gov>, I wrote... >I was told that I could get a list of all registered FITS extensions by >sending a mail message to Preben Grosbol at pgrosbol at eso.org. I've been >trying this for some time now, and it doesn't work. Does anybody know how to >make contact with this person, or how to get this information in another way? >From some of the responses that I've gotten, I now realize that this sounds like Dr. Grosbol doesn't respond to his mail. I regret this misapprehension; this was not the case. My problem was simply that my mail messages always came back as undeliverable. I expect this is because there is something wrong with the address I have, which is why I asked for help. I apologize for any offense I may have given anyone. Bill Thompson From thompson at stars.gsfc.nasa.gov Wed Oct 23 10:26:49 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["150" "" "21" "October" "91" "19:46:00" "GMT" "William Thompson, code 682.1, x2040" "thompson at stars.gsfc.nasa.gov " "<8042950 at toto.iv>" "6" "IMAGE extension" "^From:" nil nil "10" "1991102119:46:00" "IMAGE extension" (number " " mark " William Thompson, Oct 21 6/150 " thread-indent "\"IMAGE extension\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Organization: NASA/GSFC-Laboratory for Astronomy and Solar Physics News-Software: VAX/VMS VNEWS 1.4-b1 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040) Subject: IMAGE extension Date: 21 Oct 91 19:46:00 GMT Could somebody please post the proposal for the IMAGE extension? I understand there is one available from the IAU office. Thank you, Bill Thompson From dmehring at zia.aoc.nrao.edu Wed Oct 23 10:27:01 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["696" "" "22" "October" "91" "20:40:26" "GMT" "David Mehringer" "dmehring at zia.aoc.nrao.edu " "<6711830 at toto.iv>" "14" "program to copy a FITS image (request)" "^From:" nil nil "10" "1991102220:40:26" "program to copy a FITS image (request)" (number " " mark " David Mehringer Oct 22 14/696 " thread-indent "\"program to copy a FITS image (request)\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Distribution: usa Organization: National Radio Astronomy Observatory, Socorro NM From: dmehring at zia.aoc.nrao.edu (David Mehringer) Subject: program to copy a FITS image (request) Date: 22 Oct 91 20:40:26 GMT Sorry if this is a FAQ, but does someone have a program whose sole purpose is to copy a FITS image at disk level and if so, may I have a copy? I am trying to manipulate images at disk level and am in need of such a program. Since I will be changing the pixel values, a program that also checks for the max and min pixel values and incorporates them into the header would be great. Thanks in advance. -- Dave Mehringer | "Every so often someone comes along and tries dmehring at zia.aoc.nrao.edu | to re-invent the wheel, but usually ends up National Radio Astronomy | with an octogon that has an off-center hole." Observatory, Socorro, NM | -- E.N. Parker (the solar wind guy) From pence at heawk1.gsfc.nasa.gov Thu Oct 24 15:14:53 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2067" "" "23" "October" "91" "16:49:56" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " "<2601167 at toto.iv>" "64" "Re: program to copy a FITS image (request)" "^From:" nil nil "10" "1991102316:49:56" "program to copy a FITS image (request)" (number " " mark " William Pence Oct 23 64/2067 " thread-indent "\"Re: program to copy a FITS image (request)\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Distribution: usa Organization: Goddard Space Flight Center Nntp-Posting-Host: heawk1.gsfc.nasa.gov From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Re: program to copy a FITS image (request) Date: 23 Oct 91 16:49:56 GMT dmehring at zia.aoc.nrao.edu (David Mehringer) writes: >Sorry if this is a FAQ, but does someone have a program whose sole >purpose is to copy a FITS image at disk level and if so, may I have a >copy? I am trying to manipulate images at disk level and am in need >of such a program. Since I will be changing the pixel values, a >program that also checks for the max and min pixel values and >incorporates them into the header would be great. >Thanks in advance. David I am not exactly clear what you want the program to do, but the following fortran program is an example of how you could read a FITS image with 3 calls to the FITSIO subroutine package. -Bill Pence C---------------------------------------------------------------------- program rdimag C simple program to open and read a FITS image C this assumes that the image is in INTEGER*4 format, and is no C larger than 100 X 100 pixels. Would need to change the following C declaration statements if this is not the case... parameter (ndim1=100) parameter (ndim2=100) integer image(ndim1,ndim2),nulval character*40 name integer iunit,rwmode,block,status integer bitpix,naxis,naxes(99),pcount,gcount logical simple,extend,anyflg print *,'enter name of FITS file:' read(*,1000)name 1000 format(a) C use fortran unit number 12 iunit=12 C open the FITS file with readonly access: rwmode=0 call ftopen(iunit,name,rwmode,block,status) C read the important primary array header keywords: call ftgprh(iunit,simple,bitpix,naxis,naxes,pcount,gcount, & extend,status) C should check that this is really a 2-D array, and that the C FITS array size does not exceed the IMAGE array dimensions, C but we will just assume that everything is OK... C assign the value to be used to represent undefined pixels: nulval=-999 C now read the array call ftg2dj(iunit,0,nulval,ndim1,naxes(1),naxes(2), & image,anyflg,status) C Now do whatever you want with the IMAGE array.... end C--------------------------------------------------------------------------- From rlw at stsci.edu Tue Oct 29 17:44:47 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["8637" "" "28" "October" "91" "18:16:22" "GMT" "Rick L. White" "rlw at stsci.edu " "<4273323 at toto.iv>" "152" "image compression" "^From:" nil nil "10" "1991102818:16:22" "image compression" (number " " mark " Rick L. White Oct 28 152/8637 " thread-indent "\"image compression\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Keywords: image compression Organization: Space Telescope Science Institute From: rlw at stsci.edu (Rick L. White) Subject: image compression Date: 28 Oct 91 18:16:22 GMT I got into this discussion late (thanks to Bob Hanisch for directing me to this group and to Don Wells for maintaining the archive); I apologize that this posting is so long. As an introduction, I am the person who developed the image compression technique we will be using to distribute the digitized sky survey plates that were used in the construction of the Guide Star Catalog. There are about 1500 images, each of with 14000x14000 I*2 pixels, so the total data volume is 600 GBytes. For distribution we will compress the images by about a factor of 10, i.e. the compressed data will require about 1.5 bits/pixel. This is, of course, lossy compression -- but the only loss (according both to my analysis and to tests we have done) is that we have thrown away most of the noise in the image. In fact, at 1.5 bits/pixel we are still keeping 1 bit of noise everywhere -- we can compress down to about 0.7 bits/pixel before we start to throw away real information. We will soon be distributing a CD-ROM sampler with 8 different plates (north & south, galactic plane & pole, etc.) compressed at 3 different levels (about 1.5, 0.7, and 0.3 bits/pixel). Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: % In general, compression is tricky - binary files just don't compress % very well. Greg Scott Hennessy (gsh7w at fermi.clas.Virginia.EDU) writes: % As with many things, the answer is "that depends". Don Wells (dwells at fits.cx.nrao.edu) writes: % I agree with both statements. The real issue is what fraction of the % bits of the file are "random"... Don has it exactly right. It is easy to put a lower bound on the data volume required to store an image (thus an upper bound on the compression) if it is to be compressed with no loss: just count the total number of bits of noise in the image. For the GS digitized plates, the noise is typically sigma = 200 DN in each pixel, so lossless compression will always require at least 8.5 bits/pixel (+/- 1 sigma requires 8.5 bits). Not only is this a lower bound to the data volume, but it is actually a pretty good estimate of what can be achieved. The real structure in most astronomical images can be stored with very little information, typically less than 1--2 bits/pixel (even for crowded fields in the galactic plane). If your lossless method requires much more than N+2 bits/pixel, where N is the number of noise bits, then you probably need a better method. An upper bound to the data volume can be calculated from the entropy of the image, S = Sum[f*log2(f)], where the sum is over all different pixel values and f is the fraction of pixels having the given value. This is an upper bound because if there are any correlations between pixels (as there always are in real images), then less than S bits/pixel are required. Note that one can generate a highly incompressible image simply by adding a whole bunch of noise bits to each pixel. One way to do this, for example, is to convert the image to float and divide by a flat field. A flat-fielded R*4 image is very difficult to compress -- if you insist on lossless compression! Of course the key is to allow some loss in the compression. Lossless compression may make sense for low-noise CCD images, but no sensible person cares whether a compressed R*4 image agrees with the original in the last bit. Markus Buchhorn (markus at mso.anu.edu) writes: % ... one way of helping the compression algorithm might be to store the data % as differences from one pixel to the next. If the Universe was well behaved % and skies were truely smooth, the gain would be astounding ... As the noise % increases, the efficiency would decrease. Another advantage (possibly the % main one) is that all the dp/dx values would be closer in number-space, % which means that a trick like compressing high and low bytes separately % would work much better. Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: % ... this is virtually the algorithm IHW used for compressing the % digitized large-scale images. We observed that, for most of our 1600 % images, although the data itself required 2 bytes for storage, the % pixel-to-pixel differences fit in 1 byte. This method is often suggested for compression, but it does not work very well for noisy images. Suppose we have a completely empty image with each pixel = SKY+NOISE. If we take the difference of adjacent pixels, then each pixel has SQRT(2)*NOISE (assuming that the noise is independent and equal in each pixel). Taking differences of consecutive pixels _increases_ the noise, which actually makes the image _harder_ to compress (by about 0.5 bit/pixel). Even when there is real structure in the image, taking differences often does not help. Astronomical images are filled with point sources, so when a signal appears it tends to change very rapidly from one pixel to the next. If you take the derivative of an image you spread the structure over more pixels and make it more complicated, which does not help compression. Smooth objects are pretty rare in astronomical images. I think Markus is right that the main reason this method has been found to be useful is that it shifts the values of pixels so that they are centered around zero. One would do better, though, just to subtract the SKY value directly so that the noise does not get combined. Of course, if SKY varies (i.e. there is structure in the image), this doesn't work and one needs a fancier method. Nonetheless, just taking differences of adjacent pixels is not a very good first step in compressing images. Markus Buchhorn (markus at mso.anu.edu) writes: % But what about a lossy compression technique ? Before anyone jumps on % that, look at the way HST will store their digitised sky survey... % they have achieved quite respectable compression (4:1), and, *they claim* % that photometry is not compromised by this. (Dunno about crowded fields % though ???) Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: % Excellent idea - especially if we can arrange the compression so what's % lost is the noise, not the signal. Still, astronomers are conservative % folks and want to preserve every last bit of noise. Don Wells (dwells at fits.cx.nrao.edu) writes: % The general strategy which I speculate will generally make the maximum % gain is to split the datastream into streams which have homogeneous % statistics, and compress each stream separately... It may be advantageous % to compress even and odd bytes separately for 16-bit data, or use four % streams for 32-bit. Jean-Pierre Veran (veran at cfht.hawaii.edu) writes: % The images are generally very noisy and I have found that I can improve % considerably the compression if I get rid of a certain number of the least % significant bits, then use COMPACT 1.0 to compress the remaining most % significant bits and finally just append the least significant bits to % the file, without trying any compression on them. Yes, the trick is to separate the noise and the signal. The signal is highly compressible, while you are wasting your time if you try to compress the noise. I've tried a number of techniques similar to those mentioned above. The difficulty with just stripping off the bottom N bits is that there is still some signal buried in those bits. If you throw the bits away (lossy compression) you have lost some potentially valuable information; if you write them out directly, you have saved a lot of noise to get a little more information. If you leave more of the noise bits in the signal when you do your split, the signal becomes much harder to compress (e.g. run-length coding fails completely in the presence of even a little noise.) The trick in the method I use -- based on the H-transform, see Fritze et al. Astron.Nachr (1978) 298, 189 and Capaccioli et al. (1988) Astron.Nachr. 309, 69 -- is that it preserves the mean of the image on all scales. Effectively it blocks the image in groups of 1x1, 2x2, 4x4, ... pixels and preserves the average of each such block if it differs signficantly from the average of neighboring blocks. That means that low-surface brightness objects, which are not detected in single pixels but _are_ detected if one averages over blocks, can be preserved even if you throw away the noise. This method manages to keep as much as possible of the information which is buried in the pixel noise without keeping too much of the noise itself. Of course, you can still keep the noise if you want it -- but with this method it is much clearer where to draw the line and stop trying to compress the data. Rick White, rlw at stsci.edu Space Telescope Science Institute From rlw at stsci.edu Tue Oct 29 17:44:54 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["4211" "" "28" "October" "91" "18:18:32" "GMT" "Rick L. White" "rlw at stsci.edu " "<1242799 at toto.iv>" "74" "FITS compression" "^From:" nil nil "10" "1991102818:18:32" "FITS compression" (number " " mark " Rick L. White Oct 28 74/4211 " thread-indent "\"FITS compression\"\n") nil] nil) Newsgroups: alt.sci.astro.fits Keywords: FITS image compression Organization: Space Telescope Science Institute From: rlw at stsci.edu (Rick L. White) Subject: FITS compression Date: 28 Oct 91 18:18:32 GMT A few remarks on the FITS compression proposal: Greg Scott Hennessy (gsh7w at fermi.clas.Virginia.EDU) writes: % It is very common already to have FITS data sets that will not fit % into RAM, and I think that it would be a mistake to rely on a % compression scheme that required reading the whole data set into ram. Doug Tody NOAO/IRAF CCS (tody at noao.edu) writes: % Unless there are good reasons to do otherwise it would be wise to use a % compression techinique which is local in nature ... This is particularly % important if the data is to be randomly accessed at runtime, for example, % when reading a FITS image from a CD-ROM, or from disk. It would also aid in % recovery from data losses. A simple technique would be to compress each % image line independently, using an index to record the offset of each % compressed line. This is an important point. The GS digitized plates are 14000x14000. It would be terrible to have to uncompress the entire plate to get at any of the data. In fact, it would be terrible to have to uncompress an entire 14000 pixel line of data just to extract a small section of the plate! The approach we have taken is to compress the image in blocks (500x500); then one can extract an image section anywhere on the plate by uncompressing only the required blocks. The compression technique we're using is fast enough that I don't think we would bother breaking small images up -- e.g. it takes only 4 secs to decompress a 500x500 image on a Sparc-1. Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: % The second "trick" [of the FITS compression scheme] is that, when you % compress the data, you compress everything, including a FITS header. The % reason for this is that it allows you to compress _any_ FITS data stream, % and the result of the decompression will always be a fully-qualified FITS % data stream, which you can feed directly back into your FITS reader. Any scheme which is appropriate for binary data will not be appropriate for text, so it does not really make sense to think of putting both the header and data through some generic compression tool. (E.g. if you apply your successive differences to text and then feed it to unix compress, I am sure it will not compress as well as the original text.) The beauty of the FITS compression proposal, though, is that one is free to define a compression method that applies one technique to the header (or does nothing to the header) and an entirely different technique to the data. After all, you only have to look for the END record to know when to change horses. Greg Scott Hennessy (gsh7w at fermi.clas.Virginia.EDU) writes: % Table 1 in the Warnock et. al. draft clearly shows what we all know now. % Some data sets compress well (using a generic compress, not referring to any % particular algorithm) and some data sets do not. My conclusion from % Table 1 is that the space savings from different algorithms is a small % fraction of the range of savings possible, hence picking a good but % admittedly not best for all cases algorithm to standarize on is a win % in keeping down software complexity over supporting multiple % compression algorithms to gain 5-10 percent savings. Archie Warnock (warnock at nssdca.gsfc.nasa.gov) writes: % Bang for the buck - a good concept. Computing time goes up % substantially with increasing compression ratio. I hope no one claims % to be able to decide for everyone else what investment level is % sufficient. Actually, deciding how much noise to keep is far more important than computing time. It may be that the best technique for lossy compression is not the best for lossless compression. The JPEG group took this approach -- their lossless method is tacked on after the fact and has little in common with their main (lossy) method. (Incidentally, my method we're using for the GS plates performs well for either lossless or lossy compression.) In any case, I think the decisions to be made can effect the compression achieved at the factor of 2--3 level, not just the 5-10 percent level. I think the idea of having a fair number of methods to choose from is a good one. Rick White, rlw at stsci.edu Space Telescope Science Institute From veran at cfht.hawaii.edu Wed Oct 9 09:37:37 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["6394" "Thu" "3" "October" "1991" "00:09:13" "GMT" "Jean-Pierre Veran" "veran at cfht.hawaii.edu " "<864584 at toto.iv>" "175" "compression of fits" "^From:" nil nil "10" "1991100300:09:13" "compression of fits" nil nil] nil) Newsgroups: alt.sci.astro.fits Keywords: fits, image compression Nntp-Posting-Host: nano.cfht.hawaii.edu Reply-To: veran at cfht.hawaii.edu (Jean-Pierre Veran) Organization: Canada-France-Hawaii Telescope Corporation From: veran at cfht.hawaii.edu (Jean-Pierre Veran) Subject: compression of fits Date: Thu, 3 Oct 1991 00:09:13 GMT The Canada-France-Hawaii Telescope has large format CCDs of 2K by 2K pixels (2 bytes per pixel even if only 15 bits are used at the moment). An average of 100 images are taken each nignt ie almost 1 gigabytes each night. The 8 megabytes images are stored in FITS format on optical disk, after being compressed with the Unix COMPRESS program. The performances of COMPRESS are generally poor so we decided to initiate a research project to find out how it would be possible to improve the compression ratio of our FITS files. I am the one in charge for that. First I tried compression programs available in the public network, (COMPRESS 4.1, COMPACT 1.0, ARC 5.21, LHARC 1.0, ZOO 2.10, FREEZE 2.2, ZIP 0.9) on a bunch of objects, flats and bias images. I have found that the best was generally COMPACT 1.0, using adaptative Huffman coding. The images are generally very noisy and I have found that I can improve considerably the compression if I get rid of a certain number of the least significant bits, then use COMPACT 1.0 to compress the remaining most significant bits and finally just append the least significant bits to the file, without trying any compression on them. The problem is that the optimal number of LSB to get rid of depends on the image and I'm trying to work out a criterion based on simple statistic properties. This seems not very difficult, needs some more calculation but only during compression, not during decompression and it is possible to perform these calculations only on a part of the image, the central part for example. Before compressing the MSB, I tried to use a linear predictor. The results show that the compression ratio is generally slightly better but not always. It is certainly possible to improve the performances of the predictor with a larger neighborhood for example but it will not lead to miracles anyway. I have also tested the previous pixel compression method suggested in the FITS Compression proposal. Well, the algorithm is very simple and the results are pretty good. Sometimes, it is possible to even reduce the size of the obtained file using a compressing programm (COMPRESS 4.1 seems to be the best). But still the results show that better performances can simply be achieved separating the least significants bits from the most significant ones as said above and I do think that some improvements can still be made in that way. You will find below the results of some tests. If anybody is interested, I can mail some more figures. Any comments or suggestions are very welcome. Jean-Pierre Veran Canada-France-Hawaii Telescope Email: veran at cfht.hawaii.edu ------------------------------------------------------------------------ --------- This document gives the results of compression tests of 3 FITS object images, without header. First the SHIFT least significant bits were removed from the image ie Image = Image >> SHIFT has been performed. Then a linear prediction was computed, using the predictor: P(x) = 0.351*A + 0.346*C + 0.292*B B C D A X and each pixel was replaced by its error of prediction X-P(X). Then this image was compressed with an adaptative Huffman coding (Compact 1.0) Finally all the bits first removed were compacted and added to the file without any compression. The tests have been carried out without and with prediction. Size of the original image 2048*2048*2 8388608 RATIO = ----------------------------- = ----------- = ----------- Size of the compressed image KBYTES*1000 KBYTES*1000 <-- indicates a global maximum of the compression ratio. For comparisons, the performances of the old version of COMPRESS and COMPACT 1.0 used alone are reminded. Then the previous pixel compression algorithm has been tried (PREVPIX), followed by a compression of the file with COMPRESS 4.1 (PREVPIX + COMPRESS 4.1) which turned out to be the best compression algorithm for those files. ********************************************** * Source image: 112069o.fits Size = 8388608 * ********************************************** Un. COMPRESS: Ratio = 1.25 KBytes = 6719 COMPACT 1.0 : Ratio = 1.31 KBytes = 6398 PREVPIX: Ratio = 1.53 Kbytes = 5481 PREVPIX + COMPRESS 4.1: Ratio = 1.53 Kbytes = 5481 ### SHIFT + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 1.31 6398 4 1.70 4936 1 1.48 5684 5 1.77 4750 2 1.52 5506 6 1.81 4624 3 1.62 5178 7 1.83 4594 <-- ### SHIFT + PREDICTOR + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 1.40 5996 4 1.77 4737 1 1.57 5326 5 1.83 4564 2 1.60 5253 6 1.89 4444 <-- 3 1.70 4943 7 1.85 4533 ********************************************** * Source image: 111907o.fits Size = 8388608 * ********************************************** Un. COMPRESS: Ratio = 1.79 KBytes = 4696 COMPACT 1.0 : Ratio = 1.87 KBytes = 4485 PREVPIX: Ratio = 1.98 Kbytes = 4250 PREVPIX + COMPRESS 4.1: Ratio = 1.98 Kbytes = 4250 ### SHIFT + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 1.87 4485 4 2.31 3636 1 2.01 4177 5 2.33 3597 <-- 2 2.12 3954 6 2.30 3645 3 2.23 3766 7 2.14 3912 ### SHIFT + PREDICTOR + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 1.95 4304 4 2.34 3580 1 2.09 4005 5 2.38 3528 <-- 2 2.21 3795 6 2.31 3631 3 2.33 3607 7 2.21 3800 ********************************************** * Source image: 121685o.fits Size = 8388608 * ********************************************** Un. COMPRESS: Ratio = 2.86 KBytes = 2937 COMPACT 1.0 : Ratio = 3.02 KBytes = 2777 PREVPIX: Ratio = 1.97 Kbytes = 4251 PREVPIX + COMPRESS 4.1: Ratio = 2.97 Kbytes = 2825 ### SHIFT + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 3.02 2777 4 3.19 2627 1 3.19 2630 5 3.06 2737 2 3.32 2523 6 2.58 3250 3 3.35 2501 <-- 7 2.23 3763 ### SHIFT + PREDICTOR + COMPACT 1.0 ### SHIFT RATIO KBYTES SHIFT RATIO KBYTES ----- ----- ------ ----- ----- ------ 0 2.97 2827 4 3.15 2665 1 3.13 2673 5 3.07 2736 2 3.23 2589 <-- 6 2.59 3243 3 3.20 2619 7 2.23 3753 Jean-Pierre Veran Canada-France-Hawaii Telescope Email: veran at cfht.hawaii.edu From koffley at nrlvx1.nrl.navy.mil Wed Oct 9 09:36:44 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1829" "" "1" "October" "91" "10:29:29" "GMT" "koffley at nrlvx1.nrl.navy.mil" "koffley at nrlvx1.nrl.navy.mil" "<3732164 at toto.iv>" "37" "Re: FITS V3.0 available from NRLREAD/NEXT" "^From:" nil nil "10" "1991100110:29:29" "FITS V3.0 available from NRLREAD/NEXT" nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: NRL SPACE SYSTEMS DIVISION From: koffley at nrlvx1.nrl.navy.mil Subject: Re: FITS V3.0 available from NRLREAD/NEXT Date: 1 Oct 91 10:29:29 GMT In article , pence at heawk1.gsfc.nasa.gov ( William Pence) writes: >In general I would discourage users from obtaining 2nd hand copies of the >FITSIO software because one may not get the latest version. As a case in >point, the new version 3.01 of FITSIO, which fixes a relatively minor bug, >was released just last Friday. Currently there a 2 ways to receive the >most recent version of FITSIO: > >1) by anonymous ftp from tetra.gsfc.nasa.gov in subdirectory pub/fitsio3 > >2) over the SPAN network (e.g., with DECNET copy) from > > NDADSA::HEASARC:[EXOSAT.XANADU.FITSIO.VERSION3] > >If any users have difficulty accessing the files by either of these >methods, then they should send me a message. > Bill, I wasn't trying to subvert GSFC's efforts here. The FITS package was placed here as a service to our local community who for the most part are not too saavy about the network at large and even less so about FTP or DECNET. I only posted to the group because I figured there might be more of the same out in the general population. Everyone seems to know how to send e-mail. BTW, was there a general announcement that I missed re: 3.01 ?? I didn't see it in this group. Is there some forum that such an announcement would take place without having to FTP to GSFC to find out ? I only ask because the only reason I would have known there was a 3.00 was the announcement in this group. --- \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ < Joe Koffley KOFFLEY at NRLVAX.NRL.NAVY.MIL > < Naval Research Laboratory KOFFLEY at CCF.NRL.NAVY.MIL > < Space Systems Division AT&T : 202-767-0894 > \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ From pence at heawk1.gsfc.nasa.gov Wed Oct 9 09:36:53 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["888" "" "1" "October" "91" "15:54:16" "GMT" " William Pence" "pence at heawk1.gsfc.nasa.gov " "<1318779 at toto.iv>" "17" "Re: FITS V3.0 available from NRLREAD/NEXT" "^From:" nil nil "10" "1991100115:54:16" "FITS V3.0 available from NRLREAD/NEXT" nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: Goddard Space Flight Center Nntp-Posting-Host: heawk1.gsfc.nasa.gov From: pence at heawk1.gsfc.nasa.gov ( William Pence) Subject: Re: FITS V3.0 available from NRLREAD/NEXT Date: 1 Oct 91 15:54:16 GMT koffley at nrlvx1.nrl.navy.mil writes: >BTW, was there a general announcement that I missed re: 3.01 ?? I didn't see >it in this group. Is there some forum that such an announcement would take >place without having to FTP to GSFC to find out ? I only ask because the only >reason I would have known there was a 3.00 was the announcement in this >group. I am unsure of the best way to advertise new releases of the FITSIO software, especially since minor bug fixes or enhancements will probably be released nearly every week for the next month or 2. Would users prefer to see an announcement here, or should they be expected to check the distribution directories (by FTP or DECNET) to see if there is a newer version? In any case, I would post a message here whenever a significant new release was available. < Bill Pence, HEASARC pence at tetra.gsfc.nasa.gov or LHEAVX::PENCE > From thompson at stars.gsfc.nasa.gov Wed Oct 9 09:37:06 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1393" "" "1" "October" "91" "15:36:52" "GMT" "Bill Thompson" "thompson at stars.gsfc.nasa.gov " "<746049 at toto.iv>" "26" "Proposed TAXISnnn convention" "^From:" nil nil "10" "1991100115:36:52" "Proposed TAXISnnn convention" nil nil] nil) Newsgroups: alt.sci.astro.fits Reply-To: thompson at stars.gsfc.nasa.gov Organization: NASA Goddard Space Flight Center - Greenbelt, MD, USA News-Software: VAX/VMS VNEWS 1.3-4 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (Bill Thompson) Subject: Proposed TAXISnnn convention Date: 1 Oct 91 15:36:52 GMT "Multidimensional Table" Convention One way to look at binary tables is as way of expressing data structures. In this view, each row is an element in a array whose data type is a structure, and each column represents a member of this structure. The basic binary tables convention expresses this structure array as one dimensional. However, one could also organize these structure elements as a multidimensional array. For instance, if a binary table had 15 rows, then one could also organize this as a 5 x 3 array. The "Multidimensional table" convention would express this organization using the optional keywords TAXIS, TAXIS1, TAXIS2, etc., which are defined in the manner of NAXIS, NAXIS1, NAXIS2, etc. The product of the values of the keyword variables TAXIS1 x TAXIS2 x ... x TAXISn is required to be equal to the total number of rows in the binary as given by NAXIS2. The example discussed above would have the following values for these keywords. NAXIS2 = 15 / The table has 15 rows TAXIS = 2 / These rows for a 2D array TAXIS1 = 5 / The first dimension is 5 TAXIS2 = 3 / The second dimension is 3 The adherence to this convention will be indicated by the presence of the keyword TAXIS, along with the associated TAXISnnn keywords. From thompson at stars.gsfc.nasa.gov Wed Oct 9 09:37:02 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2559" "" "1" "October" "91" "15:37:52" "GMT" "Bill Thompson" "thompson at stars.gsfc.nasa.gov " "<1116841 at toto.iv>" "43" "Proposed TDESCnnn convention" "^From:" nil nil "10" "1991100115:37:52" "Proposed TDESCnnn convention" nil nil] nil) Newsgroups: alt.sci.astro.fits Reply-To: thompson at stars.gsfc.nasa.gov Organization: NASA Goddard Space Flight Center - Greenbelt, MD, USA News-Software: VAX/VMS VNEWS 1.3-4 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (Bill Thompson) Subject: Proposed TDESCnnn convention Date: 1 Oct 91 15:37:52 GMT "Dimension Label" Convention This convention describes a way to assign labels to the dimensions of arrays in a binary table. The purpose of this convention is to provide a simple standard for documenting the interrelationships between arrays in a binary table. This convention operates by assigning the keyword TDESCnnn to a column nnn which contains an array with one or more dimensions. This keyword would have the format TDESCnnn='(label1,label2,...)' where label1, etc. are the dimension labels. There would be one dimension label for each fundamental dimension. The labels are character strings of up to eight character strings. The same rules that apply to user defined keywords also apply to dimension labels. Adherence to this convention is indicated by the presence of the keywords TDESCnnn. Optional documentation can be assigned to the dimension labels in one of two ways. The simplest way is to have a keyword with the same name as the dimension label. This keyword would take a character string value which would describe the meaning of the dimension label. A second approach would be to have a separate FITS extension with the keyword EXTNAME='extname.labeln', where extname is the value of EXTNAME in the current extension, and labeln is the dimension label. The structure of this associated extension is left up to the user. It is not required that the dimension label be documented with either of these two schemes. For instance, suppose that column 1 in a binary table contains a 100x100x50 array, where the first two dimensions represent spatial position in right ascension and declination, and the third dimension represents time. Column 2 contains an array which depends only on RA and DEC, and the third column contains an array depending only on time. The binary tables extension header could then contain the following entries: TFORM1 = '500000I' / 16 bit integers TDIM1 = '(100,100,50)' / Dimensions TDESC1 = '(RA,DEC,TIME)' / Dimension labels TFORM2 = '10000I ' / 16 bit integers TDIM2 = '(100,100)' / Dimensions TDESC2 = '(RA,DEC)' / Dimension labels TFORM3 = '50I ' / 16 bit integers TDESC3 = '(TIME) ' / Dimension label RA = 'Right Ascension in 1 arcsec inc.' / Meaning of RA DEC = 'Declination in 1 arcsec increments' / Meaning of DEC TIME = 'Time in 10 second increments' / Meaning of TIME From thompson at stars.gsfc.nasa.gov Wed Oct 9 09:37:10 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2960" "" "1" "October" "91" "15:38:43" "GMT" "Bill Thompson" "thompson at stars.gsfc.nasa.gov " "<1557667 at toto.iv>" "60" "Proposed TNDIMnnn convention" "^From:" nil nil "10" "1991100115:38:43" "Proposed TNDIMnnn convention" nil nil] nil) Newsgroups: alt.sci.astro.fits Reply-To: thompson at stars.gsfc.nasa.gov Organization: NASA Goddard Space Flight Center - Greenbelt, MD, USA News-Software: VAX/VMS VNEWS 1.3-4 Nntp-Posting-Host: stars.gsfc.nasa.gov From: thompson at stars.gsfc.nasa.gov (Bill Thompson) Subject: Proposed TNDIMnnn convention Date: 1 Oct 91 15:38:43 GMT "Variable Dimensions" Convention This convention describes how columns in a binary table not only can contain multidimensional arrays as its elements, but also how the number and size of the dimensions can be different for each row of the table. The convention works by storing the dimensions of an array stored in one column (nnn) in another array stored in a second column (mmm). The format of this second description array is discussed below. The interrelationship between the data arrays (column nnn) and the description arrays (column mmm) is indicated by the use of the optional keyword "TNDIMnnn=mmm". Adherence to this convention will be indicated by the presence of the TNDIMnnn keyword. The description array used by this convention is a 16 bit integer array with the following format: the first element contains the number of dimensions in the data array being described; the second element contains the size of the most rapidly varying dimension of the data array; the third the size of the next most rapidly varying dimension; and so forth up to the total number of dimensions. Any extra elements in the description array after the final dimension parameter will be ignored, and should be padded with zeros. The product of the dimension sizes must be less than or equal to the size of the data array as given by the keyword TFORMnnn. The number of dimensions can be zero to signal that no data is stored in the data array for that row in the binary table. If multiple columns in the binary table represent arrays with the same dimensional structure, then they can all refer to the same description array. The "Variable length array" facility could also be used with this convention. Either the data array or description array, or both, could be defined as variable length arrays. As an example, suppose that both columns 3 and 4 contain a series of multidimensional data arrays of different sizes, and that the associated array sizes are in column 5. The header could contain the following keywords and values: TFORM3 = '100I ' / 16 bit integers TNDIM3 = 5 / Dimensions contained in column 5 TFORM4 = '100I' / 16 bit integers TNDIM4 = 5 / Dimensions contained in column 5 TFORM5 = '4I ' / Dims. stored as 16 bit integers The data arrays in columns 3 and 4 could then contain up to three dimensions, one less than the size of the array in column 5. Valid values of the array stored in column 5 can be represented symbolically as (1,100) (2,10,10) (2,20,5) (3,5,5,2) (3,4,4,4) even though the last example adds up to only 64 points and not 100. However, the description array (3,5,5,5) would not be valid because this requires 5 x 5 x 5 = 125 elements, which is bigger than the 100 elements allocated to the arrays in columns 3 and 4. From tody at noao.edu Wed Oct 9 09:37:14 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1991" "" "1" "October" "91" "16:24:19" "GMT" "Doug Tody" "tody at noao.edu " "<2157090 at toto.iv>" "40" "Re: Proposed TAXISnnn/TDESCnnn/TNDIMnnn conventions" "^From:" nil nil "10" "1991100116:24:19" "Proposed TAXISnnn/TDESCnnn/TNDIMnnn conventions" nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: tody at noao.edu (Doug Tody) Subject: Re: Proposed TAXISnnn/TDESCnnn/TNDIMnnn conventions Date: 1 Oct 91 16:24:19 GMT Bill, I suppose any number of conventions can be proposed and tried. Since conventions are not part of the FITS standard one is free to try anything in this regard. For the sake of argument, I will offer some comments on your proposals. TAXISnnn The one-dimensional array-of-records format upon which binary tables is based was not chosen arbitrarily, it is what is used in a relational database. There is a wealth of experience with this model, not only in commercial databases, but in the existing "table" implementations found in various astronomical software systems. Your N-dimensional array of structures, on the other hand, is in my opinion a fairly arbitrary generalization of the simple table, and does not justify departing from the relational model. If we want to abandon the relational model and propose a database with more complex structures then there are no end of ways this can be done - and currently, no standards (this is a hot area of research currently). TDESCnnn You are starting to reinvent world coordinate systems. See the paper by Hanisch et.al. describing a proposed convention for representing world coordinate systems in FITS. In the general case this is a very difficult problem. TNDIMnnn This is a good idea (allowing the array dimension to be specified separately for each record). A generalization of the TDIMnnn convention along these lines is probably needed. This discussion illustrates further why Appendix A is a convention and not part of the core proposal. These proposed conventions would all be layered upon the basic binary tables format. Ultimately many such conventions are likely to be developed to represent different types of data. Discussion of the basic binary tables format is also welcome. -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From tody at noao.edu Wed Oct 9 09:38:03 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2317" "" "4" "October" "91" "21:37:17" "GMT" "Doug Tody" "tody at noao.edu " "<2720600 at toto.iv>" "40" "Re: Maximum blocking factor" "^From:" nil nil "10" "1991100421:37:17" "Maximum blocking factor" nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: tody at noao.edu (Doug Tody) Subject: Re: Maximum blocking factor Date: 4 Oct 91 21:37:17 GMT >From article <9110031830.AA05079 at fits.cx.nrao.edu>, by CUR%STARLINK.RUTHERFORD.AC.UK at VTVM2.CC.VT.EDU (Malcolm J. Currie): > In the IRAF 2.9.1 patch of Sept. 6 there is an `enhancement' that makes > the maximum FITS blocking factor 22 rather than 10. In my humble > opinion this is a mistake. I think that the importance of > transportability far outweighs the efficiency gain. Users should be > protected from shooting themselves (or the recipients of the > non-standard tape) in the foot. The program is capable of the higher blocking factor but will issue a stern warning message that the tape is non standard and may be unreadable elsewhere if the user attempts to do this. Obviously, any user producing a tape to be read elsewhere should produce a standard FITS tape. FITS is not always used to produce data to be read elsewhere. > On the question of increasing the maximum blocking factor for fixed- > block sequential media, I'm not convinced that we need blocksizes > greater than 28800 bytes for Exabyte... The blocking factor has little effect on tape usage for either Exabyte or DAT since the drives are intelligent and will reblock the data and store it on the tape in fixed size blocks. The only wasted storage is that required to pad the last physical device record to accomodate the logical block size specified by the client. In principle a larger block size could aid streaming but in tests I have performed the block size has had little effect on the sustaiied transfer rate. The larger block size is useful mainly for 6250 bpi reel tape tape and cartridge tape. 6250 bpi tapes require more storage for a smaller blocking factor. In the case of cartridge tapes this doesn't matter but the larger block can aid streaming (this is best dealt with at a lower level than FITS however). Yes, the huge (2 Mb) file mark associated with current Exabyte drives is a serious problem if you write many small FITS files. If you are going to write many small files you are probably better off with DAT, which has a small file mark and which can position to any file very quickly. -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From sla at fast.ucsc.edu Wed Oct 9 09:38:09 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1574" "" "4" "October" "91" "22:39:32" "GMT" "Steve Allen" "sla at fast.ucsc.edu " "<7221301 at toto.iv>" "27" "Exabyte tape marks (was Re: Maximum blocking factor)" "^From:" nil nil "10" "1991100422:39:32" "Exabyte tape marks (was Re: Maximum blocking factor)" nil nil] nil) Newsgroups: alt.sci.astro.fits Summary: many device drivers are immature or incomplete Keywords: 8-mm Exabyte tape EOF marks Organization: UCO/Lick Observatory From: sla at fast.ucsc.edu (Steve Allen) Subject: Exabyte tape marks (was Re: Maximum blocking factor) Date: 4 Oct 91 22:39:32 GMT In article <9110031830.AA05079 at fits.cx.nrao.edu> CUR%STARLINK.RUTHERFORD.AC.UK at VTVM2.CC.VT.EDU (Malcolm J. Currie) writes: > More of a limitation on Exabyte is the >maximum number of files caused by the huge tape marks (equivalent to >about 2Mb). You cannot get more than 320 files on a labelled cartridge >or 950 on an unlabelled cartridge. >I was intrigued by Richard Stover's findings. Are there no inter-block >gaps on Exabyte? >Malcolm Currie Janet: CUR at UK.AC.RL.STAR >Starlink SPAN : RLVAD::CUR Richard Stover has complete control over his Exabyte, including the ability to write the short file marks (as opposed to the default long file marks). The short file marks take up only ~300 kbytes and in addition to being short, they are faster to write. This is extremely important for some of the rapid-readout of large CCDs done at Lick. All Exabyte hardware should be able to write these short marks; the deficiency usually lies in the device driver software of the OS. Richard Stover's data acquisition system actually puts a limit of 1600 FITS files per Exabyte tape to prevent having too much valuable data on a single easy-to-lose-or-destroy tape. _______________________________________________________________________________ Steve Allen | | sla at helios.ucsc.edu UCO/Lick Observatory | This space for rent. | If the UC were opining, Santa Cruz, CA 95064 | | it wouldn't tell me. From tody at noao.edu Wed Oct 9 09:38:16 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1558" "" "5" "October" "91" "07:58:55" "GMT" "Doug Tody" "tody at noao.edu " "<7645480 at toto.iv>" "28" "Re: Exabyte tape marks (was Re: Maximum blocking factor)" "^From:" nil nil "10" "1991100507:58:55" "Exabyte tape marks (was Re: Maximum blocking factor)" nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: National Optical Astronomy Observatories, Tucson, AZ, USA From: tody at noao.edu (Doug Tody) Subject: Re: Exabyte tape marks (was Re: Maximum blocking factor) Date: 5 Oct 91 07:58:55 GMT >From article <21756 at darkstar.ucsc.edu>, by sla at fast.ucsc.edu (Steve Allen): > Richard Stover has complete control over his Exabyte, including the > ability to write the short file marks (as opposed to the default > long file marks). The short file marks take up only ~300 kbytes > and in addition to being short, they are faster to write. The Exabyte can write two types of filemarks: short filemarks (about 0.5 Mb) and long filemarks (2.2 Mb). The difference is that the long filemark includes a a gap allowing the filemark to be overwritten. So long as one only writes to a tape at EOT the difference is not important. Whether or not one makes use of short filemarks depends upon the driver used. The Sun SCSI tape driver, at least to date, writes only long filemarks. Most people with Suns use the Sun st driver since it comes with SunOS. Better drivers can be found from third party vendors. ApUNIX sells such a driver, although we have not tried it here (we do use their DAT driver). Features such as short filemarks for Exabyte or fast file seeks for DAT may not be available with a poor driver. If I recall correctly DAT filemarks are 130 Kb or so... These drives employ considerable data buffering internally (246 Kb in the case of Exabyte) which is why the block size has little effect on the sustained transfer rate. -- Doug Tody, National Optical Astronomy Observatories, Tucson AZ, 602-325-9217 UUCP: {arizona,decvax,ncar}!noao!tody or uunet!noao.edu!tody Internet: tody at noao.edu SPAN/HEPNET: NOAO::TODY (NOAO=5355) From dwells at fits.cx.nrao.edu Tue Oct 1 10:50:56 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["3891" "Tue" "1" "October" "1991" "14:40:40" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " nil "82" "Draft Proposal re Fixed-block Sequential Media." "^From:" nil nil "10" nil nil nil nil] nil) X-VM-Summary-Format: "%n %*%a %-17.17F %-3.3m %2d %4l/%-5c %I\"%s\"\n" X-VM-Labels: nil X-VM-VHeader: ("Resent-" "From:" "Sender:" "To:" "Apparently-To:" "Cc:" "Subject:" "Date:") nil X-VM-Bookmark: 2 Newsgroups: alt.sci.astro.fits Organization: National Radio Astronomy Observatory From: dwells at fits.cx.nrao.edu (Don Wells) Subject: Draft Proposal re Fixed-block Sequential Media. Date: Tue, 1 Oct 1991 14:40:40 GMT This draft proposal has been placed in the anonymous-FTP server on fits.cx.nrao.edu [192.33.115.8] in directory FITS/doc: rw-r--r-- 1 dwells vlb 2183 Oct 1 09:51 blocking90.txt This is a short file, and so I am appending it to this announcement. FITS is a bitstream data format which is media-independent (contrary to the impression given by the Wells-Greisen-Harten Basic FITS paper of 1981). FITS *logical* records have length 2880 bytes. Blocking is regarded as being a media-dependent matter, not a part of the FITS standard per se. For 9-tk tapes the FITS community has agreed to use blocking factors between 1 and 10 (28800 bytes/block) inclusive for FITS files. Other types of media need to be separately negotiated. During the past few years it has become clear that 9tk tapes are being superceded by several types of cartridge media. This text proposes rules for several of these media. This draft has been under consideration by the FITS committees for about two years, and has not yet been endorsed by them. I encourage everyone interested in FITS to consider the implications carefully. For example, what would be the increase in efficiency if DATs and/or Exabytes were allowed to have blocking factor greater than 10, and do we want that efficiency enough to change our software which already supports 28800-byte blocks? Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N =-=-=-=-=-=-=-=-=-=-=-=-=-=-= cut here =-=-=-=-=-=-=-=-=-=-=-=-=-=-= Draft Proposal for Blocking of Fixed-block Sequential Media. and bitstream Devices. P.Grosbol and D.Wells 1990-August-10 Introduction A number of high density storage media (e.g. optical disks and helical scan tapes) is of significant interest of exchange of large volumes of astronomical data in FITS format. Many controllers and devices for these media can only access data in blocks of fixed length, typically 2**n bytes. Thus, a special blocking agreement for FITS files is required for this type of media. Draft Proposal for fixed-block media. For fixed-block sequential media, FITS files consisting of an integer number of 2880 byte logical records should be regarded as a bitstream written out with the fixed blocking size of the media with the last block being padded out with zeros to the length of the fixed blocks. Reading an incomplete FITS logical record should be regarded as an end-of-file. This proposal applies to optical disks (accessed as a sequential set of records), QIC format 1/4inch cartridge tapes and Local Area Networks. The fixed-block blocking proposal conforms to the general rules for blocking of FITS files (Grosbol et al. 1988, Astron. Astrophys. Suppl. 73, p359) using a blocking factor of 2**n/2880 for a media with a fixed block size of 2**n bytes. Draft Proposal for bitstream devices. For bitstream devices, FITS files should be written with a blocking factor of one i.e. with fixed blocks of 2880 bytes corresponding to the logical record size. This proposal applies to FITS files written to logical file systems. Draft Proposal for variable-block media. For variable block length sequencial media, FITS files may be written with an interger blocking factor between 1 and 10 including as specified in the blocking agreement for 1/2 inch 9 track tapes. This proposal applies to FITS files written to DDS/DAT 4mm cartridge tapes and 8mm cartridge tape (Exabyte). From sla at fast.ucsc.edu Tue Oct 1 20:51:41 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1984" "" "1" "October" "91" "22:07:37" "GMT" "Steve Allen" "sla at fast.ucsc.edu " nil "35" "Re: Draft Proposal re Fixed-block Sequential Media." "^From:" nil nil "10" nil nil nil nil] nil) Newsgroups: alt.sci.astro.fits Summary: efficiency gains are not large Keywords: FITS, exabyte Organization: UCO/Lick Observatory From: sla at fast.ucsc.edu (Steve Allen) Subject: Re: Draft Proposal re Fixed-block Sequential Media. Date: 1 Oct 91 22:07:37 GMT In article <9110011440.AA29235 at fits.cx.nrao.edu> dwells at fits.cx.nrao.edu (Don Wells) writes: >This draft has been under consideration by the FITS committees for >about two years, and has not yet been endorsed by them. I encourage >everyone interested in FITS to consider the implications carefully. >For example, what would be the increase in efficiency if DATs and/or >Exabytes were allowed to have blocking factor greater than 10, and do >we want that efficiency enough to change our software which already >supports 28800-byte blocks? > Richard Stover at Lick observatory has investigated the issues of blocksize on Exabyte tapes. Richard is the author and maintainer of the Lick data acquisition system. His observations are included below. -------------------- Years ago I looked into Exabyte efficiency with respect to blocking. One can query the Exabyte for the number of 1024-byte blocks remaining on the tape. Therefore, one can write some number of n-byte variable length blocks and then ask the Exabyte how many 1024-byte blocks are left. >From this one can easily deduce the tape efficiency. My results were: 2048 or 4096 byte block are 100% efficient. 2880 byte blocks are 93% efficient. 28880 byte blocks are 96.7% efficient. Given these small efficiency differences I have chosen to write 2880-byte records, although the software was designed to handle a blocking factor up to 10. One reason to write 28800-byte records instead of 2880-byte records might be that the number of system calls is reduced by a factor of 10. But machines are so fast these days that this didn't seems to me like an overriding concern. Richard Stover -------------- _______________________________________________________________________________ Steve Allen | | sla at helios.ucsc.edu UCO/Lick Observatory | This space for rent. | If the UC were opining, Santa Cruz, CA 95064 | | it wouldn't tell me. From CUR%STARLINK.RUTHERFORD.AC.UK at VTVM2.CC.VT.EDU Fri Oct 4 07:18:03 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["285" "Fri" "4" "October" "1991" "10:04:00" "GMT" "\"Malcolm J. Currie\"" "CUR%STARLINK.RUTHERFORD.AC.UK at VTVM2.CC.VT.EDU" nil "7" "Optimum blocking factor for Exabyte" "^From:" nil nil "10" nil nil nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: National Radio Astronomy Observatory From: "Malcolm J. Currie" Subject: Optimum blocking factor for Exabyte Date: Fri, 4 Oct 1991 10:04:00 GMT Since the efficiency differences on Exabyte are due to the padding of data blocks to a multiple of 1024-bytes, the optimum FITS blocking factor is 6. Malcolm Currie Janet: CUR at UK.AC.RL.STAR Starlink SPAN : RLVAD::CUR From forveill at gag.observg.grenet.fr Tue Oct 8 09:11:39 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["2916" "Tue" "8" "October" "1991" "08:33:01" "GMT" "Thierry Forveille" "forveill at gag.observg.grenet.fr " nil "54" "Blocking factor" "^From:" nil nil "10" nil nil nil nil] nil) Newsgroups: alt.sci.astro.fits Path: cv3.cv.nrao.edu!mail-to-news-gateway Message-ID: <199110080835.AA17688 at corton.inria.fr> Organization: National Radio Astronomy Observatory Lines: 54 From: forveill at gag.observg.grenet.fr (Thierry Forveille) Sender: news at nrao.edu Subject: Blocking factor Date: Tue, 8 Oct 1991 08:33:01 GMT >>From article <9110031830.AA05079 at fits.cx.nrao.edu>, by CUR%STARLINK.RUTHERFORD.AC.UK at VTVM2.CC.VT.EDU (Malcolm J. Currie): >> In the IRAF 2.9.1 patch of Sept. 6 there is an `enhancement' that makes >> the maximum FITS blocking factor 22 rather than 10. In my humble >> opinion this is a mistake. I think that the importance of >> transportability far outweighs the efficiency gain. Users should be >> protected from shooting themselves (or the recipients of the >> non-standard tape) in the foot. > >The program is capable of the higher blocking factor but will issue a stern >warning message that the tape is non standard and may be unreadable >elsewhere if the user attempts to do this. Obviously, any user producing a >tape to be read elsewhere should produce a standard FITS tape. FITS is not >always used to produce data to be read elsewhere. > >> On the question of increasing the maximum blocking factor for fixed- >> block sequential media, I'm not convinced that we need blocksizes >> greater than 28800 bytes for Exabyte... > >The blocking factor has little effect on tape usage for either Exabyte or >DAT since the drives are intelligent and will reblock the data and store >it on the tape in fixed size blocks. The only wasted storage is that >required to pad the last physical device record to accomodate the logical >block size specified by the client. In principle a larger block size >could aid streaming but in tests I have performed the block size has had >little effect on the sustaiied transfer rate. > >The larger block size is useful mainly for 6250 bpi reel tape tape and >cartridge tape. 6250 bpi tapes require more storage for a smaller blocking >factor. In the case of cartridge tapes this doesn't matter but the larger >block can aid streaming (this is best dealt with at a lower level than FITS >however). Even with stern warning messages, this alteration of the maximum blocking factor remains a violation of the FITS standard. If the IRAF group wants to stick to their "enhancement", they should simultaneously set SIMPLE to F, and shouldn't call the result a FITS file. A standard is a standard! When the IRAF group vetoed hierarchical keywords a few years ago, the radio community kept complying to the unmodified FITS agreement, even though most of these tapes will never enter a non-radio package. Even though many FITS tapes aren't meant to be read elsewhere in principle, some of them do turn out to. Even worse, some tapes will not be read elsewhere, but they will be read later, and some of them will survive IRAF. The fact that this modification is only justified by a marginal capacity enhancement on an already obsolescent medium makes it even harder to justify. Who cares about 20% on a 6250 bpi reel when DATs and Exabyte drives are now cheaper than any tape drive? Thierry Forveille Observatoire de Grenoble forveill at frgag51.bitnet From dwells at fits.cx.nrao.edu Tue Oct 22 09:44:32 1991 Status: RO X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil] ["1157" "Tue" "22" "October" "1991" "12:28:42" "GMT" "Don Wells" "dwells at fits.cx.nrao.edu " nil "23" "XTENSION='IMAGE' text" "^From:" nil nil "10" nil nil nil nil] nil) Newsgroups: alt.sci.astro.fits Organization: National Radio Astronomy Observatory From: dwells at fits.cx.nrao.edu (Don Wells) Subject: XTENSION='IMAGE' text Date: Tue, 22 Oct 1991 12:28:42 GMT I asked J. Ramon Munoz to supply a machine-readable version of the IUE 'IMAGE' proposal, and he has done so, and I have put his text into the anonymous-FTP server on fits.cx.nrao.edu [192.33.115.8] in directory FITS/doc: -rw-r--r-- 1 dwells vlb 6043 Oct 22 08:17 image_extension.txt This is not the published version. It is "a portion of an internal note (1988) where a short description of the IMAGE Extension is given". NOTE: The XTENSION='IMAGE' proposal is still unofficial. I have heard that the European FITS committee asked the IUE group to pick another name so that 'IMAGE' could be reserved for a possible universal agreement in the FITS community. This would be analogous to the history of binary tables; originally they were to be called '3DTABLE', but at the request of the committee members the prototype implementation was called 'A3DTABLE'. Donald C. Wells Associate Scientist dwells at nrao.edu National Radio Astronomy Observatory +1-804-296-0277 520 Edgemont Road Fax= +1-804-296-0278 Charlottesville, Virginia 22903-2475 USA 78:31.1W, 38:02.2N