Subject: Magic numbers, was Re: Where is there a FITs ftp site?

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Thu, 11 May 1995 09:40:21 +0200 (MET DST)

On 10 May 1995, William Thompson wrote:

> Peter Barvoets <zardoz at albany.net> writes:
> >I have a question perhaps you can help me with.  When I try to get a fits 
> >image from a perticular site, the "image" comes as a text of odd 
  [...]

> There are several ways to take care of this.  One way is to modify the way that
> the computer guesses the file type.  I understand that in Unix this involves
> modifying the file /etc/magic, but am unfamiliar with the details.

  I enclose what I have recorded of an exchange about this topic occurred
  on sci.data.format. I hope it can be of some use (I played with the magic
  file some time ago but not for this). BTW the magic file is not always
  on /etc, e.g. on Ultrix is on /usr/lib/file.

---------------------------------------------------------------------------- 
Lucio Chiappetti - IFCTR/CNR     | Ma te' vugl' da' quost avis a ti' Orsign  
via Bassini 15 - I-20133 Milano  | Buttet rabios intant te se' pisnign       
Internet: LUCIO at IFCTR.MI.CNR.IT  |                                           
Decnet:   IFCTR::LUCIO           |             (Rabisch, II 46, 119-120)     
---------------------------------------------------------------------------- 

Article: 219 of sci.data.formats
Newsgroups: sci.data.formats
Path: sparc10!ghost.dsi.unimi.it!rpi!gatech!howland.reston.ans.net!spool.mu.edu!umn.edu!lynx.unm.edu!Mr-Hyde.aoc.nrao.edu!laphroaig!cflatter
From: cflatter at nrao.edu (Chris Flatters)
Subject: Re: Data format 'signatures' for the FAQ
Message-ID: <1993Aug26.192719.13210 at Mr-Hyde.aoc.nrao.edu>
Sender: news at Mr-Hyde.aoc.nrao.edu
Reply-To: cflatter at nrao.edu
Organization: NRAO
References: <DWELLS.93Aug26120427 at fits.cv.nrao.edu>
Date: Thu, 26 Aug 93 19:27:19 GMT
Lines: 46

In article 93Aug26120427 at fits.cv.nrao.edu, dwells at fits.cv.nrao.edu (Don Wells) writes:
>The proposed FAQ for sci.data.formats should address the problem of
>how to recognize the format of an arbitrary data file. It will never
>be possible to provide a definitive answer to this question, but it is
>practical to provide a set of heuristic classification rules, because
>every data format has a characteristic "signature" which can be
>recognized by a simple test applied to the first part of a file.
>
>The data format classification rules should be expressed in the form
>of a portable program. As a start in this direction I have produced a
>prototype.  File fits.cv.nrao.edu:/pub/signatur.tar.Z includes the
>source for signatur.c (which I append), a Makefile, one sample binary
>data file and the output from a test run.  Fetch the file with
>anonymous-FTP in binary mode, then do commands 'uncompress
>signatur.tar.Z', 'tar xvf signatur.tar' and 'make'. The make will
>compile signatur.c and will execute a series of test cases which
>demonstrate the four possible error exits plus the two levels of
>output for data file tests.

Of course system administrators of UNIX systems could be be persuaded to
add the signatures to the /etc/magic file used by the find(1) command.
The following patterns will recognize a FITS data file and distinguish
between the 5 FITS data types.  If you want to use these you may want to
cut and paste them from this message since blanks and tabs are significant.

# Magic number patterns for FITS data files
0       string          SIMPLE  =                    T  FITS
>109    string          8                8-bit integer data
>108    string          16               16-bit integer data
>107    string           32              32-bit integer data 
>107    string          -32              32-bit IEEE floating-point data 
>107    string          -64              64-bit IEEE floating-point data 

I assume that HDF, netCDF and others also have distinguishing patterns.

OpenWindows 3.x users can also make the filemanager recognise FITS files
by registering the "SIMPLE  =                  T" string with the binder.
This would allow a distinctive icon to be associated with FITS files and/or
a program (eg SAOimage) to be run if the icon is double clicked.

NB: file(1) is a POSIX.2a command but POSIX.2a does not specify how file(1)
should recognise file types so although a POSIX.2a system must have file(1)
it need not necessarily have an /etc/magic database.

	Chris Flatters
	cflatter at nrao.edu

Article: 220 of sci.data.formats
Path: sparc10!ghost.dsi.unimi.it!rpi!uwm.edu!math.ohio-state.edu!howland.reston.ans.net!agate!ames!skates.gsfc.nasa.gov!Hypatia!leb
From: leb at Hypatia.gsfc.nasa.gov (Lee E. Brotzman)
Newsgroups: sci.data.formats
Subject: Re: Data format 'signatures' for the FAQ
Date: 26 Aug 93 19:45:51 GMT
Organization: NASA Goddard Space Flight Center -- InterNetNews site
Lines: 21
Message-ID: <leb.746394351 at Hypatia>
References: <DWELLS.93Aug26120427 at fits.cv.nrao.edu>
NNTP-Posting-Host: hypatia.gsfc.nasa.gov
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

dwells at fits.cv.nrao.edu (Don Wells) writes:
>The proposed FAQ for sci.data.formats should address the problem of
>how to recognize the format of an arbitrary data file. 
...
>As a start in this direction I have produced a prototype. 

Couldn't this be done just as easily by defining the signatures in /etc/magic
and letting the 'file' command do the work?  I have 'file' for my MS-DOS
machine as well on all the Unix boxes, and I think there may be some VMS
variant out there.  

For instance, I think a suitable entry for FITS would be:

0	string		SIMPLE		FITS
>30	string		T		standard file
>30	string		F		w/ non-standard extensions

--
-- Lee E. Brotzman                    Internet:  leb at hypatia.gsfc.nasa.gov
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB at GIBBS

Article: 222 of sci.data.formats
Newsgroups: sci.data.formats
Path: sparc10!ghost.dsi.unimi.it!batcomputer!caen!destroyer!gatech!howland.reston.ans.net!vixen.cso.uiuc.edu!news.cso.uiuc.edu!xongmao!chouck
From: chouck at xongmao.ncsa.uiuc.edu (Chris Houck)
Subject: Re: Data format 'signatures' for the FAQ
Date: Thu, 26 Aug 1993 22:36:03 GMT
Message-ID: <chouck.746404563 at xongmao>
References: <DWELLS.93Aug26120427 at fits.cv.nrao.edu> <1993Aug26.192719.13210 at Mr-Hyde.aoc.nrao.edu>
Sender: usenet at news.cso.uiuc.edu (Net Noise owner)
Organization: University of Illinois at Urbana
Lines: 27

cflatter at nrao.edu (Chris Flatters) writes:

>Of course system administrators of UNIX systems could be be persuaded to
>add the signatures to the /etc/magic file used by the find(1) command.
>The following patterns will recognize a FITS data file and distinguish
>between the 5 FITS data types.  If you want to use these you may want to
>cut and paste them from this message since blanks and tabs are significant.
>
># Magic number patterns for FITS data files
>0       string          SIMPLE  =                    T  FITS
>>109    string          8                8-bit integer data
>>108    string          16               16-bit integer data
>>107    string           32              32-bit integer data 
>>107    string          -32              32-bit IEEE floating-point data 
>>107    string          -64              64-bit IEEE floating-point data 
>
>I assume that HDF, netCDF and others also have distinguishing patterns.

# Data file checking codes
0       long            0x0e031301      HDF file
0       long            0x43444601      XDR-based netCDF file

Are valid tests.  Dunno the magic number for CDF.

-Chris Houck
HDF Group
NCSA Software Development Group