Subject: Data format 'signatures' for the FAQ

From: [email-masked] (Don Wells)
Date: Thu, 26 Aug 1993 17:04:27 GMT

The proposed FAQ for sci.data.formats should address the problem of
how to recognize the format of an arbitrary data file. It will never
be possible to provide a definitive answer to this question, but it is
practical to provide a set of heuristic classification rules, because
every data format has a characteristic "signature" which can be
recognized by a simple test applied to the first part of a file.

The data format classification rules should be expressed in the form
of a portable program. As a start in this direction I have produced a
prototype.  File fits.cv.nrao.edu:/pub/signatur.tar.Z includes the
source for signatur.c (which I append), a Makefile, one sample binary
data file and the output from a test run.  Fetch the file with
anonymous-FTP in binary mode, then do commands 'uncompress
signatur.tar.Z', 'tar xvf signatur.tar' and 'make'. The make will
compile signatur.c and will execute a series of test cases which
demonstrate the four possible error exits plus the two levels of
output for data file tests.

I have tested signatur.tar.Z, with Makefile and signatur.c unchanged,
on four rather different Unix environments: SPARC+SunOS4.1.2+cc (also
gcc & acc), SPARC+SunOS5.1(Solaris)+gmake+cc, RS/6000+AIX+cc and
Alpha+OSF/1+cc. I have also tested signatur.c on a PC/XT+DOS with
TurboC.

Programmers can select suitable sample files for their own formats
(hopefully of modest size) and hack signatur.c and the Makefile in the
obvious ways so that the package will test their formats as well as
the one I have coded.  They can augment variables TARSRC and ANONFTP
in the Makefile and deliver their new versions of signatur.tar.Z into
their local anonFTP servers, where others can pick them up, and add
code and sample files for their own formats in turn.

/* signatur.c --- A program to classify data files */
#include "stdio.h"
#define TESTSIZE 1024

main(argc, argv)
     int argc;
     char *argv[];
{
  FILE *stream;
  int i, nextbyte, verbose;
  char *filename, array[TESTSIZE];
  
  verbose = 0;
  if (argc == 2) 
    filename = argv[1];
  else if ((argc == 3) && (strcmp (argv[1], "-v") == 0)) {
    verbose = 1;
    filename = argv[2];
  } else {
    fprintf (stderr, "Usage:     signature [-v] filename\n");
    exit (1); 
  }
  if ((stream = fopen (filename, "r")) == NULL) {
    fprintf (stderr, "Cannot open <%s> for reading.\n", argv[1]);
    exit (2);
  }
  for (i = 0; i < TESTSIZE; i++) {
    if ((nextbyte = fgetc (stream)) == EOF) {
      fprintf (stderr, "Encountered EOF at byte %d of <%s>.\n", i+1, argv[1]);
      exit (3);
    }
    array[i] = nextbyte;
  }
  /* -=-=- Test for FITS: -=-=- */
  if (strncmp (&array[0], "SIMPLE  =                    T ", 31) == 0) {
    if (!verbose)
      printf("FITS\n");
    else {
      printf("FITS [Flexible Image Transport System] is the standard\n");
      printf("data interchange and archive format of astronomy.\n");
      printf("For information about FITS see document\n");
      printf("    nssdca.gsfc.nasa.gov:/fits/fits_standard.ps\n");
      printf("and directory tree\n");
      printf("    fits.cv.nrao.edu:/fits/*,\n");
      printf("both available via anonymous-FTP.\n");
    }
    exit(0);
  }
  /* -=-=-=-=-=-=-=-=-=-=-=-=- */
  fprintf (stderr, "Format of file <%s> is not recognized.\n", filename);
  return(4);
}

--

  Donald C. Wells           Associate Scientist       [email-masked]
  National Radio Astronomy Observatory                +1-804-296-0277
  520 Edgemont Road                              Fax= +1-804-296-0278
  Charlottesville, Virginia 22903-2475 USA         78:31.1W, 38:02.2N

---

From: [email-masked] (Chris Flatters)
Date: Thu, 26 Aug 93 19:27:19 GMT

>output for data file tests.

Of course system administrators of UNIX systems could be be persuaded to
add the signatures to the /etc/magic file used by the find(1) command.
The following patterns will recognize a FITS data file and distinguish
between the 5 FITS data types.  If you want to use these you may want to
cut and paste them from this message since blanks and tabs are significant.

# Magic number patterns for FITS data files
0       string          SIMPLE  =                    T  FITS
>109    string          8                8-bit integer data
>108    string          16               16-bit integer data
>107    string           32              32-bit integer data 
>107    string          -32              32-bit IEEE floating-point data 
>107    string          -64              64-bit IEEE floating-point data 

I assume that HDF, netCDF and others also have distinguishing patterns.

OpenWindows 3.x users can also make the filemanager recognise FITS files
by registering the "SIMPLE  =                  T" string with the binder.
This would allow a distinctive icon to be associated with FITS files and/or
a program (eg SAOimage) to be run if the icon is double clicked.

NB: file(1) is a POSIX.2a command but POSIX.2a does not specify how file(1)
should recognise file types so although a POSIX.2a system must have file(1)
it need not necessarily have an /etc/magic database.

	Chris Flatters
	[email-masked]

---

From: [email-masked] (Chris Houck)
Date: Thu, 26 Aug 1993 22:36:03 GMT

>I assume that HDF, netCDF and others also have distinguishing patterns.

# Data file checking codes
0       long            0x0e031301      HDF file
0       long            0x43444601      XDR-based netCDF file

Are valid tests.  Dunno the magic number for CDF.

-Chris Houck
HDF Group
NCSA Software Development Group

---

From: [email-masked] (Lee E. Brotzman)
Date: 26 Aug 93 19:45:51 GMT

[email-masked] (Don Wells) writes:
>how to recognize the format of an arbitrary data file. 
...
>As a start in this direction I have produced a prototype. 

Couldn't this be done just as easily by defining the signatures in /etc/magic
and letting the 'file' command do the work?  I have 'file' for my MS-DOS
machine as well on all the Unix boxes, and I think there may be some VMS
variant out there.  

For instance, I think a suitable entry for FITS would be:

0	string		SIMPLE		FITS
>30	string		T		standard file
>30	string		F		w/ non-standard extensions

--
-- Lee E. Brotzman                    Internet:  [email-masked]
-- Hughes STX                         DECNET:    NDADSA::BROTZMAN
-- NASA Goddard Space Flight Center   BITNET:    ZMLEB@GIBBS

---

From: [email-masked] (Ata Etemadi)
Date: Fri, 27 Aug 93 19:58:50 BST

In article <[email-masked]>, [email-masked] (Chris Flatters) writes:
|> In article [email-masked], [email-masked] (Don Wells) writes:
|> >The proposed FAQ for sci.data.formats should address the problem of
|> >how to recognize the format of an arbitrary data file. It will never
|> >be possible to provide a definitive answer to this question, but it is
|> >practical to provide a set of heuristic classification rules, because
|> >every data format has a characteristic "signature" which can be
|> >recognized by a simple test applied to the first part of a file.

Xloadimage automatically recognises the types below. Maybe someone is
willing to hack the code and extract the recognition part. xv does a 
similar job but its not freeware anymore. Also you can get a patch for
xloadimage to make it recognize Vicar format to.

  FBM Image                              X Window Dump
  Sun Rasterfile                         McIDAS areafile
  CMU WM Raster                          G3 FAX Image
  Portable Bit Map (PBM, PGM, PPM)       PC Paintbrush Image
  Faces Project                          GEM Bit Image
  GIF Image                              MacPaint Image
  Utah RLE Image                         X Pixmap
  JFIF-style JPEG Image                  X Bitmap

	adios
		Ata <(|)>.

---

From: [email-masked] (Joel Plutchak)
Date: 27 Aug 1993 18:36:29 GMT

In article <leb.746394351@Hypatia> [email-masked] (Lee E. Brotzman) writes:
>and letting the 'file' command do the work?

   Will that work for file formats which have no "magic number"?  For
instance, one format used in the planetary arena doesn't seem to have
any obvious identifying characteristic, i.e. it has some binary numbers
giving header and image sizes, and that's it.  (I handle this in programs
by checking for all the "nice" formats first, then do a few sanity checks
to see if the file could by that image format.)
--
Joel Plutchak, Research Programmer/Analyst
"I'm mentally OVERDRAWN!"

---

From: [email-masked] (Chris Flatters)
Date: Fri, 27 Aug 93 23:43:11 GMT

In article [email-masked], [email-masked] (Joel Plutchak) writes:
>   Will that work for file formats which have no "magic number"?  

No.  Implementations of file(1) that use a magic number file rely on being
able to check data at some fixed location in the file.  The magic number
file also has some portability problems: magic number patterns are not
portable between machines with different short and long lengths and may
not be portable between little-endian and big-endian machines where the
lengths of short and long do match.  The /etc/magic file format is also
poorly defined.

		Chris Flatters
		[email-masked]