Subject: Re: more FITS fits

From: [email-masked] (Ata Etemadi)
Date: Fri, 20 Aug 93 19:25:27 BST

G'Day Pat

In article <[email-masked]>, [email-masked] (Pat Murphy) writes:
you made the following points which I have summarised (I hope to your liking):

|> I've noticed you emphasizing MPEG here before. I cannot see MPEG as being 
|> suitable for astronomical data.  

OK. So I am REALLY interested in MPEG which does use lossy compression. But
there is also MHEG which is data preserving and uses both compression and 
difference-between-frames techniques. In space physics which is quite a large 
field and is roughly speaking planetary astronomy I am making plans to use 
MPEG/MHEG. I can't say much more for reasons of copyright, but watch this space 
and if you are going to WorldCon 95 in Glasgow look out for me there. Therefore
it really depends what sort of science you are doing as to whether you would
consider it useful. Clearly using MPEG for 1 or 2 image sequences is silly.

|> paraphrasing: lone scientists archiving data are archivists and the data
|> represents an archive.

What _I_ mean by an archive is an orderly set of data which is gathered for
the express purpose of allowing people to use it, not just some person backing 
up data on a Vax or something in some silly format no matter how good their 
intentions.

|> I challenge you to respond constructively and
|> rationally to this post, e.g. by summarizing for us who don't know about
|> HDF _how_ it might be suited to astronomical or other data.

Now we are getting somewhere. I put down a similar challange for others 
regarding other formats. Now first off lets not just concentrate on HDF since 
netCDF, and CDF are also in the same ball park and equally good. There
is an HDF-netCDF interface, and I am writing HDF<-->CDF convertors but I
sure wish there was an HDF-CDF interface.

In so far as suitability below is a list SPECIALLY if you have tabular data 
or time series etc.. or images in GIF, JPEG etc.. Vast majority of my data 
is in the form:

year, day, hour, min, sec, density, velocity .....

NOTE: what I say about HDF is in general also true of netCDF and CDF and:::::
IS NOT AN ATTACK ON FITS. MAYBE THE FITS FOLKS CAN GIVE YOU A SIMILAR LIST.

1) As part-and-parcel of the format you get toolkits in C, F77, and C++ for 
   manipulating the data. This _really_ simplifies things during development.
2) Your code can be easily made portable to lots of platforms.
3) Direct support within PV-Wave, IDL, Matlab, AVS, SpyGlass etc... and lots 
   of PD packages ie you just click on the HDF/netCDF/CDF button and the data 
   is read in and ready to use.
4) Standard routines for creating/manipulating global/variable attributes 
   eg max, min, fillvalue, etc...
5) International Solar Terrestrial Physics (ISTP) committee standards for
   naming and defining attributes (get a copy of ISTP Brown Book) making it 
   easier to work with the metadata.
6) Hypertext interface to the data. Check out Xmosaic using URL:
   http://www.ncsa.uiuc.edu/SDG/Software/Mosaic/Docs  and look under
   HDF browsing.
7) Dozens (really) of very nice packages distributed by folks at NSSDC, NCSA 
   and NCAR for manipulating data in these formats. Some favourites: Data 
   Transfer Mechanism DTM from NCSA, IDL routines from NSSDC, NCAR graphics.
8) There are the SDSC image tools for converting to and from HDF image format
   and many many other image formats (I've appended a list to the end of this 
   post so you know what I mean). NCSA also distributes FITS2HDF (me thinks).
9) New features are added regularly (--HDF, netCDF and CDF--) and I have not 
   as yet had any problems or had to do any major rewrites or anything.
   Hopefully compression/encryption etc.. will be added.
10) EXCELLENT user support (--CDF, netCDF, and HDF--)
11) Image compression ie JPEG, and GIF are supported internally.

I am not sure if thats really what you wanted when you said "_how_ it might be 
suited to astronomical or other data". The point I am trying to make is there
are lots of tools for data analysis/visualisation using these formats and if
there is something not out there already you can build a tool very quickly and
efficiently which can handle generic data. My recommendation however is:

Use your own judgement to decide what best suits YOUR particular situation.

Bloody obvious I would have thought :-)

	adios
		Ata <(|)>.

Image formats supported by SDSC imconv tool:

    -alias                              Alias image file
    -avs                                AVS X image file
    -bm                                 X11 bitmap file
    -bmp                                Microsoft Windows bitmap image file
    -cur                                Microsoft Windows cursor image file
    -cursor                             Sun Icon and Cursor file
    -df                                 Hierarchical Data File
    -epi                                Adobe Encapsulated PostScript file
    -eps                                Adobe Encapsulated PostScript file
    -epsf                               Adobe Encapsulated PostScript file
    -epsi                               Adobe Encapsulated PostScript file
    -gif                                Compuserve Graphics image file
    -giff                               Compuserve Graphics image file
    -hdf                                Hierarchical Data File
    -ico                                Microsoft Windows icon image file
    -icon                               Sun Icon and Cursor file
    -iff                                Sun TAAC Image File Format
    -infile image_filename              Specify an input image file name
    -iris                               SGI RGB image file
    -ivb                                Truevision Targa image file
    -macp                               Apple Macintosh MacPaint file
    -mbfavs                             AVS X image file
    -mbfx                               AVS X image file
    -mpnt                               Apple Macintosh MacPaint file
    -ncsa                               Hierarchical Data File
    -outalpha                           Force output of alpha plane with image
    -outchandepth no_bits               Force output of # of bits/channel/pixel
    -outclt                             Force output of a CLT with the image
    -outcompress comp_type              Force output of compress image
    -outindex                           Force output as a color indexed image
    -outinterleave inter_type           Force output of interleaved RGB image
    -outnchan no_channels               Force output of # of channels/pixel
    -outnoalpha                         Block output of alpha plane with image
    -outnoclt                           Block output of a CLT with the image
    -outrgb                             Force output as an RGB image
    -pbm                                PBM Portable Bit Map file
    -pcc                                ZSoft IBM PC Paintbrush file
    -pcx                                ZSoft IBM PC Paintbrush file
    -pgm                                PBM Portable Gray Map file
    -pic                                PIXAR picture file
    -picio                              PIXAR picture file
    -pict                               Apple Macintosh QuickDraw/PICT file
    -pict2                              Apple Macintosh QuickDraw/PICT file
    -pix                                Alias image file
    -pixar                              PIXAR picture file
    -pnm                                PBM Portable aNy Map file
    -pntg                               Apple Macintosh MacPaint file
    -postscript                         Adobe PostScript file
    -ppm                                PBM Portable Pixel Map file
    -pr                                 Sun Icon and Cursor file
    -ps                                 Adobe PostScript file
    -ras                                Sun Rasterfile
    -register                           Generate a user registration form
    -rgb                                SGI RGB image file
    -rla                                Wavefront raster image file
    -rlb                                Wavefront raster image file
    -rle                                Utah Run length encoded image file
    -scr                                Sun Rasterfile
    -sgi                                SGI RGB image file
    -sr                                 Sun Rasterfile
    -sun                                Sun Rasterfile
    -suniff                             Sun TAAC Image File Format
    -synu                               SDSC Synu image file
    -taac                               Sun TAAC Image File Format
    -tga                                Truevision Targa image file
    -tif                                Tagged image file
    -tiff                               Tagged image file
    -vda                                Truevision Targa image file
    -vff                                Sun TAAC Image File Format
    -viff                               Khoros Visualization image file
    -x                                  AVS X image file
    -x11                                X Window System window dump image file
    -xbm                                X11 bitmap file
    -xv                                 Khoros Visualization image file
    -xwd                                X Window System window dump image file

---

From: [email-masked] (Chris Houck)
Date: Fri, 20 Aug 1993 22:12:47 GMT

>   http://www.ncsa.uiuc.edu/SDG/Software/Mosaic/Docs  and look under

If I can jump in here (and quickly jump out before becomming enmeshed
in the festivities :-)  I'd like to let people know that prerelease
copies of NCSA X Mosaic 2.0 are available on ftp.ncsa.uiuc.edu.  These
versions of Mosaic support the HDF 3.3 library and can thus display
hypertext summaries of HDF and netCDF files.  Check out the 2nd to
last paragraph of the "Demo Document"

Local files can also be browsed via the "Open Local" function of the
"File" menu.  However, the files need to end with the extentions .hdf
.nc or .cdf for Mosaic to handle them correctly.

-Chris Houck
(I think its a bit too dangerous to give my title here :-)