Subject: storage of drift scan images?

From: steve at cfht.hawaii.edu (Steven S Smith)
Date: 19 Sep 91 20:22:50 GMT

Has anyone used FITS to store drift scanned images?

In our case, the telescope will not be tracking an object,
but the detector will be read out, row by row, at a rate equal to the
normal tracking rate.  Since this gives a strip of the sky, the
resulting data can be very long.  While I am sure that the FITS
standard can easily do this, I doubt that very many FITS
implementations would be able to utilize the resulting data.

I guess just arbitrarily hacking the data into some nice length would
work, but it seems inelegant (it could leave an object partly on two
FITS files).

Any prior experience with this problem, or general comments are
welcome.

steve
--
Steven S. Smith 		Canada-France-Hawaii Telescope Corp. 
steve at cfht.hawaii.edu

---

From: pwb at newt.phys.unsw.oz.au (Paul W. Brooks)
Date: 22 Sep 91 14:49:19 GMT

> implementations would be able to utilize the resulting data.

My implementation could, and it runs on a lowly PC!
In general, I would expect most implementations which deal with FITS
images would not try to load the entire image into RAM, but would
deal with it on a line-by-line basis, as a means of ensuring that the 
software would not be limited as the images grow larger, which they are
steadily doing from year to year. As your drift-scanned images have 
a limited (normal) number of columns, I would hope that most software
would not be too fazed by having more ROWS than the programmer expected
-
A programmer can never outguess the users of the software, and set
limits that "will never be exceeded in practice". Rather s/he should
create software without limits!

> Any prior experience with this problem, or general comments are

Paul Brooks        |Internet: pwb at newt.phys.unsw.edu.au
Uni. of N.S.W.     |If you have trouble sleeping, try lying on the end of
Kensington NSW 2033|   your bed. With a little luck you'll drop off. 
AUSTRALIA          |                              - Mark Twain.

---

From: steve at cfht.hawaii.edu (Steven S Smith)
Date: 23 Sep 91 21:00:49 GMT

pwb at newt.phys.unsw.oz.au (Paul W. Brooks) writes:

...

>create software without limits!

While I agree that this is a good goal, in reality something has to
give.  Assuming a 2048x2048 ccd, with a shift rate of ~30 seconds for
a full frame equivalent, the data sizes we are looking at are on the
order of a gigabyte/hour.  An individual scan could last several
hours. Your program may handle this, but the operating and/or file
systems may fail.  And indeed, since the main benefit of FITS (for us,
anyhow) is that it allows people to easily drag their data back to
their home institutions, the data we produce needs to work with most
of the FITS software available.  Expecting Joe Astronomer's
FITS-to-whatever program to work on these monsters is not realistic,
since simply changing locations and (or names) of some header cards
causes complaints.

I believe that we will have to wait a few years before the computer
hardware/software infrastructure will have expanded to allow
multi-gigabyte files to work.  The FITS "standard" may not need any
improvement to allow this, but individual implementations will.

So until then, any ideas on how to chop them up?  Is it useful to add
some heuristics to the splitting routine, for example, only split on
rows that contain relatively low data values?  Or do we duplicate a few
rows from one FITS file to the next, so that an object of interest
will be contained entirely on one of the files?  And how many is a few?