Subject: Re: ASCII table tricks

From: sla at umbra.ucolick.org (Steve Allen)
Date: 31 Aug 1996 00:03:42 GMT

In article <30AUG199616195125 at nssdca.gsfc.nasa.gov>,
Barry M. Schlesinger <bschlesinger at nssdca.gsfc.nasa.gov> wrote:

>The standard says that all data must be ASCII text, not only data in
>fields.

Given the wording in 8.1.3 and the section title of 8.1.5 this is
not clear.

>The original Harten et al. tables paper states, "The data records are
>stored as a large character array....All information is stored as
>8-bit printable ASCII characters..."  This language excludes control
>characters.  There is no entry in appendix C to indicate that the
>standard differs from the FITS papers in this regard.

I believe that it is arguable that in the context of this paper the
word "information" only applies to the content within the fields of
the rows.

This interpretation appears defensible, and it has demonstrable
benefits.  Are there any technical arguments against it?

>This question will probably be discussed at the FITS Technical Panel
>meeting next month.

See y'all at ADASS.
--
Steve Allen          UCO/Lick Observatory       Santa Cruz, CA 95064
sla at ucolick.org      Voice: +1 408 459 3046     FAX: +1 408 454 9863
WWW: http://www.ucolick.org/~sla                PGP public keys:  see WWW

---

From: groth at pupgg.princeton.edu (Edward J. Groth 609-258-4361)
Date: 31 Aug 1996 14:10:51 GMT

In article <5079np$ae7 at darkstar.ucsc.edu>, sla at umbra.ucolick.org (Steve Allen) writes:
>In article <199608301312.JAA06260 at tetra.gsfc.nasa.gov>,
>William Pence  <pence at tetra.gsfc.nasa.gov> wrote:
>>Steve Allen wrote:
>>> Does the standard permit this, or did I miss some constraint?
>>
>>The FITS Standard does not allow the ASCII newline character or any
>>other ASCII control characters in an ASCII table.  Section 8.1.5 states
>>that "All data in an ASCII tables extension record shall be ASCII text ..."
>>and "ASCII text" is defined to be the ASCII characters hexidecimal 20-7E.
>
>But wait, playing standards lawyer for a moment I point out that
>section 8.1.5 is about "entries" in the table.
>
>In section 8.1.3 it states
>  "The table is constructed from a two-dimensional array of ASCII characters."
>which means anything in 7-bit ASCII.
>
>I believe the standard is saying that the data fields themselves may
>only contain printable text, but that portions of the table outside
>the fields may contain other ASCII charcters.
>
>In particular, I believe this permits the tricks I suggested.
>
>If this is not what the standards writers intended then the
>language of section 8.1.3 needs clarification.
>
>--

Well, the standards writers certainly never intended that one
play tricks and we certainly don't need people to look for
loopholes and take advantage of them.

I point out that the newline character is a line feed only on
Unix.

On Macs it's a carriage return.

On PCs it's a carriage return and a line feed.

On VMS it can be any of the above as well as other formats.

Are you prepared to deal with the following?

PC user ftp's table, messes around with it, uploads a new version
and all of a sudden your software croaks because there are now
carriage returns as well as line feeds where they used to be only
line feeds and all the positions of the fields are off????

					- Ed

/----------------------------------------------------------------------\
| Edward J. Groth            | Phone: 609-258-4361   Fax: 609-258-6853 |
| Physics Dept., Jadwin Hall | URL: http://pupgg.princeton.edu/~groth/ |
| Princeton University       | SPAN/HEPNET:  PUPGG::GROTH=44117::GROTH |
| Princeton, NJ 08544        | Internet:     groth at pupgg.princeton.edu |
\----------------------------------------------------------------------/

---

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry M. Schlesinger)
Date: 30 Aug 1996 16:19 EDT

In article <5079np$ae7 at darkstar.ucsc.edu>, sla at umbra.ucolick.org (Steve Allen) writes...
>which means anything in 7-bit ASCII.

It says that only ASCII characters are allowed, not that all ASCII 
characters are allowed.

				Barry Schlesinger
				FITS Support Office
				GSFC/ADF

---

From: sla at umbra.ucolick.org (Steve Allen)
Date: 2 Sep 1996 06:55:33 GMT

In article <509h5b$4np at cnn.princeton.edu>,

>line feeds and all the positions of the fields are off????

Using a good editor the file is just as trivial to return to FITS
conformance as it was for a bad editor to break it.  There are far
worse things that can happen when accepting files from a careless or
disreputable PC user.

The casual inspection and modification of FITS tables using generic
text editors rather than FITS-specific ones opens many possibilities.
ASCII tables could be subjected to powerful regular expression
matching, substituting, and sorting tools which do not otherwise exist
for FITS.  The similarity with database inload/outload functions can
permit FITS tables to be used as a state-saving and transfer device
for the contents of relational databases.  FITS ASCII tables are
much more robust and self-documenting than most such RDB tools.

I see this as a way of putting FITS capabilities into the hands of
people who would otherwise not have it because of the bother of
constructing programs that do FITS I/O.  I doubt that there can be any
FITS ASCII table reader which would be unable to read such files.

Whether intentional or not the standards documents seem to have
permitted a format which could make the data in ASCII tables much more
accessible to everyone.  Is this a bad thing that must be squelched
or a good thing that should be pursued?
--
Steve Allen          UCO/Lick Observatory       Santa Cruz, CA 95064
sla at ucolick.org      Voice: +1 408 459 3046     FAX: +1 408 454 9863
WWW: http://www.ucolick.org/~sla                PGP public keys:  see WWW

---

From: seaman at noao.edu (Rob Seaman)
Date: 5 Sep 1996 08:17:38 GMT

Ed Groth says:

> Well, the standards writers certainly never intended that one play
> tricks and we certainly don't need people to look for loopholes and
> take advantage of them.

Steve counters with:

> permitted a format which could make the data in ASCII tables much
> more accessible to everyone.  Is this a bad thing that must be
> squelched or a good thing that should be pursued?

I'll register my vote against squelching...

Another thread in this newsgroup also just touched (tangentially) on
the question of ambiguous FITS usage.  It seems to me that FITS can
be viewed as ambiguous in a couple of different ways.

The first is by intent.  As with other standards such as X11 and
PostScript, FITS mandates no strong usage policy of its own, but relies
on contingent standards (or software) to define the policies.  Examples
are Motif for X11 and EPSF for PostScript - while FITS relies on header
keyword conventions (for instance) that are put forth by the different
segments of the astronomical community, or de facto usage as defined
by AIPS, IRAF or whatever.

Like other standards, FITS can also be unintentionally ambiguous (and
hopefully only ambiguous and not internally inconsistent).  Of course,
there are also always examples of data that are simply non-conforming.

The usual way to allow for ambiguity is for software that writes data
structures to strictly adhere to the standard, but for software that
interprets the data structures to only loosely require adherence to
the standard.  Ideally a FITS reader would require only the loosest
possible self-consistent interpretation of the standard.

Question:  Does the robust functioning of a FITS table reader really
require that only printable ASCII characters appear in the data array?

The NOST standard is more ambiguous than the original paper (A&A Suppl
73, p365) about the question of non-printing characters.  While a little
maneuvering room is still available in Harten, et.al., it's hard to argue
that the authors' original intent was to allow non-printing ASCII anywhere
in the data array.  This surely would have been discussed in the paper -
precisely because it's a useful feature as Steve has pointed out.

However, I don't think we need be inordinately solicitous of the intent
of the standards writers (or rather, of the apparent lack of intent in
this case).  The astronomical community governs FITS, not the other way
around.  If the wording of the standard allows a particular usage, then
either the wording should be changed - or the usage should be accepted.

One could even use the golden rule of "once FITS, always FITS" to argue
that any ambiguity that creeps through is instantly incorporated into
the standard for all time.  If nothing else, this certainly argues for
wording every proposed change to the standard with extreme care.

But I think we should think twice about rejecting all "loopholes" out
of hand...  A lot of the FITS deliberations lately have risen out of
these same loopholes - or out of the seams between different parts of
the standard.  I'd argue that this is a sign of the maturity of FITS.
Issues that were left unaddressed by the early history of FITS are
now of increasing importance.  Without the elbow room provided by the
(intentional or unintentional) ambiguity of the standard, FITS would
be handcuffed when dealing with the current issues.

An alternate interpretation is that each change to the standard
should be as tightly constraining as possible, since it will always
be possible to relax the constraints later should that prove to be
desirable.  Perhaps including non-printing characters in FITS ASCII
tables is an example of this.

Note that even if the original tables specification is reaffirmed as
not allowing non-printing characters, adding this as a new feature is
entirely consistent with the rules of FITS.  No preexisting data would
be harmed.  Software might break, but it would likely break in the
politically accepted manner - by totally bombing, that is.  On the
other hand, it would also be unsurprising if current tables software
passed over non-printing characters with nary a burp.

Steve's suggestion appears to make ASCII tables much easier to use.
Non-printing characters, especially between fields if not in the fields
themselves, do not complicate the standard to any great extent.

Meanwhile, Lucio Chiappetti wonders:

> Is anyone using FITS ASCII tables at all nowadays, or not only BINTABLEs ?

Folks likely will continue to use ASCII tables for the same purposes
they originally did - for large catalogs and as an easily accessible
human readable format.  Note that ease of use was integral to the
original design and may certainly continue to distinguish ASCII from
binary FITS tables.  Also, not withstanding that Harten, et.al., state
that "...  the format is primarily intended to be used for the transfer
of information rather than the storage of information", ASCII tables
are likely to be preferred for archival applications.

Why am I showing any interest in this issue?  Well, it occurs to me that
by permitting non-printing ASCII field and record separators, it would be
immediately possible to store a /rdb database in a FITS table extension -
or rather, a legal FITS table could be operated on and maintained by this
relational database.

/rdb is a Unix tools style RDB that uses ASCII files with tab delimited
fields.  It can support surprisingly large databases - 800,000 image
headers from the NOAO archive occupy about 145Mb in one R&D database
we've constructed.  Separate binary index files can be generated as
needed.  By design, /rdb encourages access to the DB using tools such
as awk, sed, perl and the Unix shells.  This is precisely the same
functionality that Steve has suggested for FITS tables.

Note that while some scripting languages nominally support remappable
field and record delimiting characters (perhaps providing a way of
working with FITS tables using generic tools from the other end) -
these are only marginally useful in practice.  Embarrassingly, probably
the best known example of such usage for shell scripts is as the
prototypical setuid security hole.

Allowing easy user level FITS support for such text based tools would
indeed require permitting tabs, newlines and perhaps other ASCII
control characters into the data.  Is this feature worth the trouble?
I think so.  Is this extremely dangerous?  No.  Is this something
to worry about right now?  Opinions will differ.

On the other hand, constraining tabular data arrays (not only FITS)
to contain only printable ASCII makes many classes of applications
that rely on "out-of-band" data (non-printing control characters,
in the case of ASCII) impossible.

Rob
-- 
seaman at noao.edu, http://iraf.noao.edu/~seaman
NOAO, 950 N Cherry Ave, Tucson AZ 85719, 520-318-8248
PGP: 98 8D 8B 49 74 9A 41 88  3A 43 87 54 51 BF 30 4B

---

From: William Pence <pence at tetra.gsfc.nasa.gov>
Date: Fri, 6 Sep 1996 11:14:01 -0400

Would the proponents for allowing ASCII control characters in 
ASCII tables also want to allow CR and LF characters at the
end of each header keyword?  It seems like this would
also be required if one wants to be able to use standard
text editors on FITS files.  

-Bill Pence

---

From: RANDY THOMPSON (301) 286-8800 <RTHOMPSON at iuedac.gsfc.nasa.gov>
Date: Tue, 10 Sep 1996 14:33:10 -0400 (EDT)

I agree with Bill Pence and Barry Schlesinger in interpreting the 
current NOST standard as saying only ASCII text characters (i.e hexidecimal
20-7E) should appear in the records of an ASCII Tables extension. However,
it seems to me that a Binary Tables extension could be used to store the 
data described earlier by Steve Allen? Maybe I'm missing something (since I 
haven't seen this suggested already), but why not store the ASCII text as 
character strings and define fields of byte data type to contain the byte 
values of non-printable ASCII characters? If you strip off the headers, 
the remaining records should be identical to those that could be produced 
with the ASCII Tables extension.
   If this isn't suitable, I would think one could still propose a change to 
the current FITS standard that would (unambiguously) allow non-printable ASCII 
characters as non-field values. If existing FITS readers are really not 
impacted, why should such a proposed change not be approved? Of course, the
change would still need to go through the established approval process.
   Alternatively, one could define a new table extension following the 
requirements for conforming extensions. The new extension could simply follow 
the existing rules for ASCII Tables but without the ASCII text restrictions.  

Randy Thompson

---

From: "Thomas A. McGlynn" <tam at silk.gsfc.nasa.gov>
Date: Tue, 10 Sep 1996 16:38:26 -0400

RANDY THOMPSON (301) 286-8800 wrote:
>... However,
...
> Randy Thompson

I think that adding additional meaningless columns to a binary
table does substantial violence to the file since, e.g., it changes
the column numbers of the real data.  But I really like the idea that
ASCII tables be readable as readable as possible.  When ASCII tables
were introduced they were the only standard way for creating tabular
data.  But with the
advent of binary tables the only rationale I can see for using them
is that they are directly readable by humans.  I can attest from
personal
experience that it is very helpful to be able to read the tables -- one
trick I use is to adjust the width of my screen to the length of the
table.
Even if the rows don't start in the beginning it still helps a lot.

In writing a STBRYAFWFIDL (Someday-to-be-released,
Yet-another-FITS-writer-
for-IDL), I had anticipated users wanting to separate and delimit rows
and I'm quite disappointed that it is at best questionable FITS to
delimit the columns in the most natural way.

I think we are choosing between making ASCII tables completely obsolete
or leaving them a niche market in the FITS file economy.  If it truly
is 'illegal' to use \n in a FITS table, I would second Randy's call
for an amendment to their definition.

		Tom McGlynn
		GSFC.

		Tom McGlynn

---

From: seaman at noao.edu (Rob Seaman)
Date: 9 Sep 1996 16:04:47 GMT

Bill Pence writes:

> end of each header keyword?

I was wondering when someone would mention that.  It would be
fairly straightforward to arrange to skip over a header and apply
some general operation beginning at a byte offset into a file.  I'd
be interested in hearing specific plans for this, though.  Note also
that not all editors will choke on a line 2880+ characters long.

Of course, reformatting the table data on the fly to include the
necessary control characters (and remove them again afterwards) is
also possible, but this is a much more expensive and obscure operation.

Has anyone simply tried embedding newlines in a table to see what
happens with current software (and what the handling advantages are)?

Rob

---

From: sla at umbra.ucolick.org (Steve Allen)
Date: 9 Sep 1996 17:58:08 GMT

In article <199609061514.LAA01796 at tetra.gsfc.nasa.gov>,
>text editors on FITS files.

Certainly not.  I do not pretend that it is possible to make a FITS
file comprehensible to an arbitrary text editing application.  I do
find that the addition of carriage control information between the
fields makes the table trivial to view and process by many more
applications.

My editor has no problem with a FITS header as is.  With the addition
of separators between the fields the ASCII table records become
relatively easy to edit, too.  This is obviously not something that
should be tried by a naive user with a poor editor, but that is hardly
a reason to prohibit the possibility.
--
Steve Allen          UCO/Lick Observatory       Santa Cruz, CA 95064
sla at ucolick.org      Voice: +1 408 459 3046     FAX: +1 408 454 9863
WWW: http://www.ucolick.org/~sla                PGP public keys:  see WWW

---

From: Archie Warnock <warnock at clark.net>
Date: Mon, 09 Sep 1996 23:49:05 -0400

> text editors on FITS files.

Not necessarily.  It's pretty easy to split off the header from the data
and just access the data directly.  Now, if someone wanted to propose
allowing CR/LF characters in the header, I'd sure think that was a fine
idea, too, but it's not related to the idea of placing extra (and
undefined) characters in the tables.

I always thought it was pretty obvious that one could put those extra
characters into the tables.  There's no requirement that the table
header define every column of the table.

-- 

Archie

-- Archie Warnock                           Internet:  warnock at clark.net
-- A/WWW Enterprises                        Phone/FAX: 301-854-2987
--	     http://www.clark.net/pub/warnock/warnock.html
--	   As a matter of fact, I _do_ speak for my employer.