Subject: Variable length character string arrays.

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Tue, 14 Jul 1992 17:57:00 GMT

A while back there was a bunch of discussion about devising a standard for
denoting the length of character strings in an array of character strings in a
binary table.  It started off when Bill Pence suggested using

				TFORMn = 'rAw'

where 'r' is the total number of characters in the array, and 'w' is the number
of characters in each character string element of the array.  However, it was
also pointed out (by Doug Tody?) that character strings could also be variable
length, delimited by a particular byte value.  I believe the suggestion for
that was

			       TFORMn = 'rA\ooo'

where 'ooo' was the octal representation of the delimiting byte.  I myself
suggested using separate keywords, TCLENn and TCDELn, to express either the
fixed length, or variable length delimiter.

I've been thinking some more about storing variable length strings in binary
tables.  I've convinced myself that two pieces of information have to be
supplied to allow all readers to be able to parse the information.  The first
piece, of course, is the value of the delimiter byte.  The second desirable
piece of information is the maximum length of the character string.  I have
worked with environments which could only handle string arrays where each
element of the array had the same string length as all the others (e.g. the
original VMS version of IDL).  While I don't currently work with anything that
has this limitation, I'm not sure we can guarantee that all environments won't
be so limited.  In such a situation, it would be an advantage to know a priori
the largest size the string element could obtain, so that the string array
could be properly initialized.  I think that this approach is similar to that
taken by the "Variable Length Array Facility" described in the binary tables
proposal, where the number of elements of the array is variable, but the
maximum possible number of elements is stored in the TFORM keyword.

Hence, the most desirable standard for delimiting the individual character
strings in a character string array would have the following characteristics:

	1.  It would work for both fixed-length and variable-length
	    (byte-delimited) character strings.

	2.  It would work for both direct (TFORM='rA') and pointer (TFORM='PA')
	    binary table columns.

	3.  For fixed-length strings, it would specify the character length.

	4.  For byte-delimited strings, it would specify both the byte code
	    used to delimit the strings, and a maximum length that the string
	    will not exceed.

A remaining question is whether this information is best appended to the TFORM
keyword, or stored in one or more separate keywords.

Bill Thompson

---

From: pence at heawk1.gsfc.nasa.gov ( William Pence)
Date: Wed, 15 Jul 1992 18:02:00 GMT

>	    (byte-delimited) character strings.

TFORM1  = 'rAw'          / fixed length strings; r = total no. of characters
                         /    and w = length of each fixed-length string
TFORM1  = 'rA\ooo'       / byte delimited strings where 'ooo' is the octal byte

>	2.  It would work for both direct (TFORM='rA') and pointer (TFORM='PA')

TFORM1  = 'rAw'           / fixed length strings, direct
TFORM1  = 'rA\ooo'        / byte delimited strings, direct
TFORM1  = 'PAw(r)'        / fixed length strings, pointer
TFORM1  = 'PA\ooo(r)'     / byte delimited strings, pointer

>	3.  For fixed-length strings, it would specify the character length.

The 'w' integer field in all the above examples represents the fixed-length

>	    used to delimit the strings, and a maximum length that the string

The integer 'r' field in all the above cases is the maximum length of a
delimited string.  In general, a single string could fill the entire field

>keyword, or stored in one or more separate keywords.

I strongly believe all this information should be encoded into the
single TFORM keyword.  Otherwise, FITS readers/writers have to be
made more complicatd to handle the extra keywords, which are not needed
for columns with any other datatype.

-Bill Pence

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: 15 Jul 92 21:24:00 GMT

In article <pence.711223320 at heawk1>, pence at heawk1.gsfc.nasa.gov ( William Pence) writes...
>Here is how I would implement Bill Thompson's requirements for string arrays:
		(stuff deleted)

>delimited string.  In general, a single string could fill the entire field

If one has

>	TFORM1  = 'PA\ooo(r)'     / byte delimited strings, pointer

and I want to read this into a string array, then I don't know how many string
elements to assign to the array, nor do I know many characters to allocate for
each string, if my environment required me to do it.  The extremes are that
there could be as many as r/2 strings of one character each (if r is even), or
one string with r characters.  The most conservative approach would be to
allocate an array of r/2 elements, each of which with r characters, leading
to a total of r^2/2 characters for the resulting array, which is ridiculous.

I guess that I don't just want to know how big the string can get; I also want
to know how many strings there are.  Note that this is not a problem with the
simpler approach

>	TFORM1  = 'PAw(r)'        / fixed length strings, pointer

For my own part, it's this kind of string I would most expect to store in FITS
files.  It's worth thinking about variable length strings, though.

Finally,

>for columns with any other datatype.

The philosophy behind the TFORM keyword seems to be to define things at an
atomic level.  The question then comes up: is one atom of character string data
a string or a character?  From a user's point of view, it should be the former,
but the answer may be software sensitive.

Bill Thompson