Subject: Re: triple precision keywords

From: landsman at stars.gsfc.nasa.gov (Wayne Landsman (301)-286-3625)
Date: Thu, 1 Oct 1992 16:49:00 GMT

In article <pence.717873310 at heawk1>, pence at heawk1.gsfc.nasa.gov (William Pence) writes...

>In principle there is no reason that this information could not all
>be expressed in a single keyword value, like:
> 
>MJD     =      48043.8797453703700742 /  MJD

    My current IDL FITS reader takes the NOST manual literally.   It
only extracts the 20 characters in columns 11-30 to determine the FITS
keyword value.    Any characters in columns 31-50 are interpreted as being
the imaginary part of a complex number.

        My suggestion would be to express this value as a string.

MJD     =      '48043.8797453703700742' /  MJD

       This would guarentee that all FITS readers preserve the full precision.
The calling program could then parse the string into its integer plus 
fractional component.

--Wayne Landsman                        landsman at stars.gsfc.nasa.gov

---

From: bschlesinger at nssdca.gsfc.nasa.gov (Barry Schlesinger)
Date: Fri, 2 Oct 1992 15:38:00 GMT

In article <1OCT199212490103 at stars.gsfc.nasa.gov>, landsman at stars.gsfc.nasa.gov (Wayne Landsman (301)-286-3625) writes...

>the imaginary part of a complex number.

This requirement applies only to fixed format.  The fixed format is
required only for values of mandatory keywords.  (At present, there
are no mandatory keywords with floating point values.)  For other
keyword values, it is recommended but not required.  See FITS Paper I,
which says "recommended, and partly required."  This language is
interpreted in 5.3.1 of the NOST standard by saying, "The fixed format
is required for values of mandatory keywords and recommended for all
others."  A well-recognized problem is that columns 11-30 do not
contain enough space to express the full precision of a double
precision floating point number.  The most common, but not standard or
required solution, is to extend the value beyond column 30.  A number
of proposed designs reviewed by the FITS office use this solution.  A
FITS reader that assumes that values of standard optional keywords or
user-defined keywords end in column 30 will not extract correct values
for some user-defined keywords in these data sets. 

Incidentally, EVERYBODY should be certain that their copy of the NOST
Standard is current.  The most recent version is 0.3b.  Some
misunderstandings have arisen because users are referring to 
superseded versions.

				Barry Schlesinger
				NOST FITS Support Office

---

From: Lucio Chiappetti <lucio at ifctr.mi.cnr.it>
Date: Mon, 05 Oct 92 14:12:12 ITA

On Wed, 30 Sep 1992 17:15:10 GMT  <pence at heawk1.gsfc.nasa.gov> said:
>
>Here at the HEASARC we often need to express keyword values which have
>more significant figures than can be stored in a double precision

  I actually wonder how this is SPECIFICALLY related to the example of
  MJD discussed below, or is really general.

>A common example of this problem is the Modified Julian Date
>which has a value something like 48043.8797453703700742

  I am inclined to think this is somehow related to the need of expressing
  time quantities like this.
  What I believe is needed is a way of defining some "compound" or "non-
  primitive" data types. By non-primitive I mean they should go beyond the
  usual CHARACTER, INTEGER, REAL and DOUBLE PRECISION supported by FITS.
  I raised this point already in a previous posting.
  In my view this concerns BOTH header keywords AS WELL AS data quantities
  (e.g. columns in binary tables).
  The sort of data types I am thinking of are quantities like :

  ANGLE   :  an angle in degrees, expressed e.g. as a DOUBLE PRECISION
             floating point number (but liable to be displayed as
             ddd:mm:ss.ff... and/or read as an array of ddd mm ss etc.)
  HANGLE  :  an angle in hours, possibly stored as the previous one,
             but liable to be displayed as hh:mm:ss.ff (with an implicit
             multiplication/division by 15 whenever needed)
  DATE    :  a quantity liable to be displayed as YYYY-Mon-DD (or YYYY-MM-DD
             or DD-MM-YYYY or MM-DD-YYYY according to whatever European or
             US or other form ... I personally prefer YYYY-MM-DD because it
             is simpler to sort). This quantity is likely to be more used
             in headers (e.g. to time stamp a file) than in data areas.
             However it might be worthwhile to represent it natively as
             a numeric quantity on which one can perform arithmetic operations.
             Any choice of a running count of days since an arbitrary zero
             is good, so IAU MJDs are a reasonable choice.
             Since one is interested only at a precision at the day level
             here, an INTEGER*4 is sufficient (using MJD instead of JD is
             preferred)
  DATETIME : a quantity liable to be displayed as YYYY-MM-DD-hh:mm:ss
             that is, similar to the previous one in usage, but with a
             precision at the level of seconds.
             Again more likely to be used in headers than in data areas.
             A running count of seconds since an arbitrary zero, expressed
             as an INTEGER*4, is good. I would recommend using seconds since
             1 Jan 1970 - which is what most Unix systems do - and which
             will cover about 68 years since then.

  TIME :     here is where I am in trouble. I want to express times with
             an arbitrarily high precision (say 1 ms or less). This is
             more the case of a column in a binary table (say, representing
             the time of a bin in a time profile, or the arrival time of a
             photon).
             I am very reluctant to renounce the usage of traditional data
             types, namely either INTEGER*4 or DOUBLE PRECISION.
             One way to cut out the number of significant figures needed is
             therefore to subtract a standard offset from the values.
             I would be personally inclined to use a representation like
             the following :

          -  time in an arbitrary unit (this most likely will be either a
             negative power of 2, 2**-N seconds, in an INTEGER quantity, or
             a decimal fraction like 1 ms, 10 ms, 0.1 ms in a DOUBLE PRECISION
             quantity) since 0 UT of the day containing the first bin or
             first photon of the dataset. This is the quantity stored as
             "data".
          -  a keyword in the header will define the time units (e.g. a
             real number as a fraction of seconds)
          -  another keyword in the header will define the reference time
             corresponding to "0" units, this could be a quantity of type
             DATETIME as above

             In general one does not need the full range of figures when
             doing processing, nor the units to be seconds (if one is doing
             a Fourier transform, one would anyhow subtract the time of the
             first bin, and one would have a power spectrum vs a frequency
             in 1/timeunit instead of Hz), but just when displaying the data
             and in such case one can do the conversion using the info in
             the header keywords

>Currently, we  express this number as 2 separate
>This however has several disadvantages:
>	- it requires 2 keywords = 160 bytes to express the value

  Ah ... sorry Bill. This is the usual "FITS as storage system vs FITS as
  transport system"  ... of course using a native internal representation
  for local storage will allow a more compact representation than FITS
  allows .... while for transport one can gladly spare 160 bytes

>	- it clutters up the header with extra keywords which makes it
>          more difficult for people to read
>The only difficultly with this is that there has been no convenient way
>to read or write values in this format, so I have proposed to add (in
>

  The real problem is NOT that there is no convenient way to READ or WRITE
  values in this format, but there is no portable way of STORING them in
  memory (REAL*16 is not standard). That's why I would try the best effort
  to avoid using quantities either than I*4 or R*8.

  The above line of reasoning tries to demonstrate that, at least in the
  area of expressing times and angles, one could do without using "exotic"
  representations.

  One is left however with an important problem, which is at a SEMANTIC level.
  That is, how to know a quantity (a binary table column, or an header
  keyword) is of a specific type (particularly if the type is a nonstandard
  one like DATETIME or ANGLE).
  In a non-FITS system this might be easier (for instance I store all kewyords
  as <type> <length> <name> <values>, and I could code <type> as I wish).
  But I am still left with the problem of expressing them into FITS kewyords
  for export (either a single "hyper-precision" one, like in Bill Pence's
  proposal, or a couple of associated keywords, which is after all not so
  awful).

  The only solution I may think of is of semantic nature, that is to know
  beforehand that a keyword with a particular NAME has a specific TYPE
  associated to it.
  This is already the case e.g. of NAXISn or TFIELDS, which everybody know
  are integers, or of DATE and DATE-OBS which are strings of a particular
  format, or of OBSERVER and ORIGIN, which are strings, etc. etc.
  Definition of additional conventions at this level should remain completely
  within the FITS spirit, and would be highly beneficial to facilitate
  data interchange.
  I would therefore endorse the case that such conventions be defined.

-----------------------------------------------------------------------------
Lucio Chiappetti - IFCTR/CNR Milano        |         U N I C U I Q U E
via Bassini 15 - I-20133 Milano - Italy    |          System  Manager
Internet: LUCIO at IFCTR.MI.CNR.IT            |           FORTUNAE SUAE
Decnet:   IFCTR::LUCIO (39610::LUCIO)      |
Bitnet:   EXOSAT at IMISIAM                   |
-----------------------------------------------------------------------------

---

From: thompson at stars.gsfc.nasa.gov (William Thompson, code 682.1, x2040)
Date: Mon, 5 Oct 1992 18:01:00 GMT

In article <9210051405.AA21114 at fits.cv.nrao.edu>, Lucio Chiappetti <lucio at ifctr.mi.cnr.it> writes...
>  The sort of data types I am thinking of are quantities like :

	(He goes on to describe suggested definitions of the quantities ANGLE,
	HANGLE, DATE, DATETIME, and TIME.  Definitions omitted to save space.)

I strongly suggest looking into the document "Time Code Formats" from the
Consultative Committee for Space Data Systems, CCSDS 301.0-B-2, April 1990.
This is available from

	CCSDS Secretariat
	Communications and Data Systems Division (Code-OS)
	National Aeronatuics and Space Administration
	Washington, DC 20546, USA

The topic of encoding date and time values into binary and ASCII values is
discussed there in some detail.  The suggested format for ASCII encoding is
briefly outlined below for the following example (5 Oct 1992 at 1:23:45 UT)

			   1992-10-05T01:23:45.000Z

The number of decimal places after the seconds field ("45") can be whatever is
appropriate, or can be omitted entirely.  Encoding only date, or only time,
values is also discussed.

It's always better to use a pre-existing standard than to reinvent the wheel.

On another note, I agree with Wayne Landsman that encoding the more-than-double
precision value as a character string, i.e.

is preferable to

Bill Thompson