Subject: Re: HDF data structures support questions

From: [email-masked] (Doug Ilg)
Date: 25 Aug 1995 14:33:08 GMT

In article <[email-masked]>, [email-masked] (Robert Daniel Kennedy) writes:
|> Hello,
|> 
|> 	I am currently evaluating HDF for use in the storage of event data
|> organized by a rich set of data structures, for experiments in the year 1999.  
|> Data in an event comes from a large number of detectors, each sub-divided by elements
|> and readout devices. Efficiency of storage and the comprehensibility of the data 
|> structure are both issues in the evaluation, as well as the speed of reading/writing 
|> data of course.  I have gone through the documentation, FAQ, and so on... and have a few 
|> questions that remain unanswered.  Perhaps someone would be good enough to shed some light
|>  on these, or perhaps provide some off-the-cuff work-around.

No work-arounds necessary, Rob.  HDF does exactly what you want, already.

|> (1) Is there support for a variable-length array?  We often summarize a detector with a
|> just a few elements "hit" by a variable-length array such as
|> {Nhits, hit_location_1, hit_location2, ... hit_location_n}
|> where Nhits << number_of_elements in the detector.

Both SDSs and Vdatas are variable length structures.  SDSs can be extended along the
first dimension (using C dimension order) by specifying SD_UNLIMITED or NC_UNLIMITED
as the size of that dimension.  Vdatas can be extended in the record/element
"dimension."  There's no need to keep around info like Nhits, though.  HDF takes care
of that.
 
|> (2) Is there a way of "grouping" type information in the Vdata/Vgroup model
|> to reduce the type information overhead?  Many blocks in our data are the same
|> set of types repeated N times for N detector elements.  Our current data storage
|> format has a repeat count to reduce the size of the type specification tremendously...
|> N(type1, type2) means N consecutive groups of type1 followed by type2.

Okay, so maybe this is a work-around.  I didn't understand your question on first
reading.  Anyway, we're doing similar things in the EOS project at NASA.  There are
several ways that come to mind to handle this.

1) Re-order your data so that you have multiple instances of type1 and type2 sitting
next to each other.  Then you can use the `order` argument when you define your vdata
fields to get N(type1), N(type2).  Maybe not the best, but it would work.

2) Define each as an individual field and get type1(1), type2(1), type1(2), type2(2),... 
type1(N), type2(N).  This way seems quite tedious and really ignores the point of
grouping.

3) Define one field of a type whose size (times its order) is equal to sizeof(type1) +
sizeof(type2), then do your own monkeying around with picking apart the values after 
you read them.  This is not a good solution at all, but it *is* an option.

4) Define a vdata with one field of type1 and one of type2.  Put all data records for
detector 1 in the vdata, followed by all records for detector 2, etc.  Then create an
index vdata with (at least) a "begin" field and an "extent" field.  The index has one
record for each detector which tells where in the other vdata to look to find the data
for each detector.  For instance, if detector 1 has 6 records and detector 2 has 9,
the index would look like this:
                begin  extent
		  0      6
                  6      9
You can add any extra info in the index or the data vdata, wherever it makes sense.

This is the scheme we're using in EOS to handle what we call "Point Data."  It works 
well if your data is not growing dynamically.  It's a real pain to insert a new record 
for detector 1 after you have begun to write out records from detector 2, though.

If you need to do that, you could split the data records into several vdatas, one per
detector, and each one could grow independently.  Of course, then you need to worry
about a bunch of horribly fragmented vdatas, if you're not careful about how they're
written out.  A good buffering scheme goes a long way in this situation.

If you want to know more about our "Point Data Model," let me know.  I can probably
arrange to get you some documentation.

Hope this helps.

			-Doug Ilg

PS.  Check out the HDF Web server at "http://hdf.ncsa.uiuc.edu:8001/"  It's got lots
     of good info, including a page on HDF's limitations and copies of all the docs.
     Make sure you read the new "Users Guide."