Hi Rafal,
it looks both the dimensions and the structural type of your dataset
are supposed to be dynamical in your use case. This sounds like a demand
that would be possible but unhealthily nonperforming to manage if all is
put into one dataset that is dynamically updated and re-organized both
in memory and disk when inserting new data.
I'd rather use a group with many datasets in such a case. Each dataset
can have unlimited dimensions in the one direction where you update
data, but use only one type for each dataset, not a compound structure.
So if a new column is added, you add a new dataset. To iterate over the
compounds over your data type, you iterate over the containing group and
check the type of each dataset there.
Admittedly I have no experience with the HDF5 Table API, probably it's
not possible to use that one and you'd need to use the lower level H5D
and H5G API's.
Regards,
Werner
On 24.08.2017 06:47, Rafal Lichwala wrote:
Hi Walter, hi All,
Thank you for sharing your work.
I've analyzed your codes briefly and it seems you manually create a
dataset with compound type and then you put your values in such a
dataset - is that correct?
But that means for my use case I need to collect all columns (their
types) first and then create a compound dataset.
When the number of such columns is really huge this operation can be
time and resource consuming. But that's OK if there is no other
solution...
If I well understand your codes, you are collecting your columns
(which are separate classes in your case) just in vectors and then you
calculating columns offsets basing on std::vector::data - is that
correct?
Any other suggestions from HDF Forum Team which could help to solve my
use case?
Thank you.
Regards,
Rafal
W dniu 2017-08-23 o 16:25, Walter Landry pisze:
Rafal Lichwala <syri...@man.poznan.pl> wrote:
Let's say my final HDF table will look like this:
['a', 1, 3.14]
['b', 2, 2.11]
['c', 3, 1.89]
So we simply have a HDF table with 3 columns of types: char, int,
float
and 3 rows with some values.
Creation of that table must be divided into some "steps".
After 1st "step" I should have a table:
['a']
After 2nd step:
['a', 1]
After 3rd step:
['a', 1, 3.14]
After 4th step:
['a', 1, 3.14]
['b', x, x]
where x after 4th step is undefined and can be some default values
which will be overwritten in the next steps.
How to achieve that use case?
I have to do something similar for my program tablator
https://github.com/Caltech-IPAC/tablator
I read in tables in other formats and write out as HDF5. So I do not
know the types of the rows at compile time. It is all in C++. The
details of writing HDF5 are in
src/Table/write_hdf5/
Is it possible to create a table by calling H5TBmake_table(), but
having no fields and no records at the beginning and then just call
H5TBinsert_field() in the next steps?
I do not think that is going to work, because you need to know the
sizes of rows when you create the table.
Is it possible to have "data" attribute of H5TBinsert_field() function
a NULL value when we insert a new field to a table dataset with no
records yet?
What about 4th step - can I create just a first column value for a new
record in a table?
I do not know of a way to do that. I would end up creating a whole
new table with the new field. You can then populate the empty fields
with appropriate default values.
I know it's maybe a strange use case, but the problem is that I could
have really huge structure model (a lot of columns and a lot of
records) which should be stored in the HDF table dataset, so I need to
avoid "collecting" required information (number of fields, their
types, values) by initial iterating over whole structure.
The second problem is that I have a vector of objects which need to be
stored as HDF table (where table row is the given object and columns
are its fields), but all examples I've seen just work on C struct.
That sounds similar to the internal data structure I use in tablator.
Hope that helps,
Walter Landry
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
