Hi,
currently I am trying to create a virtual dataset for our measurement data. At
the moment the highest sample rate of the measurement system is 100 Hz and I
would like to collect the daily files into one virtual dataset which starts on
1st January, 2000 and end on 1st January, 2030. So this is a total dataset
length of 86400 s/day * 100 samples/s * 10958 days = 94677120000 which in turn
doesn't fit into a 32 bit number.
I use C# in combination with the HDF.PInvoke Nuget package. The parameters of
H5S.select_hyperslab (needed to create a virtual dataset) are actually of type
uint64 so at this point there is no problem. But when I create that virtual
dataset and analyze it later with h5dump, the start, stride, count and block
parameters of the hyperslab are truncated to 32 bit values.
In the C source code (hdf5-1.10.0-patch1) I identified three functions that are
related to that problem:
* H5S_hyper_serial_size (to calculate the size that is needed by an
hyperslab when it is serialized to a file)
* H5S_hyper_serialize (to serialize the hyperslab structure to a file)
* H5S_hyper_deserialize (to serialize the hyperslab structure from a file)
Each function checks if there are any unlimited dimensions in the hyperslab. If
true, then the hyperslab parameters are serialized as 64 bit numbers. If false,
a 32 bit macro is used. I can see the same behavior also when I inspect the
file with an hex-editor and locate the start/stride/count/block parameters (in
the 32 bit version I could only find the truncated start parameter).
So my question is: Is this a bug or is there a way to enforce the use of the
64-bit macro without modifying the source code?
I made a short C# example, which should be easily adaptable:
var fileId = H5F.create(@"C:\Users\Vincent\Desktop\HDF\VDS.h5",
H5F.ACC_TRUNC);
var spaceId_source = H5S.create_simple(1, new ulong[] { 10 }, new
ulong[] { H5S.UNLIMITED });
var spaceId_VDS = H5S.create_simple(1, new ulong[] { 100000000000 },
new ulong[] { H5S.UNLIMITED });
var dcpl = H5P.create(H5P.DATASET_CREATE);
H5S.select_hyperslab(spaceId_VDS, H5S.seloper_t.SET, new ulong[] {
90000000000 }, new ulong[] { 1 }, new ulong[] { 1 }, new ulong[] { 10 });
H5P.set_virtual(dcpl, spaceId_VDS, @"C:\Test.h5", "test_dataset",
spaceId_source);
var datasetId = H5D.create(fileId, "Test", H5T.NATIVE_INT,
spaceId_VDS, H5P.DEFAULT, dcpl, H5P.DEFAULT);
H5S.close(spaceId_VDS);
H5S.close(spaceId_source);
H5D.close(datasetId);
H5F.close(fileId);
If I then analyze that file with h5dump -p --header
"C:\Users\Vincent\Desktop\HDF\VDS.h5", I get the following:
HDF5 "C:\Users\Vincent\Desktop\HDF\file.h5" {
...
VIRTUAL {
SELECTION REGULAR_HYPERSLAB {
START (4100654080)
STRIDE (1)
COUNT (1)
BLOCK (10)
}
}
SOURCE {
FILE "C:\Test.h5"
DATASET "test_dataset"
SELECTION ALL
}
...
}
so the "start" parameter is:
410065408010
=
0x00F46B0400
instead it should be:
9000000000010
=
0x14F46B0400
Hopefully someone can guide me into the right direction.
Thank you very much
Vincent
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
