I'm not an expert at all (yet) so please be kind... I recently started writing a converter from our proprietary measurements format to HDF5 in C# and using the HDF.PInvoke nuget distribution. I've read that the HDF.PInvoke is the way forward and hopefully it discloses all the features you need.
Maybe too trivial to mention but it is important to implement IDispose properly on classes that have pointers to HDF entities to ensure that they are properly released on software failure and release of managed objects. But this won't save the file if your system halts in such a way that the disposers and finalizers and garbage collection won't be executed anymore. Kind regards, Gerco Gerco de Jager | Software Engineer | MARIN Software Group MARIN | T +31 317 49 33 51 | mailto:g.d.ja...@marin.nl | http://www.marin.nl MARIN news: http://www.marin.nl/web/News/News-items/Predicting-broadband-hull-pressure-fluctuations-and-underwater-radiated-noise.htm -----Original Message----- From: Hdf-forum [mailto:hdf-forum-boun...@lists.hdfgroup.org] On Behalf Of Ewan Makepeace Sent: vrijdag 22 september 2017 3:34 To: hdf-forum@lists.hdfgroup.org Subject: [Hdf-forum] Avoiding corruption of the HDF5 File Dear Experts, We are building a data acquisition and processing system on top of an HDF5 file store. Generally we have been very pleased with HDF5 - great flexibility in data structure, performant, small file size, availability of third party data access tools etc. However our system needs to run for 36-48 hours at a time - and we are finding that if we (deliberately or accidentally) stop the process while running (and writing data) the file is corrupted and we lose all our work. We are in C# and wrote our access routines on top of HDF5.net (which I understand is deprecated). We tend to keep all active pointer objects open for the duration of the process that reads or writes them (file, group and dataset handles in particular). 1) Is there a full featured replacement for HDF5.net now, that I was unaware of? Previous contenders were found to be missing support for features we depend on. If so will it address the corruption issue? 2) Should we be opening and closing all the entities on every write? I would have thought that would dramatically slow access but perhaps not. Guidance? 3) Are there any other tips to making the file less susceptible to corruption if writing is abandoned unexpectedly? Please help - this issue could be serious enough to make us reconsider our storage choice, which would be expensive now. rgds, Ewan _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5 _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
