Subject: Pure Go HDF5 implementation - 9 years later Hi all,
Replying to this 9-year-old thread because the original question has finally been answered. Back in 2015, the consensus was that HDF5 is "so complicated that there is only one implementation" and too difficult for pure Go. I'm happy to report that's no longer true. **What exists now (2025)**: A pure Go HDF5 library with full read support and beta write support: - Repository: https://github.com/scigolib/hdf5 - Read: Feature-complete (superblock v0/2/3, all layouts, compression, attributes) - Write: Beta (v0.11.1-beta - chunked datasets, GZIP, dense groups, attributes) ```go // Reading (works today) file, _ := hdf5.Open("data.h5") dataset := file.Datasets["/temperature"] data := dataset.Data() // []float64, []int32, etc. // Writing (beta, but functional) file, _ := hdf5.CreateForWrite("output.h5", hdf5.Truncate) file.CreateDataset("data", myData, hdf5.WithChunked([]uint64{100, 100}), hdf5.WithCompression(6), ) ``` **How it was done**: The C library (D:\projects\scigolibs\hdf5c\src) served as reference implementation. Instead of "figuring out" the format, we ported proven algorithms to Go. Format spec + reference code = solvable problem. Development time: ~1 year from concept to write MVP (with AI assistance for rapid prototyping). **Why it matters**: - No CGo = actual cross-compilation, no C dependencies - Type safety = Go's compiler catches HDF5 format errors at compile time - Standard library integration (io.ReaderAt, encoding/binary) - Single binary deployment **Current status**: - Test coverage: 70-88% depending on package - Platforms: Linux, macOS, Windows - Recognition: HDF Group acknowledged it on their forum - Beta limitations: Some write features in progress (dense storage read-modify-write, h5dump compatibility) **For the scientific Go community**: If you're working with HDF5 files and want to avoid CGo, this is now viable. Looking for beta testers with real-world datasets (astronomy, climate, genomics, etc.). Installation: `go get github.com/scigolib/[email protected]` The format is indeed complex, but it's been tackled. Thought this group might appreciate the update after 9 years. Best, Andrey Kolkov P.S. NetCDF4 being "stripped down HDF5" means this library could potentially support it too, though that's not implemented yet. On Monday, 2 April 2012 at 12:34:23 UTC+4 Sebastien Binet wrote: > Rémy Oudompheng <[email protected]> writes: > > > Le 31 mars 2012 22:30, Fazlul Shahriar <[email protected]> a écrit : > >> I started on hdf4 a while ago: https://bitbucket.org/fhs/gohdf > >> I haven't had time to work on it further, but I'm very much interested. > >> > >> I also want to work on hdf5 but I don't deal with hdf5 as often as > >> hdf4. NetCDF4 format is pretty much a stripped down version of HDF5 as > >> far as I know, and I think NetCDF3 is the simplest format to > >> implement. > >> > >> Of course, you can always use cgo. Both pytables and pyhdf are > >> bindings to the C libraries. > > > > Sébastien Binet has Go bindings for libhdf5 at > > https://bitbucket.org/binet/go-hdf5/ > > I didn't try them and I don't know if it uses reflection for dataset > > reading/writing. > > it does: > https://bitbucket.org/binet/go-hdf5/src/50c6c6f0bdc4/pkg/hdf5/h5t.go#cl-386 > > -s > > -- > ######################################### > # Dr. Sebastien Binet > # Laboratoire de l'Accelerateur Lineaire > # Universite Paris-Sud XI > # Batiment 200 > # 91898 Orsay > ######################################### > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/6ffaff5a-d93d-480a-8f4e-8ebefeebf0e0n%40googlegroups.com.
