Oh, regarding the comments about breaking tool chains: valgrind fails on this binary:
$ valgrind --version valgrind-3.16.1 $ valgrind ./goembeddb -S -q valgrind: mmap(0x54e000, 1977884672) failed in UME with error 22 (Invalid argument). valgrind: this can be caused by executables with very large text, data or bss segments. $ Previously reported: https://bugs.kde.org/show_bug.cgi?id=290061 Glen On Thursday, September 23, 2021 at 8:15:36 PM UTC-4 Glen Newton wrote: > I have a MWE here: https://github.com/gnewton/goembeddb > > TLDR: a MWE embedding a key-value store file of 20 million k/v pairs > (1.9GB) into a Go binary and using the db from the binary. > > The same Go program that writes the db then is recompiled, embedding the > db in the newly compiled Go binary. Size: ~1.9GB > > Then the same Go program is used to read the embedded db. > > *To read 1 random record from the embedded 1.9GB 20 million record > key-value db, takes 133.166µs, 2244k (~2.2MB) resident. Cold start.* > This does not seem to bad, for this particular use case. :-) > > I'd be interested in trying out BBolt <https://github.com/etcd-io/bbolt> > or Badger <https://github.com/dgraph-io/badger>, as the K/V db I am using > here (cdb https://en.wikipedia.org/wiki/Cdb_(software) ) does not support > prefix or range scans or buckets. > > Comments welcome. > > Thanks, > Glen > > On Wednesday, September 22, 2021 at 1:22:53 PM UTC-4 Howard C. Shaw III > wrote: > >> Before the addition of binary packaged assets into Go as a standard >> library feature, there were various tools to accomplish the same task. Some >> of them, such as https://github.com/GeertJohan/go.rice , could use an >> alternate embedding. Basically, instead of having the binary files packaged >> as Go code in the executable itself, they simply appended a .zip file to >> the binary, and accessed that directly. >> >> Now, I am not suggesting that as a direct solution (though it may be, as >> Go does apparently support Zip64 which does away with the 4GB limit), but >> perhaps instead looking at how it does what it does, and adapting that to >> simply appending a sqlite or other single-file database format to the Go >> binary and using the same basic technique for accessing it. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4df6d635-630d-42ae-8106-a4dbc426ed17n%40googlegroups.com.
