Go back to the venerable SFX zip archives: Append the file as a (uncompressed?) ZIP file to the Go binary. That binary can then open itself as a ZIP and read/use the database, and every other program will see it as a zip file.
Or just simply append the database and its uint64 size to the go binary. Then it can open/mmap the exact bytes needed. glen....@gmail.com a következőt írta (2021. szeptember 21., kedd, 21:41:42 UTC+2): > Hello Ian, > > Yes, I think it is time I explain the 'why' of my inquiries into this. :-) > > My use case is this: Go, with its fast startup, pretty fast execution and > pretty small memory footprint, as well as it's ability to deploy as a > single binary, makes it a great language for cloud function-as-a-service > (FAAS). > > My specific FAAS use case is for static, read-only databases: I am looking > at embedding modest sized (few GB to 10s of GB) read-only databases in the > Go binary. > > This makes it possible to avoid the cost/complication of either using a > cloud db, or storing the db in (in the case of AWS) on a file system like > EFS (NFS), which the lambda has to mount. The databases are only updated > every couple of months / yearly. > This is perhaps rather an obscure use case, but one that I think a number > of people would want to take advantage of. > > >Note that such files are going to take a long time to create and will be > unwieldy to use. > Agreed. On my older desktop (Dell 7010 Kubuntu 20.10 4core i5 16GM) it > takes 45s to compile a trivial Go program that embeds three 500GB files. > > Actually, this is no different from those who want the simplicity and cost > savings of serving an entire static web site from embedded files. The > primary difference is the scale. > > Thanks, > Glen > > > On Tuesday, September 21, 2021 at 1:28:12 PM UTC-4 Ian Lance Taylor wrote: > >> On Tue, Sep 21, 2021 at 6:23 AM Glen Newton <glen....@gmail.com> wrote: >> > >> > Looking at https://groups.google.com/g/golang-codereviews/c/xkHLQHidF5s >> and https://github.com/golang/go/issues/9862 and the code to which they >> refer, it seems to me that the limit is 2GB independent of platform (is >> this true?), as the linker is limited to 2e9. >> > Again, should have done more of my homework! :-) >> > >> > >> https://github.com/golang/go/blob/master/src/cmd/internal/obj/objfile.go#L305 >> >> > >> > // cutoff is the maximum data section size permitted by the linker >> > // (see issue #9862). const cutoff = 2e9 // 2 GB (or so; looks better >> in errors than 2^31) >> > const cutoff = 2e9 // 2 GB (or so; looks better in errors than 2^31) >> > >> > The comment indicates that this is a limit in the linker; is this a >> limit in the elf format? If No, could the linker et al be modified to >> accept larger static data? >> >> You're right: it does look like cmd/link imposes a 2G total limit. >> This is not a limitation of the 64-bit ELF format. It should be >> possible to make it larger. >> >> Note that such files are going to take a long time to create and will >> be unwieldy to use. While it should be possible to increase the size, >> I would not be surprised if you hit other limits fairly quickly. >> >> Ian >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8c311d03-f9ee-4703-b9ce-1593ea3e227fn%40googlegroups.com.
