Hi, I’m doing something a bit unusual with my application and I’m trying to figure out if there’s a better way to do this that I’m missing.
Basically my application Fasten Health <https://github.com/fastenhealth/fasten-onprem> is designed to allow patients to pull their medical records from healthcare institutions — of with there are 10,000 currently supported, and 100,000s more that we'd like to support in the US. We have a repo/library called fasten-sources <https://github.com/fastenhealth/fasten-sources> which is made up of mostly generated code (using dave/jennifer <https://github.com/dave/jennifer>). It has a couple of primary packages: - pkg <https://github.com/fastenhealth/fasten-sources/blob/main/pkg/source_type.go> which contains a list 10,000 enums, one for each support healthcare institution. - definitions/internal/source <https://github.com/fastenhealth/fasten-sources/tree/main/definitions/internal/source> which contains a public function defining the OAuth configuration for each healthcare institution - clients/internal/source <https://github.com/fastenhealth/fasten-sources/tree/main/clients/internal/source> which contains a public function effectively instantiating a custom API/Http client for each healthcare institution All external references to this library use a Factory pattern - https://github.com/fastenhealth/fasten-sources/blob/main/clients/factory/factory.go — so the public functions do not need to be public in reality. At this point the build of our application (which leverages this library) is causing OOM issues during our Docker image build <https://github.com/fastenhealth/fasten-onprem/actions/runs/5515748969/jobs/10056394327> on Github Actions and custom Depot.dev build nodes (64GB of memory) Here’s what I’m wondering: 1. Could this OOM issue be caused by the number of public symbols in the packages? As we start supporting more and more institutions, I’d expect this number to grow by orders of magnitude, which will definitely cause even more OOM issues - My functions/structs all match an interface, so if I change my public functions to private functions, would that solve the problem in a permanent way? 2. Is the OOM issue caused by the number of files in a package? Instead of having 100,000s of individual files, if I had 1 massive file, would have have any impact? 3. Is the OOM issue caused by the number of symbols in a single package? Should I find a way to distribute the enums, functions across multiple packages would have have any impact? 4. Does the `internal` path have any impact on RAM usage during compilation? 5. Instead of using a switch in the Factory for instantiating the clients and definition files, is there some other compiler friendly way to do this instead? Basically I recognize that I'm doing something a bit unusual, but I cant find a way around it. I need to support 100,000's of institutions in the future, so bandaid fixes (or "just use a bigger build machine") won't work for us. A fundamental refactor of the repo is completely acceptable, since its all generated code. I just don't know which changes will actually impact the amount of RAM used during compilation Thanks! -Jason -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com.
