[go-nuts] Re: Issue with integration test coverage

[savita juyal]

Thank you so much Brian for helping me identify the problem. Now I am using 
kill -2 to pass SIGINT to the process, with this program is exiting 
politely and we are getting covcounter files. 

Regards
Savita Juyal
On Tuesday, July 11, 2023 at 7:18:57 PM UTC+5:30 Brian Candler wrote:

> Why are you using kill -9?  That doesn't allow the process any time to 
> tidy up whilst exiting.
>
> But also see https://go.dev/testing/coverage/#FAQ
>
> "If my program panics, will coverage data be written?
>
> Programs built with go build -cover will only write out complete profile 
> data at the end of execution if the program invokes os.Exit() or returns 
> normally from main.main. If a program terminates in an unrecovered panic, 
> or if the program hits a fatal exception (such as a segmentation violation, 
> divide by zero, etc), profile data from statements executed during the run 
> will be lost."
>
> On Tuesday, 11 July 2023 at 14:17:18 UTC+1 savita juyal wrote:
>
>> Hello All,
>>
>> I am trying to collect integration test coverage but its not generating 
>> covcounters file. Please looks steps I am following to generate covearge.
>>
>> - *Build go binary* : go build -race -*cover* -o 
>> $(ROOT)/go/bin/appwithrace -ldflags "-X app/handler.proxyVersion=$(VERSION) 
>> -X app/handler.gitVersion=$(GITVERSION) -X 
>> app/handler.buildTime=$(BUILDTIME)" cmd/proxy/proxy.go
>>
>> - *Run go binary* : ENABLE_TLS=false ENABLE_TLS_TO_DB=false 
>> LOAD_BALANCED=false *GOCOVERDIR=./coverage/int *./go/bin/appwithrace 
>> 2>&1 >/dev/null &
>>
>> - *Exit go program* : Kill -9 go_process_id
>>
>> Now we only covmetadata file generated under the given path. Please me 
>> know what I am doing wrong or if I am missing something?
>>
>> Regards
>> Savita Juyal
>>
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/857eb25e-3f51-4f2c-b7c2-167389c3218en%40googlegroups.com.

[go-nuts] How to minimize RAM usage during Go binary compilation

[Jason Kulatunga]

Hi,

I’m doing something a bit unusual with my application and I’m trying to 
figure out if there’s a better way to do this that I’m missing. 

Basically my application Fasten Health 
 is designed to allow 
patients to pull their medical records from healthcare institutions — of 
with there are 10,000 currently supported, and 100,000s more that we'd like 
to support in the US. 

We have a repo/library called fasten-sources 
 which is made up of mostly 
generated code (using dave/jennifer ).
It has a couple of primary packages:


   - pkg 
   
 
   which contains a list 10,000 enums, one for each support healthcare 
   institution.
   - definitions/internal/source 
   

 
   which contains a public function defining the OAuth configuration for each 
   healthcare institution
   - clients/internal/source 
   

 
   which contains a public function effectively instantiating a custom 
   API/Http client for each healthcare institution


All external references to this library use a Factory pattern - 
https://github.com/fastenhealth/fasten-sources/blob/main/clients/factory/factory.go
  
— so the public functions do not need to be public in reality. 


At this point the build of our application (which leverages this library) 
is causing OOM issues during our Docker image build 

 
on Github Actions and custom Depot.dev build nodes (64GB of memory)

Here’s what I’m wondering:


   1. Could this OOM issue be caused by the number of public symbols in the 
   packages? As we start supporting more and more institutions, I’d expect 
   this number to grow by orders of magnitude, which will definitely cause 
   even more OOM issues
  - My functions/structs all match an interface, so if I change my 
  public functions to private functions, would that solve the problem in a 
  permanent way?
   2. Is the OOM issue caused by the number of files in a package? Instead 
   of having 100,000s of individual files, if I had 1 massive file, would have 
   have any impact?
   3. Is the OOM issue caused by the number of symbols in a single package? 
   Should I find a way to distribute the enums, functions across multiple 
   packages would have have any impact?
   4. Does the `internal` path have any impact on RAM usage during 
   compilation?
   5. Instead of using a switch in the Factory for instantiating the 
   clients and definition files, is there some other compiler friendly way to 
   do this instead?


Basically I recognize that I'm doing something a bit unusual, but I cant 
find a way around it. I need to support 100,000's of institutions in the 
future, so bandaid fixes (or "just use a bigger build machine") won't work 
for us. A fundamental refactor of the repo is completely acceptable, since 
its all generated code. I just don't know which changes will actually 
impact the amount of RAM used during compilation

Thanks!
-Jason

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com.

Re: [go-nuts] How to minimize RAM usage during Go binary compilation

['Sean Liao' via golang-nuts]

considering it is primarily static information, maybe it should be passed
in and as data during run time, instead of being embedded and compiled as
code?

- sean

On Wed, Jul 12, 2023, 13:18 Jason Kulatunga  wrote:

> Hi,
>
> I’m doing something a bit unusual with my application and I’m trying to
> figure out if there’s a better way to do this that I’m missing.
>
> Basically my application Fasten Health
>  is designed to allow
> patients to pull their medical records from healthcare institutions — of
> with there are 10,000 currently supported, and 100,000s more that we'd like
> to support in the US.
>
> We have a repo/library called fasten-sources
>  which is made up of
> mostly generated code (using dave/jennifer
> ).
> It has a couple of primary packages:
>
>
>- pkg
>
> 
>which contains a list 10,000 enums, one for each support healthcare
>institution.
>- definitions/internal/source
>
> 
>which contains a public function defining the OAuth configuration for each
>healthcare institution
>- clients/internal/source
>
> 
>which contains a public function effectively instantiating a custom
>API/Http client for each healthcare institution
>
>
> All external references to this library use a Factory pattern -
> https://github.com/fastenhealth/fasten-sources/blob/main/clients/factory/factory.go
> — so the public functions do not need to be public in reality.
>
>
> At this point the build of our application (which leverages this library)
> is causing OOM issues during our Docker image build
> 
> on Github Actions and custom Depot.dev build nodes (64GB of memory)
>
> Here’s what I’m wondering:
>
>
>1. Could this OOM issue be caused by the number of public symbols in
>the packages? As we start supporting more and more institutions, I’d expect
>this number to grow by orders of magnitude, which will definitely cause
>even more OOM issues
>   - My functions/structs all match an interface, so if I change my
>   public functions to private functions, would that solve the problem in a
>   permanent way?
>2. Is the OOM issue caused by the number of files in a package?
>Instead of having 100,000s of individual files, if I had 1 massive file,
>would have have any impact?
>3. Is the OOM issue caused by the number of symbols in a single
>package? Should I find a way to distribute the enums, functions across
>multiple packages would have have any impact?
>4. Does the `internal` path have any impact on RAM usage during
>compilation?
>5. Instead of using a switch in the Factory for instantiating the
>clients and definition files, is there some other compiler friendly way to
>do this instead?
>
>
> Basically I recognize that I'm doing something a bit unusual, but I cant
> find a way around it. I need to support 100,000's of institutions in the
> future, so bandaid fixes (or "just use a bigger build machine") won't work
> for us. A fundamental refactor of the repo is completely acceptable, since
> its all generated code. I just don't know which changes will actually
> impact the amount of RAM used during compilation
>
> Thanks!
> -Jason
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/f870ec97-5ac5-4211-9379-5f2c0dd97e0bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGabyPrM_G6yzh19mLHCcKAkY4cSUMPAzXhCkQdSNZ35aEdUJg%40mail.gmail.com.

Re: [go-nuts] How to minimize RAM usage during Go binary compilation

[Diego Joss]

Or it could be embedded using package embed (https://pkg.go.dev/embed).
Just another candidate solution.

-- Diego

On Wed, 12 Jul 2023 at 14:30, 'Sean Liao' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> considering it is primarily static information, maybe it should be passed
> in and as data during run time, instead of being embedded and compiled as
> code?
>
> - sean
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGjxhKkk9JoZqiMkZT0YvyOkj4M_5RzTJisYkbMxE_puOYbPuQ%40mail.gmail.com.