Re: [go-nuts] Re: Goroutines and channels optimization

[Robert Engels]

Of that I’m not sure. It is based on deep knowledge of how these things are built from the hardware to the kernel to network protocols to the service layers.  It broadly falls under performance tuning of which there is lots of literature.  The key element of how channels and go routines play into this - is that when dealing with synchronous requests to a service (many services offer async endpoints) you need to be able to parallelize these. Then you need to be able to provide the data as fast as the network can accept it - most networks today are faster than disk storage - even some SSD. So then the disk storage array becomes the bottleneck and you need to parallelize that (RAID etc) and the kernel access.  Often for massive uploads you might even partition the job across machines each with a portion of the data set.  On May 11, 2025, at 11:04 AM, Kanak Bhatia <kanakbhati...@gmail.com> wrote:  Does we have any kind of documented material or mathematical theory type stuff for these things or is it more like hit or try On Sun, 11 May 2025, 08:04 ren...@ix.netcom.com, <reng...@ix.netcom.com> wrote: The two most likely limiters in performance will be your network pipe to the cloud and the QPS quota offered by the service. If you are not reaching those limits you should increase the parallelism until you do. If your cpu becomes saturated first you probably need larger buffer sizes in the I/o.  On Thursday, May 8, 2025 at 3:48:36 PM UTC-5 Kanak Bhatia wrote: Does anyone have idea how to optimize consumers, producers and channels using golang. I have to upload a million objects to a cloud object storage, but unable to get a optimzed values for above parametres. Producers used to create objects and send data through channels and receive  them at consumers and call api from there. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/01BE4694-0490-4FC9-A0BC-5EF366D9849A%40ix.netcom.com.