Thanks, I'll check it out. On Sun, 11 May 2025, 22:44 Robert Engels, <reng...@ix.netcom.com> wrote:
> > This seems to be a decent meta reference > https://www.linkedin.com/advice/0/what-key-skills-tools-cloud-performance-tuning?utm_source=share&utm_medium=member_ios&utm_campaign=share_via#:~:text=Cloud%20performance%20tuning%20is%20the,%2C%20errors%2C%20and%20resource%20wastage. > > But this is probably more focused > https://cloud.google.com/blog/topics/developers-practitioners/optimize-cloud-storage-upload-performance-client-libraries > > On May 11, 2025, at 11:24 AM, Kanak Bhatia <kanakbhati...@gmail.com> > wrote: > > > > Thanks for the tip! > > Could you suggest some resources for the same regarding these design > considerations if you know of any ? > > On Sun, 11 May 2025, 21:48 Robert Engels, <reng...@ix.netcom.com> wrote: > >> Of that I’m not sure. It is based on deep knowledge of how these things >> are built from the hardware to the kernel to network protocols to the >> service layers. >> >> It broadly falls under performance tuning of which there is lots of >> literature. >> >> The key element of how channels and go routines play into this - is that >> when dealing with synchronous requests to a service (many services offer >> async endpoints) you need to be able to parallelize these. Then you need to >> be able to provide the data as fast as the network can accept it - most >> networks today are faster than disk storage - even some SSD. So then the >> disk storage array becomes the bottleneck and you need to parallelize that >> (RAID etc) and the kernel access. >> >> Often for massive uploads you might even partition the job across >> machines each with a portion of the data set. >> >> On May 11, 2025, at 11:04 AM, Kanak Bhatia <kanakbhati...@gmail.com> >> wrote: >> >> >> >> Does we have any kind of documented material or mathematical theory type >> stuff for these things or is it more like hit or try >> >> On Sun, 11 May 2025, 08:04 ren...@ix.netcom.com, <reng...@ix.netcom.com> >> wrote: >> >>> The two most likely limiters in performance will be your network pipe to >>> the cloud and the QPS quota offered by the service. If you are not reaching >>> those limits you should increase the parallelism until you do. If your cpu >>> becomes saturated first you probably need larger buffer sizes in the I/o. >>> >>> On Thursday, May 8, 2025 at 3:48:36 PM UTC-5 Kanak Bhatia wrote: >>> >>>> Does anyone have idea how to optimize consumers, producers and channels >>>> using golang. I have to upload a million objects to a cloud object storage, >>>> but unable to get a optimzed values for above parametres. Producers used to >>>> create objects and send data through channels and receive them at >>>> consumers and call api from there. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts+unsubscr...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com >>> <https://groups.google.com/d/msgid/golang-nuts/6af4acac-c8c8-462c-aad4-08e1df73f694n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CABXaED5ShjqBfwQfm72o%2BikffDuTvb1MXVHJXk%2BtCVUtRW_VtQ%40mail.gmail.com.
