Re: [go-nuts] golang multiple go routines reading from a channel and performance implications

[Robert Engels]

You need to determine how well they parallelize and what the resource consumption of a request is. For example, if every request can run concurrently at 100% (not possible btw because of switching overhead), and each request takes 0.5 secs of CPU, and 1.5 secs of IO, for a total wall time of 2 secs). At 2k request/per sec, you need a machine with 1000 CPUs. IO can run concurrently on most modern setups, so you can essentially factor this out, less so if most of the operations are writes.

Your local CPU requirements may be less if the request is then handled by a cluster (over the network, database etc), but you will still need 1000 cpus in the cluster (probably a lot more due to the network overhead).

You can look at github.com/robaho/go-trader for an example of very high CPU based processing using Go and channels (and other concurrency structs).

-----Original Message-----
From: Sankar
Sent: Nov 21, 2019 12:30 PM
To: golang-nuts
Subject: [go-nuts] golang multiple go routines reading from a channel and performance implications

We have a setup where we have a producer goroutine pumping in a few thousand objects into a channel (Approximately 2k requests per second). There are a configurable number of goroutines that work as consumers, consuming from this single channel. If none of the consumer threads could receive the message, the message gets discarded. Each consumer go routine takes about 2 seconds for each work to be completed, by which they will come back to read the next item in the channel. The channel is sized such that it can hold up to 10,000 messages.

The code is roughly something like:

producer.go:

func produce() {

 ch <- item

}

func consumer() {

 for i:=0; i < NumberOfWorkers; i ++ {

   go func() {

      for _, item := range ch {

         // process item

      }

   } ()

 }

}

With this above setup, we are seeing about 40% of our messages getting dropped.

So my questions are:

1) In such a high velocity incoming data, will this above design work ? (Producer, Consumer Worker Threads)

2) We did not go for an external middleware for saving the message and processing data later, as we are concerned about latency for now.

3) Are channels bad for such an approach ? Are there any other alternate performant mechanism to achieve this in the go way ?

4) Are there any sample FOSS projects that we can refer to see such performant code ? Any other book, tutorial, video or some such for these high performance Golang application development guidelines ?

I am planning to do some profiling of our system to see where the performance is getting dropped, but before that I wanted to ask here, in case there are any best-known-methods that I am missing out on. Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f8b5d9fb-d9b7-44b8-bc50-70dcb5f10cd0%40googlegroups.com.

-- 

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/225583457.2024.1574364299085%40wamui-scooby.atl.sa.earthlink.net.