We have a setup where we have a producer goroutine pumping in a few
thousand objects into a channel (Approximately 2k requests per second).
There are a configurable number of goroutines that work as consumers,
consuming from this single channel. If none of the consumer threads could
receive the message, the message gets discarded. Each consumer go routine
takes about 2 seconds for each work to be completed, by which they will
come back to read the next item in the channel. The channel is sized such
that it can hold up to 10,000 messages.
The code is roughly something like:
producer.go:
func produce() {
ch <- item
}
func consumer() {
for i:=0; i < NumberOfWorkers; i ++ {
go func() {
for _, item := range ch {
// process item
}
} ()
}
}
With this above setup, we are seeing about 40% of our messages getting
dropped.
So my questions are:
1) In such a high velocity incoming data, will this above design work ?
(Producer, Consumer Worker Threads)
2) We did not go for an external middleware for saving the message and
processing data later, as we are concerned about latency for now.
3) Are channels bad for such an approach ? Are there any other alternate
performant mechanism to achieve this in the go way ?
4) Are there any sample FOSS projects that we can refer to see such
performant code ? Any other book, tutorial, video or some such for these
high performance Golang application development guidelines ?
I am planning to do some profiling of our system to see where the
performance is getting dropped, but before that I wanted to ask here, in
case there are any best-known-methods that I am missing out on. Thanks.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/f8b5d9fb-d9b7-44b8-bc50-70dcb5f10cd0%40googlegroups.com.
