On Wed, 14 Mar 2018 11:58:46 -0700 Rio <m...@riobard.com> wrote: > > While implementing a SOCKS proxy in Go, I ran into an issue which is better > explained in details by Evan Klitzke in this post > https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage > > In my case, each proxied connection costs two goroutines and two buffers in > blocking read. For TCP connections the buffer size can be small (e.g. 2kb), > so the overhead per proxied TCP connection is 8kb (2 x 2kb goroutine stack > + 2 x 2kb read buffer). For UDP connections the buffer size must be large > enough to hold the largest packet due to the nature of packet-oriented > network, so the overhead per proxied UDP connection is 132kb (2 x 2kb > goroutine stack + 2 x 64kb read buffer for largest UDP packet). Handling > 10,000 UDP proxied connections requires at least 1.25GB memory, which is > unnecessary if there's a way to poll I/O readiness and use a shared read > buffer.
You will need 10K*64KiB*2 = 1.22GiB kernel bufferspace at a minimum in any case. That is still one datagram's worth of buffering per UDP socket so you can still lose some packets. This is in addition to any user level buffering passd in to recv() or read(). So you need at least 2.48GiB! > I'm wondering if there's a better way other than calling > syscall.Epoll/Kqueue to create custom poller? I did such a custom poller for a network proxy. It was a pain to get right (I only did the Eww!poll version). In my case all the traffic was via tcp. It has been a while now but IIRC my reason was not to save memory but to gain flexibility over Go runtime's networking. I don't remember the details now though. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
