On Mon, Feb 9, 2015 at 1:32 PM, Jason Baron <jba...@akamai.com> wrote: > On 02/09/2015 03:18 PM, Andy Lutomirski wrote: >> On 02/09/2015 12:06 PM, Jason Baron wrote: >>> Epoll file descriptors that are added to a shared wakeup source are always >>> added in a non-exclusive manner. That means that when we have multiple epoll >>> fds attached to a shared wakeup source they are all woken up. This can >>> lead to excessive cpu usage and uneven load distribution. >>> >>> This patch introduces two new 'events' flags that are intended to be used >>> with EPOLL_CTL_ADD operations. EPOLLEXCLUSIVE, adds the epoll fd to the >>> event >>> source in an exclusive manner such that the minimum number of threads are >>> woken. EPOLLROUNDROBIN, which depends on EPOLLEXCLUSIVE also being set, can >>> also be added to the 'events' flag, such that we round robin around the set >>> of waiting threads. >>> >>> An implementation note is that in the epoll wakeup routine, >>> 'ep_poll_callback()', if EPOLLROUNDROBIN is set, we return 1, for a >>> successful >>> wakeup, only when there are current waiters. The idea is to use this >>> additional >>> heuristic in order minimize wakeup latencies. >> >> I don't understand what this is intended to do. >> >> If an event has EPOLLONESHOT, then this only one thread should be woken >> regardless, right? If not, isn't that just a bug that should be fixed? >> > > hmm...so with EPOLLONESHOT you basically get notified once about an event. If > i have multiple epoll fds (say 1 per-thread) attached to a single source in > EPOLLONESHOT, then all threads will potentially get woken up once per event. > Then, I would have to re-arm all of them. So I don't think this addresses > this particular usecase...what I am trying to avoid is this mass wakeup or > thundering herd for a shared event source.
Now I understand. Why are you using multiple epollfds? --Andy > >> If an event has EPOLLET, then the considerations are similar to >> EPOLLONESHOT, right? >> > > EPOLLET is still going to cause this thundering herd. > >> If an event is a normal level-triggered non-one-shot event, then I don't >> understand how a round-robin wakeup makes any sense. It's level-triggered, >> after all. > > Yeah, so the current behavior is to wake up all of the threads. I'm trying to > add a new mode where it load balances among the threads interested in the > event. Perhaps, the test program I attached to 0/2 will show the issue better? > > Also, this originally came up in the context of a single listening socket > which was attached to multiple epoll fds each in a separate thread. With the > attached patch, I can measure a large decrease in cpu usage and better > balancing behavior among the accepting threads. > > Thanks, > > -Jason -- Andy Lutomirski AMA Capital Management, LLC -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
