On 14.11.2012, at 08:21, Konstantin Belousov <kostik...@gmail.com> wrote:
> On Wed, Nov 14, 2012 at 01:41:04AM +0100, Markus Gebert wrote: >> >> On 13.11.2012, at 19:30, Markus Gebert <markus.geb...@hostpoint.ch> wrote: >> >>> To me it looks like the unix socket GC is triggered way too often and/or >>> running too long, which uses cpu and worse, causes a lot of contention >>> around the unp_list_lock which in turn causes delays for all processes >>> relaying on unix sockets for IPC. >>> >>> I don't know why the unp_gc() is called so often and what's triggering this. >> >> I have a guess now. Dovecot and relayd both use unix sockets heavily. >> According to dtrace uipc_detach() gets called quite often by dovecot closing >> unix sockets. Each time uipc_detach() is called unp_gc_task is >> taskqueue_enqueue()d if fds are inflight. >> >> in uipc_detach(): >> 682 if (local_unp_rights) >> 683 taskqueue_enqueue(taskqueue_thread, &unp_gc_task); >> >> We use relayd in a way that keeps the source address of the client when >> connecting to the backend server (transparent load balancing). This requires >> IP_BINDANY on the socket which cannot be set by unprivileged processes, so >> relayd sends the socket fd to the parent process just to set the socket >> option and send it back. This means an fd gets transferred twice for every >> new backend connection. >> >> So we have dovecot calling uipc_detach() often and relayd making it likely >> that fds are inflight (unp_rights > 0). With a certain amount of load this >> could cause unp_gc_task to be added to the thread taskq too often, slowing >> everything unix socket related down by holding global locks in unp_gc(). >> >> I don't know if the slowdown can even cause a negative feedback loop at some >> point by inreasing the chance of fds being inflight. This would explain why >> sometimes the condition goes away by itself and sometimes requires >> intervention (taking load away for a moment). >> >> I'll look into a way to (dis)prove all this tomorrow. Ideas still welcome >> :-). >> > > If the only issue is indeed too aggressive scheduling of the taskqueue, > than the postpone up to the next tick could do it. The patch below > tries to schedule the taskqueue for gc to the next tick if it is not yet > scheduled. Could you try it ? Sounds like a good idea, thanks! I'm testing the patch right now. It could take a few days to know it works for sure. I'll get back to you soon. Markus _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
