Date: Tue, 13 Nov 2012 17:12:44 -0800 From: Adrian Chadd <adrian@freebsd.org> To: Markus Gebert <markus.gebert@hostpoint.ch> Cc: freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: thread taskq / unp_gc() using 100% cpu and stalling unix socket IPC Message-ID: <CAJ-Vmo=36Ob0NSeFVV4goLsaca7Aqc9B0zdPvYWEcNmBPsk40Q@mail.gmail.com> In-Reply-To: <007F7A73-75F6-48A6-9C01-E7C179CDA48A@hostpoint.ch> References: <6908B498-6978-4995-B081-8D504ECB5C0A@hostpoint.ch> <007F7A73-75F6-48A6-9C01-E7C179CDA48A@hostpoint.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Oh lordie, just hack the kernel to make IP_BINDANY usable by any uid, not just root. I was hoping that capabilitiies would actually be useful these days, but apparently not. :( Then you can stop this FD exchange nonsense and this problem should go away= . :) Adrian On 13 November 2012 16:41, Markus Gebert <markus.gebert@hostpoint.ch> wrote= : > > On 13.11.2012, at 19:30, Markus Gebert <markus.gebert@hostpoint.ch> wrote= : > >> To me it looks like the unix socket GC is triggered way too often and/or= running too long, which uses cpu and worse, causes a lot of contention aro= und the unp_list_lock which in turn causes delays for all processes relayin= g on unix sockets for IPC. >> >> I don't know why the unp_gc() is called so often and what's triggering t= his. > > I have a guess now. Dovecot and relayd both use unix sockets heavily. Acc= ording to dtrace uipc_detach() gets called quite often by dovecot closing u= nix sockets. Each time uipc_detach() is called unp_gc_task is taskqueue_enq= ueue()d if fds are inflight. > > in uipc_detach(): > 682 if (local_unp_rights) > 683 taskqueue_enqueue(taskqueue_thread, &unp_gc_task)= ; > > We use relayd in a way that keeps the source address of the client when c= onnecting to the backend server (transparent load balancing). This requires= IP_BINDANY on the socket which cannot be set by unprivileged processes, so= relayd sends the socket fd to the parent process just to set the socket op= tion and send it back. This means an fd gets transferred twice for every ne= w backend connection. > > So we have dovecot calling uipc_detach() often and relayd making it likel= y that fds are inflight (unp_rights > 0). With a certain amount of load thi= s could cause unp_gc_task to be added to the thread taskq too often, slowin= g everything unix socket related down by holding global locks in unp_gc(). > > I don't know if the slowdown can even cause a negative feedback loop at s= ome point by inreasing the chance of fds being inflight. This would explain= why sometimes the condition goes away by itself and sometimes requires int= ervention (taking load away for a moment). > > I'll look into a way to (dis)prove all this tomorrow. Ideas still welcome= :-). > > > Markus > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=36Ob0NSeFVV4goLsaca7Aqc9B0zdPvYWEcNmBPsk40Q>