Date: Sun, 29 Jul 2007 14:07:22 -0400 From: Kris Kennaway <kris@obsecurity.org> To: Jeff Roberson <jroberson@chesapeake.net> Cc: arch@freebsd.org, Alfred Perlstein <alfred@freebsd.org> Subject: Re: Fine grain select locking. Message-ID: <20070729180722.GB85196@rot26.obsecurity.org> In-Reply-To: <20070704114005.X552@10.0.0.1> References: <20070702230728.E552@10.0.0.1> <20070703181242.T552@10.0.0.1> <20070704105525.GU45894@elvis.mu.org> <20070704114005.X552@10.0.0.1>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jul 04, 2007 at 11:47:35AM -0700, Jeff Roberson wrote: > >>http://people.freebsd.org/~jeff/select2.diff > > > >Jeff, I understand you're trying to speed up mysql micro benchmarks, > >but have you done any benchmarking on large select operations? > > I don't know that I'd call mysql a micro-benchmark. This patch also > didn't help there as much as I had hoped and I'm still trying to > understand why. Here is a graph of the performance effects to sysbench with this patch: http://obsecurity.dyndns.org/select.png mysql ===== It appears that at higher loads most of the contention is now in userland, no longer within the kernel. There is also significant contention on the proc lock. Peak load (8 clients): 31 1137 169 1280 0 0 126 56 kern/kern_umtx.c:325 (sleep mutex:umtxql) 7 5688 554 4750 1 0 722 294 kern/subr_sleepqueue.c:388 (sleep mutex:process lock) 3 2335 1155 8732 0 0 669 571 kern/sys_generic.c:955 (sleep mutex:process lock) Higher load (20 clients): 88 6714 807 4763 1 0 754 276 kern/subr_sleepqueue.c:388 (sleep mutex:process lock) 3 2342 1228 7656 0 0 650 550 kern/sys_generic.c:955 (sleep mutex:process lock) 2 431 1299 1023 0 1 53 77 kern/kern_sig.c:996 (sleep mutex:process lock) 7 371 3545 635 0 5 58 131 kern/kern_mutex.c:141 (sleep mutex:umtxql) 70 5085 7433 3184 1 2 507 377 kern/kern_umtx.c:325 (sleep mutex:umtxql) I looked in the past at replacing the proc mutex with a rwlock and looking for places where shared locking could be used, but at least as the code is written currently I dont think any of those apply here. With the select locking patch overall mysql performance does not change much, but the total amount of time spent waiting for locks is greatly reduced (by about an order of magnitude), so system time should be lower with these changes (unless it's counterbalanced by greater time spent doing other things than lock waits). I have not measured this though. We might be able to obtain some further improvement at higher loads by improving the contention behaviour of umtx objects (the kernel part of the libthr pthread mutex). I suspect most of the problem is in mysql itself. What we need is a userland counterpart of lock profiling, for profiling contention on pthread mutexes. pgsql ===== Clear performance benefit from select locking, on the order of 5-10%. Reduction in lock wait time is about *two* orders of magnitude. Peak load: 5 2942 1437 2607 1 0 818 446 kern/subr_turnstile.c:546 (spin mutex:turnstile chain) 13 9250 1474 9572 0 0 1405 585 kern/subr_sleepqueue.c:388 (sleep mutex:process lock) 39 3019 2856 9458 0 0 1613 1131 kern/sys_generic.c:955 (sleep mutex:process lock) 120 5540 5494 16954 0 0 3536 2017 kern/kern_sig.c:996 (sleep mutex:process lock) 20 clients: 8 5336 3506 4338 1 0 1610 910 kern/subr_turnstile.c:546 (spin mutex:turnstile chain) 2 2828 4261 8787 0 0 1749 1298 kern/sys_generic.c:955 (sleep mutex:process lock) 56 10717 4568 8968 1 0 3092 1382 kern/subr_sleepqueue.c:388 (sleep mutex:process lock) 4 5390 7646 15766 0 0 3325 2568 kern/kern_sig.c:996 (sleep mutex:process lock) 79 9423 70619 33525 0 2 154 92 kern/uipc_syscalls.c:135 (sleep mutex:sleep mtxpool) i.e. much the same lock workload as mysql except for no umtx contention (pgsql is not threaded), and huge wait time (but not much contention) on the following: static int getsock(struct filedesc *fdp, int fd, struct file **fpp, u_int *fflagp) { ... if (fdp == NULL) error = EBADF; else { FILEDESC_SLOCK(fdp); fp = fget_locked(fdp, fd); if (fp == NULL) error = EBADF; else if (fp->f_type != DTYPE_SOCKET) { fp = NULL; error = ENOTSOCK; } else { fhold(fp); ... } I think this is mostly because it's called so often, with small incremental but large total cost. Kris
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070729180722.GB85196>