Date: Mon, 21 Oct 2013 15:59:49 +0300 From: Vitalij Satanivskij <satan@ukr.net> To: current@freebsd.org Subject: How to debug whats cause to much __mtx_lock_sleep in system Message-ID: <20131021125949.GB13109@hell.ukr.net>
next in thread | raw e-mail | index | archive | help
Hello. Have 10.0-BETA1 #7 r256765 whith terible load's "load averages: 23.31, 30.53, 31" wich degraded more and more with time. Kernel compilied with dtrace support and using script called hotkernel from DTraceToolkit-0.99 found some stange statistics zfs.ko`lz4_compress 5045 0.2% kernel`0xffffffff80 5185 0.2% kernel`uma_zalloc_arg 5302 0.2% kernel`bcopy 5322 0.2% kernel`_sx_xlock 7310 0.3% kernel`_sx_xunlock 7434 0.3% zfs.ko`l2arc_feed_thread 9797 0.4% zfs.ko`lzjb_compress 9912 0.4% zfs.ko`list_prev 17894 0.7% kernel`__rw_wlock_hard 30522 1.2% kernel`spinlock_exit 31310 1.3% kernel`acpi_cpu_c1 103495 4.1% kernel`_sx_xlock_hard 138743 5.5% kernel`vmem_xalloc 175869 7.0% kernel`cpu_idle 371159 14.8% kernel`__mtx_lock_sleep 1345815 53.8% Theris another same machine with simple data and usage but with old curent r245701 Which have none problem's with load zfs.ko`fletcher_4_native 2366 0.1% kernel`uma_zfree_arg 2387 0.1% zfs.ko`lzjb_decompress 2392 0.1% kernel`__rw_rlock 2477 0.1% zfs.ko`dmu_zfetch 2553 0.1% kernel`bcopy 3035 0.1% kernel`vm_page_splay 3089 0.1% kernel`_mtx_trylock_flags_ 3346 0.2% kernel`bzero 3411 0.2% kernel`0xffffffff80 3665 0.2% kernel`_sx_xunlock 3818 0.2% kernel`uma_zalloc_arg 4216 0.2% kernel`vmtotal 4702 0.2% kernel`_sx_xlock 5117 0.2% kernel`free 5476 0.2% zfs.ko`lzjb_compress 6674 0.3% kernel`spinlock_exit 21590 1.0% kernel`__mtx_lock_sleep 40819 1.9% kernel`acpi_cpu_c1 311077 14.1% kernel`cpu_idle 1639418 74.6% Both servers have same hardware, same software of cause not system version. So which way is the right to investigate problem and find resolution?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131021125949.GB13109>