From owner-freebsd-current@FreeBSD.ORG Wed Apr 7 09:30:59 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 132EF16A4CE; Wed, 7 Apr 2004 09:30:59 -0700 (PDT) Received: from osiris.ipform.ru (osiris.itlegion.ru [212.248.52.90]) by mx1.FreeBSD.org (Postfix) with ESMTP id D1E8E43D2D; Wed, 7 Apr 2004 09:30:57 -0700 (PDT) (envelope-from matrix@itlegion.ru) Received: from artem (artem.office.ipform.ru [192.168.0.12]) by osiris.ipform.ru (8.12.6/8.12.6) with ESMTP id i37GToXg046620; Wed, 7 Apr 2004 20:29:50 +0400 (MSD) (envelope-from matrix@itlegion.ru) X-AntiVirus: Checked by Dr.Web (http://www.drweb.net) Message-ID: <000c01c41cbd$0399e330$0c00a8c0@artem> From: "Artem Koutchine" To: Date: Wed, 7 Apr 2004 20:25:54 +0400 Organization: IT Legion MIME-Version: 1.0 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 cc: freebsd-bugs@freebsd.org cc: freebsd-hardware@freebsd.org Subject: lock order reversal and swap problems X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Apr 2004 16:30:59 -0000 This evening, before going hove i decided to load-test the new server. I remind the server is: dual xeon 2.66, 2GB ECC ram, supermicro x5dpeg2 mb, 3ware 8506-4LP sata radi controller, 4 sata seagate disks, eveyrthing is install in a RAID5 array. I run: 3 bonnies in a loop, 4 byteneches in a loop, 6 my own perl scripts to test the memory. Each allocated about 256MB ran, fill it with somethingm then free and do it over and over again. After 3 minutes of running i saw this: Apr 7 20:16:34 new kernel: lock order reversal Apr 7 20:16:34 new kernel: 1st 0xc79446b4 vm object (vm object) @ /usr/src/sys/vm/swap_pager.c:1313 Apr 7 20:16:35 new kernel: 2nd 0xc08b9e00 swap_pager swhash (swap_pager swhash) @ /usr/src/sys/vm/swap_pager.c:1803 Apr 7 20:16:35 new kernel: 3rd 0xc7898294 vm object (vm object) @ /usr/src/sys/vm/uma_core.c:886 Apr 7 20:16:35 new kernel: Stack backtrace: Apr 7 20:16:35 new kernel: backtrace(0,1,c0888798,c08898f0,c08174bc) at backtrace+0x12 Apr 7 20:16:35 new kernel: witness_checkorder(c7898294,9,c07d43dd,376) at witness_checkorder+0x593 Apr 7 20:16:35 new kernel: _mtx_lock_flags(c7898294,0,c07d43dd,376,c78888d4) at _mtx_lock_flags+0x67 Apr 7 20:16:35 new kernel: obj_alloc(c78888c0,1000,e32ffa2f,101,c78888d4) at obj_alloc+0x31 Apr 7 20:16:35 new kernel: slab_zalloc(c78888c0,1,c78888c0,c78888d4,1) at slab_zalloc+0x82 Apr 7 20:16:35 new kernel: uma_zone_slab(c78888c0,1,c78888d4,0,c07d43dd,68f) at uma_zone_slab+0xb5 Apr 7 20:16:35 new kernel: uma_zalloc_internal(c78888c0,0,1,c78888d4,0) at uma_zalloc_internal+0x28 Apr 7 20:16:35 new kernel: uma_zalloc_arg(c78888c0,0,1) at uma_zalloc_arg+0x2f8 Apr 7 20:16:35 new kernel: swp_pager_meta_build(c79446b4,38,0,2,0) at swp_pager_meta_build+0x108 Apr 7 20:16:36 new kernel: swap_pager_putpages(c79446b4,e32ffc00,4,0,e32ffb70) at swap_pager_putpages+0x2b4 Apr 7 20:16:36 new kernel: default_pager_putpages(c79446b4,e32ffc00,4,0,e32ffb70) at default_pager_putpages+0x18 Apr 7 20:16:36 new kernel: vm_pageout_flush(e32ffc00,4,0,0,e32ffbdc) at vm_pageout_flush+0x11e Apr 7 20:16:36 new kernel: vm_pageout_clean(c2c26308) at vm_pageout_clean+0x2a5 Apr 7 20:16:36 new kernel: vm_pageout_scan(0) at vm_pageout_scan+0x54b Apr 7 20:16:36 new kernel: vm_pageout(0,e32ffd48,0,c071cc3c,0) at vm_pageout+0x2d2 Apr 7 20:16:36 new kernel: fork_exit(c071cc3c,0,e32ffd48) at fork_exit+0xa8 Apr 7 20:16:36 new kernel: fork_trampoline() at fork_trampoline+0x8 Apr 7 20:16:36 new kernel: --- trap 0x1, eip = 0, esp = 0xe32ffd7c, ebp = 0 --- and then hundreds and thousands: Apr 7 20:20:29 new kernel: swap_pager_getswapspace(16): failed Apr 7 20:20:32 new last message repeated 44 times while actually there were 500MB of swap space left. What is going on? Regards, Artem