From owner-freebsd-current@FreeBSD.ORG Fri Nov 21 14:52:54 2008 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92F071065670 for ; Fri, 21 Nov 2008 14:52:54 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from conversation.bsdunix.ch (ns1.bsdunix.ch [82.220.1.90]) by mx1.freebsd.org (Postfix) with ESMTP id 221698FC1A for ; Fri, 21 Nov 2008 14:52:54 +0000 (UTC) (envelope-from freebsdlists@bsdunix.ch) Received: from localhost (localhost [127.0.0.1]) by conversation.bsdunix.ch (Postfix) with ESMTP id ABAE75D7E; Fri, 21 Nov 2008 15:52:52 +0100 (CET) X-Virus-Scanned: by amavisd-new at mail.bsdunix.ch Received: from conversation.bsdunix.ch ([127.0.0.1]) by localhost (conversation.bsdunix.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Ub0eFkHQWCTF; Fri, 21 Nov 2008 15:52:48 +0100 (CET) Received: from bert.mlan.solnet.ch (bert.mlan.solnet.ch [212.101.1.83]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by conversation.bsdunix.ch (Postfix) with ESMTP id F2F735D68; Fri, 21 Nov 2008 15:52:47 +0100 (CET) Message-ID: <4926CB3F.8010500@bsdunix.ch> Date: Fri, 21 Nov 2008 15:52:47 +0100 From: Thomas Vogt User-Agent: Thunderbird 2.0.0.17 (X11/20080929) MIME-Version: 1.0 To: Nikolay Denev References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: current@freebsd.org Subject: Re: deadlock with zfs? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 14:52:54 -0000 Hello > Am 21.11.2008 um 09:39 schrieb Nikolay Denev: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> >> On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote: >> >>> Hello >>> >>> I encounter a deadlock while running a few rsync processes mirroring >>> remote data to my local zfs pool. After a few hours my system is >>> starting more and more vsftpd sessions without closing any inactive >>> ftp sessions. I can't kill any rsync and vsftpd processes with "kill >>> -9". Even shutdown -r now does not work. >>> >>> I got a few hunderts vsftpd processes like this >>> >>> 61346 root 1 57 0 7880K 1692K zfs 1 0:00 0.00% >>> vsftpd >>> 61481 root 1 68 0 7880K 1696K zfs 1 0:00 0.00% >>> vsftpd >>> 61354 root 1 65 0 7880K 1692K zfs 1 0:00 0.00% >>> vsftpd >>> 61480 root 1 68 0 7880K 1696K zfs 0 0:00 0.00% >>> vsftpd >>> 61600 root 1 69 0 7880K 1704K zfs 1 0:00 0.00% >>> vsftpd >>> 61599 root 1 68 0 7880K 1704K zfs 1 0:00 0.00% >>> vsftpd >>> >>> Right now i'm building a debug kernel. Whats the best way to get >>> usefull information from this deadlock? The system itself is not >>> crashing and response well to ssh and i also have a serial console. >>> >>> The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008 >>> (64bit) with zpool version 13. I use zfs only as data pool. The base >>> system is running on ufs2: >>> >>> Filesystem Size Used Avail Capacity Mounted on >>> /dev/da0s1a 496M 220M 236M 48% / >>> devfs 1.0K 1.0K 0B 100% /dev >>> /dev/da0s1g 169G 15G 141G 9% /disk1 >>> /dev/da0s1f 3.9G 17M 3.5G 0% /tmp >>> /dev/da0s1e 29G 3.7G 23G 14% /usr >>> /dev/da0s1d 19G 7.1G 11G 40% /var >>> pool 853G 0B 853G 0% /usr/local/data >>> pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup >>> pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp >>> pool/portsnap 853G 633M 853G 0% /usr/local/data/portsnap >>> pool/www 853G 90M 853G 0% /usr/local/data/www >>> >>> loader.conf: >>> vm.kmem_size="1G" >>> kern.maxfiles="65536" >>> kern.maxproc="20480" >>> net.inet.tcp.tcbhashsize="4096" >>> net.inet.tcp.hostcache.hashsize="1024" >>> vfs.zfs.arc_min="64M" >>> vfs.zfs.arc_max="768M" >>> vfs.zfs.prefetch_disable="1" >>> >>> I also tried to set vfs.zfs.zil_disable=1 but the problem still exist. >>> >>> I know there are few deadlock reports listed in the freebsd wiki. >>> Maybe we can trigger the root cause of the problem and fix it :) >>> >>> Regards >>> Thomas >>> >> >> Hi Thomas, >> >> from my recent experience it seems that tuning vm.kmem_size is no >> longer required, >> maybe you can try without it. > > Sure, i rebooted the system without vm.kmem_size. It takes a a few hours > until i got deadlocks. Removing vm.kmem_size didn't fix my deadlock issue. It just took a bit longer until the deadlock was triggered. Maybe someone can check my deadlock debug output at http://www.bsdunix.ch/deadlock_2008-11-21.txt Regards, Thomas