From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 07:56:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 56E5C10656A7 for ; Sun, 5 Sep 2010 07:56:18 +0000 (UTC) (envelope-from k0802647@telus.net) Received: from defout.telus.net (defout.telus.net [204.209.205.55]) by mx1.freebsd.org (Postfix) with ESMTP id EF93F8FC14 for ; Sun, 5 Sep 2010 07:56:17 +0000 (UTC) Received: from edmwaa01.telusplanet.net ([66.183.53.162]) by priv-edmwes48.telusplanet.net (InterMail vM.7.09.01.00 201-2219-108-20080618) with ESMTP id <20100905075611.FRYX5635.priv-edmwes48.telusplanet.net@edmwaa01.telusplanet.net> for ; Sun, 5 Sep 2010 01:56:11 -0600 Received: from oliver.bc.lan (d66-183-53-162.bchsia.telus.net [66.183.53.162]) by edmwaa01.telusplanet.net (BorderWare Security Platform) with ESMTP id 00153146E7B9C816 for ; Sun, 5 Sep 2010 01:56:11 -0600 (MDT) Received: from [10.111.111.112] (unknown [10.111.111.112]) by oliver.bc.lan (Postfix) with ESMTP id 1A2C262A4; Sun, 5 Sep 2010 00:56:11 -0700 (PDT) Message-ID: <4C834D1A.2010405@telus.net> Date: Sun, 05 Sep 2010 00:56:10 -0700 From: Carl User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.7) Gecko/20100713 Lightning/1.0b2 Thunderbird/3.1.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Cloudmark-Analysis: v=1.1 cv=Kp8XkLsj+ETmpLKACjazEtTHT5jaYByuFOE5D7hc800= c=1 sm=0 a=8nJEP1OIZ-IA:10 a=HNgjH8kF64GtJ7EcXKEMsQ==:17 a=QJfgcumiAAAA:8 a=VZvz2mWK4rtnnvef2ooA:9 a=2uo827Q3rezgzPdSDI-Cuer0srIA:4 a=wPNLvfGTeEIA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 X-Mailman-Approved-At: Sun, 05 Sep 2010 11:47:41 +0000 Subject: geli'd swap and core dumps X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 07:56:18 -0000 I would like to encrypt my swap partition like Pawel does on his company's production servers: http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/ Does Pawel's method result in kernel panics having no functional dump device? What are best practices for achieving encrypted swap and functional core dump recovery? Or are these mutually exclusive goals? Carl / K0802647 From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 13:29:43 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6398A10656C7 for ; Sun, 5 Sep 2010 13:29:43 +0000 (UTC) (envelope-from prvs=18641c54ab=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id F1DF98FC08 for ; Sun, 5 Sep 2010 13:29:42 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sun, 05 Sep 2010 14:19:16 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sun, 05 Sep 2010 14:19:16 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011183987.msg for ; Sun, 05 Sep 2010 14:19:16 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=18641c54ab=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> From: "Steven Hartland" To: "jhell" , "Artem Belevich" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> Date: Sun, 5 Sep 2010 14:19:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 13:29:43 -0000 > ----- Original Message ----- > From: "jhell" >> >> Attached is the needfree patch mentioned in the URL alongside a local >> system patch to adjust kern.maxusers to no more than 512 on systems that >> can support it... No joy, still drops down to arc_min even with those two patches and changing to vm_paging_needed from the post Artem mentioned: http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html So I suspect if I hadn't put in a high arc_min as well it would be back down at silly low levels. top shows:- Mem: 26M Active, 1360M Inact, 2434M Wired, 6188K Cache, 418M Buf, 117M Free Swap: 4096M Total, 15M Used, 4081M Free ARC Size: Current Size: 57.14% 2048.00M (arcsize) Target Size: (Adaptive) 57.14% 2048.00M (c) Min Size (Hard Limit): 57.14% 2048.00M (c_min) Max Size (High Water): ~1:1 3584.00M (c_max) So there's still 1.3G of mem sitting inact, don't know what caused that but its essentially dead memory now it seems. I know I could raise arc_min even further but that could be dangerous? Any other suggestions? It seems like arc needs to cooperate with the rest of the OS pagedaemon more? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 13:58:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2BBB10656D0 for ; Sun, 5 Sep 2010 13:58:49 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 536D08FC16 for ; Sun, 5 Sep 2010 13:58:48 +0000 (UTC) Received: by gxk24 with SMTP id 24so1613400gxk.13 for ; Sun, 05 Sep 2010 06:58:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=5pP7LrgKW074R78jhi5JPxqAzVhz9/ePCCG6oO34WMk=; b=IjVCNyOeL4IoeDF8E47wYz/Y0ZMJZm/Y3KBig8lAtKkfnN0MeTYzO2ydTGCV3LQKA6 t8Il0H6uAdbLcDeFWEwmgX/JhJ32Ih7bLloDZ7azy/8CWi6f3Mdtnhlaa3Yiw0LNgpta Kia0oF3/tMmQwFKTrphMewc56CV9nCrcADw7s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=ttxrrjOLA5x9ZBIzVmrgfEuSn9gsAxJrRDQuTjiGIiIAwsn/ZpfvMJoyPRiHmMtqtK hrCubCAOAsufJer24X3X5CJJkUnGfs4OgKMdXdVz70aVPXs2+56l5fjNx1WZpCKiq9PS VkfYVAjNJyJS4InC6p3lEHU5vcsfE88wISwTI= Received: by 10.151.9.4 with SMTP id m4mr726267ybi.22.1283695128491; Sun, 05 Sep 2010 06:58:48 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id q1sm4743115ybk.20.2010.09.05.06.58.46 (version=SSLv3 cipher=RC4-MD5); Sun, 05 Sep 2010 06:58:47 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C83A214.1080204@DataIX.net> Date: Sun, 05 Sep 2010 09:58:44 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> In-Reply-To: <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 13:58:49 -0000 On 09/05/2010 09:19, Steven Hartland wrote: >> ----- Original Message ----- From: "jhell" >>> >>> Attached is the needfree patch mentioned in the URL alongside a local >>> system patch to adjust kern.maxusers to no more than 512 on systems that >>> can support it... > > No joy, still drops down to arc_min even with those two patches and > changing > to vm_paging_needed from the post Artem mentioned: > http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html > > So I suspect if I hadn't put in a high arc_min as well it would be back > down > at silly low levels. > > top shows:- > > Mem: 26M Active, 1360M Inact, 2434M Wired, 6188K Cache, 418M Buf, 117M Free > Swap: 4096M Total, 15M Used, 4081M Free > > ARC Size: > Current Size: 57.14% 2048.00M (arcsize) > Target Size: (Adaptive) 57.14% 2048.00M (c) > Min Size (Hard Limit): 57.14% 2048.00M (c_min) > Max Size (High Water): ~1:1 3584.00M (c_max) > > So there's still 1.3G of mem sitting inact, don't know what caused that but > its essentially dead memory now it seems. > > I know I could raise arc_min even further but that could be dangerous? > > Any other suggestions? It seems like arc needs to cooperate with the > rest of the OS pagedaemon more? I am not exactly sure what your source is right now but guessing its at least 8.1-RELEASE & hoping its 8.1-STABLE. Line 3656 & 3657 of arc.c ? what are they ? Do they look like the following, 3656: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count 3657: + cnt.v_cache_count); Regards, -- jhell,v From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 14:27:20 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A864A10656C6 for ; Sun, 5 Sep 2010 14:27:20 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 590788FC14 for ; Sun, 5 Sep 2010 14:27:20 +0000 (UTC) Received: by gyg4 with SMTP id 4so1612861gyg.13 for ; Sun, 05 Sep 2010 07:27:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type; bh=8ZYY9+CfmfUFJJfqRtTNbbbapTmb806QjmTJqOXefks=; b=Z1LO6vAyVHN11e/yIdErUcNiLRy4PNPDlYechtWByTV5k4fpvNYaDshIMuHxEMJ8WZ G4g95eqotXXkBQKFI47ZkzPG/QAdhwPeLDTMcTnhiGxedd1nGT4PCDyXD3gD3wnI0V9n TRuDrjhjaFmIzg8YQ61/KYgsGvK8r6Ky6do6I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type; b=pF6+BlnCO8ybsnG3RPDtBMXjNyOd6wXu6JV3DcUWCg3bZ0SuzIJaQvN086PacU5w5X tPHdMRIq4rPjm3YC5CF7JkTuK/D6nBf/v1GJbtDigSjrsP8prGzeZEP+KKEFQNirbRZ8 D4uysBdaDYp3R2ToLmgXUWh2GHQFKc8Kpj+qI= Received: by 10.151.149.10 with SMTP id b10mr477045ybo.226.1283696839423; Sun, 05 Sep 2010 07:27:19 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id q21sm2367101ybk.15.2010.09.05.07.27.17 (version=SSLv3 cipher=RC4-MD5); Sun, 05 Sep 2010 07:27:18 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C83A8C4.3040805@DataIX.net> Date: Sun, 05 Sep 2010 10:27:16 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Carl References: <4C834D1A.2010405@telus.net> In-Reply-To: <4C834D1A.2010405@telus.net> X-Enigmail-Version: 1.1.2 Content-Type: multipart/mixed; boundary="------------010407090100070106080600" Cc: freebsd-fs@freebsd.org Subject: Re: geli'd swap and core dumps X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 14:27:20 -0000 This is a multi-part message in MIME format. --------------010407090100070106080600 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/05/2010 03:56, Carl wrote: > I would like to encrypt my swap partition like Pawel does on his > company's production servers: > > > http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/ > > > Does Pawel's method result in kernel panics having no functional dump > device? > > What are best practices for achieving encrypted swap and functional core > dump recovery? Or are these mutually exclusive goals? > Encrypt your swap let it do its job as a swap device and if you are planning to gather core dumps you always have the choice of a dedicated encrypted dumpdev that is not ``onetime'' encrypted. Personally I use USB dedicated hardware for dumps thumb-drives or whatever is available at the time and allows me to grab the core from it after -> analyze it... if its interesting I can pull the dumpdev, label it and keep it till a time that I deem it recyclable. Dumps that are being done now are minidump's unless you have chosen to explicitly do a full dump ``debug.minidump'' so the size of your dumpdev should be approximate to the size of kmem at the time the dump occurs. Most times a 1GB flash card or thumb-drive suits this purpose allowing for a safe amount of room to grow. Attached is a script that was obtained from one of the ZFS wiki's out there that calculates and prints out some things about KMEM. PS: There has also been some work around getting network dumps working so you may just want to look into that. As for how far along ? I am unsure. Would be nice to have a central crash dump repository though. Regards, - -- jhell,v -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJMg6jEAAoJEJBXh4mJ2FR+/ZoIAIBbdYIuPVv29fCpfykEXVjO ghg4Il6w6FKCPCU0Mh78lWLE3r3vfLdRk8RgYl/CGC3QxkIcc/PBr7Ohi+3bjKz6 Rmb1cTg9vhamm4JqxlNQ/KAxnjdqRzCdhHNIHVKHDrTFhlUrZh9q9KzSwwndyPJl NtqCsuz4Fpvyhrx6ErzzdBEtJCxhyDqlLxCwGgHRrGmFIcfZjMYgMqfyuc0oIbJn N/XU0KH6ZPAVmT9zIgb62QZ+zMU4VerwLmSgcTA2k6FVlum4MED1ACGgzs8XuDQ/ m7B7Gl8iZPNXjf8DxZHKBmqBR49QABkOmRQzRS8s9wYUoRJBjYuA97f55i/6Pp4= =EAdn -----END PGP SIGNATURE----- --------------010407090100070106080600 Content-Type: text/plain; name="kmem.sh" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="kmem.sh" #!/bin/sh - TEXT=`kldstat | awk 'BEGIN {print "16i 0";} NR>1 {print toupper($4) "+"} END {print "p"}' | dc` DATA=`vmstat -m | sed -Ee '1s/.*/0/;s/.* ([0-9]+)K.*/\1+/;$s/$/1024*p/' | dc` TOTAL=$((DATA + TEXT)) echo TEXT=$TEXT, `echo $TEXT | awk '{print $1/1048576 " MB"}'` echo DATA=$DATA, `echo $DATA | awk '{print $1/1048576 " MB"}'` echo TOTAL=$TOTAL, `echo $TOTAL | awk '{print $1/1048576 " MB"}'` --------------010407090100070106080600 Content-Type: application/octet-stream; name="kmem.sh.sig" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="kmem.sh.sig" iQEcBAABAgAGBQJMg6jEAAoJEJBXh4mJ2FR+XnoH/iK1U3AXt6mDpSzJQJgburMIB57VnEE9 U3qsaG2MlVpHKiaeSkbUQZ7F9ZdSK1IPHfSwS15s6YZYSN406JGoOFc51jlvM1SUWsuIrwze jVuPapCJ/RtCY0sKF13FbfXy4zml5orF/JvtG8Ii91iqxKSLhIAIo/NmJsT9YC8nYq1B2Sgs QtaFBFar622k1eVpLLnTGcpcdRqEUnRxXJIsRZX915vv7ttzc6QO/BbbKdRjOI7H6itb8L3H lEeaEfLIdtSjQnjf1xw2BOzFTL3tMr7IRqAyjXKSIkH+Py0t3Y1we98lalkzflsZVLnYXXIj kVVtXACf2EoHT4TqbbeOrU8= --------------010407090100070106080600-- From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 15:04:06 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8FE8E10656C3 for ; Sun, 5 Sep 2010 15:04:06 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 39E168FC08 for ; Sun, 5 Sep 2010 15:04:05 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id A3A1C45C9F; Sun, 5 Sep 2010 17:04:04 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 71BAC45685; Sun, 5 Sep 2010 17:03:58 +0200 (CEST) Date: Sun, 5 Sep 2010 17:03:44 +0200 From: Pawel Jakub Dawidek To: Carl Message-ID: <20100905150344.GD1900@garage.freebsd.pl> References: <4C834D1A.2010405@telus.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AsxXAMtlQ5JHofzM" Content-Disposition: inline In-Reply-To: <4C834D1A.2010405@telus.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: geli'd swap and core dumps X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 15:04:06 -0000 --AsxXAMtlQ5JHofzM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Sep 05, 2010 at 12:56:10AM -0700, Carl wrote: > I would like to encrypt my swap partition like Pawel does on his=20 > company's production servers: >=20 >=20 > http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-co= nfiguration/ >=20 > Does Pawel's method result in kernel panics having no functional dump=20 > device? >=20 > What are best practices for achieving encrypted swap and functional core= =20 > dump recovery? Or are these mutually exclusive goals? Well, the idea to encrypt swap is to prevent any sensitive data to be stored on disk unencrypted where it might last for a long time. If you configure to dump kernel memory to a disk (kernel dumps are not encrypted) you kinda miss the point, as kernel memory can contain a lot of sensitive data. It would be best in such cases to use textdump(4) as there is quite a lot of useful info, but no sensitive data at all (or not much at least, depends on what you consider sensitive). Unfortunately rc.d scripts order doesn't allow geli-encrypted swap and dumpdev to co-operate, ie. geli is configured on swap device before savecore(8) has a chance to read underlying device (to be more precise not only read it, but also write to it, as it has to mark the data as already obtained, so we don't save the same crash info on every reboot). On the other hand we do want swap as soon as possible (even before mounting / or /var read-write so we have more memory for fsck(8) for example), so the solution is not as trival as changing scripts order. The better fix would be to obtain and save crash info in memory somewhere before we configure swap and copy it to /var/crash/ once we mount it. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --AsxXAMtlQ5JHofzM Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyDsVAACgkQForvXbEpPzQ/DACcDWH6RyT0T9urUBWXnF/hr5Dk F/4AoJibwtRtqm5KEtbBQ55Bdqgpiv9g =h9sb -----END PGP SIGNATURE----- --AsxXAMtlQ5JHofzM-- From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 17:49:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2C0610656A6 for ; Sun, 5 Sep 2010 17:49:21 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 8A78D8FC0A for ; Sun, 5 Sep 2010 17:49:21 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 9E64745CA0; Sun, 5 Sep 2010 19:49:19 +0200 (CEST) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 8AD6345684; Sun, 5 Sep 2010 19:49:14 +0200 (CEST) Date: Sun, 5 Sep 2010 19:49:01 +0200 From: Pawel Jakub Dawidek To: Julian Elischer Message-ID: <20100905174901.GE1900@garage.freebsd.pl> References: <4C812CEC.8080309@elischer.org> <4C814B9A.5030109@elischer.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="DEueqSqTbz/jWVG1" Content-Disposition: inline In-Reply-To: <4C814B9A.5030109@elischer.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 17:49:22 -0000 --DEueqSqTbz/jWVG1 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 03, 2010 at 12:25:14PM -0700, Julian Elischer wrote: > >I just grabbed it and slotted it into my VirtualBox test network. I > >need a little help getting nfs going, though. I'm getting the > >following error: > > > >KLD nfslockd.ko: depends on nfs - not available or version mismatch > >KLD nfsd.ko: depends on nfslockd - not available or version mismatch > >mountd: NFS server is not available > > > >Do I need to grab the source and build a new kernel with NFS enabled? >=20 > you should be able to find a kernel module to do it.. > maybe pjd has it lying around in his build tree, or maybe it's in=20 > /boot/kernel on the image? All needed kernel modules are there, I just tested it. I wanted to turn off NFS server by default in the image itself, but forgot... Doing the following should be enough: # touch /etc/exports # echo 'nfs_server_enable=3D"YES"' >> /etc/rc.conf # echo 'mountd_enable=3D"YES"' >> /etc/rc.conf # echo 'rpcbind_enable=3D"YES"' >> /etc/rc.conf # /etc/rc.d/rpcbind start # /etc/rc.d/nfsd start # /etc/rc.d/mountd start # zfs set sharenfs=3Don tank --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --DEueqSqTbz/jWVG1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyD2A0ACgkQForvXbEpPzS4uACfUjf0Ba96u5ZyWAqTOq/zaJFu f9UAoJFSp3t9QFTQb+9VgqEu7YcGhNfE =gC5o -----END PGP SIGNATURE----- --DEueqSqTbz/jWVG1-- From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 20:13:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 586661065702 for ; Sun, 5 Sep 2010 20:13:21 +0000 (UTC) (envelope-from prvs=18641c54ab=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E2D1B8FC1B for ; Sun, 5 Sep 2010 20:13:20 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sun, 05 Sep 2010 21:13:16 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sun, 05 Sep 2010 21:13:16 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011184726.msg for ; Sun, 05 Sep 2010 21:13:15 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=18641c54ab=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> From: "Steven Hartland" To: "jhell" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> Date: Sun, 5 Sep 2010 21:13:16 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 20:13:21 -0000 ----- Original Message ----- From: "jhell" > I am not exactly sure what your source is right now but guessing its at > least 8.1-RELEASE & hoping its 8.1-STABLE. > > Line 3656 & 3657 of arc.c ? what are they ? > > Do they look like the following, > 3656: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count > 3657: + cnt.v_cache_count); Sorry yes running 8.1-RELEASE with the patches mentioned atm: 3656 of arc.c looks like: if (arc_size > arc_c_min) { uint64_t evictable_memory = arc_mru->arcs_lsize[ARC_BUFC_DATA] + arc_mru->arcs_lsize[ARC_BUFC_METADATA] + arc_mfu->arcs_lsize[ARC_BUFC_DATA] + arc_mfu->arcs_lsize[ARC_BUFC_METADATA]; available_memory += MIN(evictable_memory, arc_size - arc_c_min); } earlier at 3614 I have what I think your after which is: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count); Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 20:31:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36B9710656B1 for ; Sun, 5 Sep 2010 20:31:22 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id 9BC548FC0A for ; Sun, 5 Sep 2010 20:31:21 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta02.westchester.pa.mail.comcast.net with comcast id 38FN1f0030bG4ec528XMRC; Sun, 05 Sep 2010 20:31:21 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta03.westchester.pa.mail.comcast.net with comcast id 38XL1f00E3LrwQ23P8XMWu; Sun, 05 Sep 2010 20:31:21 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 291D19B425; Sun, 5 Sep 2010 13:31:19 -0700 (PDT) Date: Sun, 5 Sep 2010 13:31:19 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20100905203119.GA55546@icarus.home.lan> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 20:31:22 -0000 On Sun, Sep 05, 2010 at 09:13:16PM +0100, Steven Hartland wrote: > ----- Original Message ----- From: "jhell" > > >I am not exactly sure what your source is right now but guessing its at > >least 8.1-RELEASE & hoping its 8.1-STABLE. > > > >Line 3656 & 3657 of arc.c ? what are they ? > > > >Do they look like the following, > >3656: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count > >3657: + cnt.v_cache_count); > > Sorry yes running 8.1-RELEASE with the patches mentioned atm: > > 3656 of arc.c looks like: > > if (arc_size > arc_c_min) { > uint64_t evictable_memory = > arc_mru->arcs_lsize[ARC_BUFC_DATA] + > arc_mru->arcs_lsize[ARC_BUFC_METADATA] + > arc_mfu->arcs_lsize[ARC_BUFC_DATA] + > arc_mfu->arcs_lsize[ARC_BUFC_METADATA]; > available_memory += MIN(evictable_memory, arc_size - arc_c_min); > } > > earlier at 3614 I have what I think your after which is: > uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count); No, that line of code is different than the one jhell provided. Additionally, there are other changes in the same commit which are being overlooked. There have been changes between 8.1-RELEASE and RELENG_8 (8.0-STABLE), some affecting ARC behaviour. Rather than go through "patch hell", staring at hundreds of lines of code and trying to work out what got changed where or hacking up a solution, I'd recommend you simply move to RELENG_8 (8.1-STABLE) and see if the behaviour you describe goes away. If it doesn't improve, patches might be written that address the problem, and those patches will only be for RELENG_8, because that's where all the attention is being applied to. RELENG_8_x (8.x-RELEASE) branches do not get this attention (and this is intentional). There is no official resource/announcement regarding when changes in the FreeBSD ZFS code changes, regardless of branch. Users are apparently "expected" to follow commits/changes as they come in. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 20:33:35 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 963C91065674 for ; Sun, 5 Sep 2010 20:33:35 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 77F4A8FC14 for ; Sun, 5 Sep 2010 20:33:35 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta01.emeryville.ca.mail.comcast.net with comcast id 38Wi1f0060QkzPwA18ZawH; Sun, 05 Sep 2010 20:33:34 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta02.emeryville.ca.mail.comcast.net with comcast id 38ZZ1f0053LrwQ28N8ZZxi; Sun, 05 Sep 2010 20:33:34 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 70BC09B425; Sun, 5 Sep 2010 13:33:33 -0700 (PDT) Date: Sun, 5 Sep 2010 13:33:33 -0700 From: Jeremy Chadwick To: Steven Hartland Message-ID: <20100905203333.GA55760@icarus.home.lan> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <20100905203119.GA55546@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100905203119.GA55546@icarus.home.lan> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 20:33:35 -0000 On Sun, Sep 05, 2010 at 01:31:19PM -0700, Jeremy Chadwick wrote: > There have been changes between 8.1-RELEASE and RELENG_8 (8.0-STABLE), Latter part should have read "... RELENG_8 (8.1-STABLE) ...". -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 23:34:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BD7010656C1 for ; Sun, 5 Sep 2010 23:34:34 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id EE6458FC16 for ; Sun, 5 Sep 2010 23:34:33 +0000 (UTC) Received: by ywt2 with SMTP id 2so1693058ywt.13 for ; Sun, 05 Sep 2010 16:34:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=ZYzZYqZ2ggyVjSEW99Uj3uaNOyCr+RU0NWq1zUBcpWo=; b=kn3YthQQ/L35o1zkTMhC7JFFFUHlFokT2ZLtTB4w1IxBhOQ+UBwjB3gxdeKdXSVVjd zYmlWfDKeBJNvL/9KhOr98k4j6XRpzvACHHkPK/oOyglyjEDV4VcRtjFXk4XiWqcBRae gyMRk1MuzFDjJ3mHv8OOnIRR+ACx4wTW/1Zjc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=drzYLeU45eiDUF7Oss5eAT3gbX4GyQOnA55FiNuDPlypXpqG5+Mf8+JfY6rzeQcRzW 2p6URlxrIkWdFSQ0N7uansgoQQipeA1bhumEO4rnaVM8ADmOnPWRQtYqh5KtK6mjeq9T s93YikWRUh2MPgpdT0y58ItL/4KXPU08CRcWk= Received: by 10.151.79.11 with SMTP id g11mr450112ybl.92.1283729672980; Sun, 05 Sep 2010 16:34:32 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id q21sm5213479ybk.23.2010.09.05.16.34.31 (version=SSLv3 cipher=RC4-MD5); Sun, 05 Sep 2010 16:34:32 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C842905.2080602@DataIX.net> Date: Sun, 05 Sep 2010 19:34:29 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> In-Reply-To: <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 23:34:34 -0000 On 09/05/2010 16:13, Steven Hartland wrote: >> 3656: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count >> 3657: + cnt.v_cache_count); > earlier at 3614 I have what I think your after which is: > uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count); Alright change this to the above, recompile and re-run your tests. Effectively before this change that apparently still needs to be MFC'd or MFS'd would not allow ZFS to look at or use cnt.v_cache_count. Pretty much to sum it up "available mem = cache + free" This possibly could cause what your seeing but there might be other changes still yet TBD. Ill look into what else has changed from RELEASE -> STABLE. Also do you check out your sources with svn(1) or csup(1) ? Regards, -- jhell,v From owner-freebsd-fs@FreeBSD.ORG Sun Sep 5 23:57:51 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 670431065697 for ; Sun, 5 Sep 2010 23:57:51 +0000 (UTC) (envelope-from prvs=18641c54ab=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id EBD9B8FC17 for ; Sun, 5 Sep 2010 23:57:50 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 00:57:45 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 00:57:45 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011185136.msg for ; Mon, 06 Sep 2010 00:57:45 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=18641c54ab=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> From: "Steven Hartland" To: "jhell" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> Date: Mon, 6 Sep 2010 00:57:46 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Sep 2010 23:57:51 -0000 ----- Original Message ----- From: "jhell" > On 09/05/2010 16:13, Steven Hartland wrote: >>> 3656: uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count >>> 3657: + cnt.v_cache_count); > >> earlier at 3614 I have what I think your after which is: >> uint64_t available_memory = ptoa((uintmax_t)cnt.v_free_count); > > Alright change this to the above, recompile and re-run your tests. > Effectively before this change that apparently still needs to be MFC'd > or MFS'd would not allow ZFS to look at or use cnt.v_cache_count. Pretty > much to sum it up "available mem = cache + free" > > This possibly could cause what your seeing but there might be other > changes still yet TBD. Ill look into what else has changed from RELEASE > -> STABLE. > > Also do you check out your sources with svn(1) or csup(1) ? Based on Jeremy's comments I'm updating the box the stable. Its building now but will be the morning before I can reboot to activate changes as I need to deactivate the stream instance and wait for all active connections to finish. That said the problem doesn't seem to be cache + free but more cache + free + inactive with inactive being the large chunk, so not sure this change would make any difference? How does ufs deal with this, does it take inactive into account? Seems a bit silly for inactive pages to prevent reuse for extended periods when the memory could be better used as cache. As an experiment I compiled a little app which malloced a large block of memory, 1.3G in this case and then freed it. This does indeed pull the memory out of inactive and back into the free pool where zfs is which happy to re-expand arc and once again cache large files. Seems a bit extreme to have to do this though. Will see what happens with stable tomorrow though :) Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 00:21:53 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74A9910656B5 for ; Mon, 6 Sep 2010 00:21:53 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2F7C68FC13 for ; Mon, 6 Sep 2010 00:21:52 +0000 (UTC) Received: by iwn34 with SMTP id 34so4234517iwn.13 for ; Sun, 05 Sep 2010 17:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=GAVA+MWMvRAAmmqcNSbOgVRLME1KPXzdFiAfWbyGmpo=; b=A7B5oBl2/JOF33He4x24XDEhtUfPrNZeEhsEbK2rIEgbWcGZaLG8ak3sG4Y0nfGx4w p7MfNGQGo99x+rClUMV2Zrk7gWJTF0so3wUdwHtstOaf3q49uJM4EK+lwJLT9KRYGIo/ 7VK/7ruQbyoMsHkrAXgegZdBFooPjvzKZFdL0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=ukznXNUcaFECMtJyPsNoiPZkVqj6+wr14k5KqLN0uQ70xbOLVh7FC2NSc7WratsGKt 0neLro0JklbCBzGU0CsfHbMqkq5XkXuBXwettQI5USCvEOwCxot0cA5khCXqGSYVlB+b 4zPkuVKl7ZgDrDGX4kjngSz2ySNOHjvjcB09o= Received: by 10.231.183.10 with SMTP id ce10mr5291121ibb.96.1283732512392; Sun, 05 Sep 2010 17:21:52 -0700 (PDT) Received: from centel.dataix.local ([99.181.137.20]) by mx.google.com with ESMTPS id i6sm2332413iba.8.2010.09.05.17.21.50 (version=SSLv3 cipher=RC4-MD5); Sun, 05 Sep 2010 17:21:51 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C84341D.8060708@DataIX.net> Date: Sun, 05 Sep 2010 20:21:49 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> In-Reply-To: <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 00:21:53 -0000 On 09/05/2010 19:57, Steven Hartland wrote: > >> On 09/05/2010 16:13, Steven Hartland wrote: >>>> 3656: uint64_t available_memory = >>>> ptoa((uintmax_t)cnt.v_free_count 3657: + >>>> cnt.v_cache_count); >> >>> earlier at 3614 I have what I think your after which is: uint64_t >>> available_memory = ptoa((uintmax_t)cnt.v_free_count); >> >> Alright change this to the above, recompile and re-run your tests. >> Effectively before this change that apparently still needs to be >> MFC'd or MFS'd would not allow ZFS to look at or use >> cnt.v_cache_count. Pretty much to sum it up "available mem = cache >> + free" >> >> This possibly could cause what your seeing but there might be >> other changes still yet TBD. Ill look into what else has changed >> from RELEASE -> STABLE. >> >> Also do you check out your sources with svn(1) or csup(1) ? > > Based on Jeremy's comments I'm updating the box the stable. Its > building now but will be the morning before I can reboot to activate > changes as I need to deactivate the stream instance and wait for all > active connections to finish. > > That said the problem doesn't seem to be cache + free but more cache > + free + inactive with inactive being the large chunk, so not sure > this change would make any difference? > If I remember correctly I thought that was already calculated into the mix but I could be wrong. I remember a discussion about it before that free was inactive + free, and for ARC the cache was never being accounted for so not enough paging was happening which would result in a situation like the one you have now. MAYBE! > How does ufs deal with this, does it take inactive into account? > Seems a bit silly for inactive pages to prevent reuse for extended > periods when the memory could be better used as cache. > I agree commented above. > As an experiment I compiled a little app which malloced a large block > of memory, 1.3G in this case and then freed it. This does indeed pull > the memory out of inactive and back into the free pool where zfs is > which happy to re-expand arc and once again cache large files. Seems > a bit extreme to have to do this though. Maybe we should add that code to zfs(1) and call it with gimme-my-mem-back 1 for all of it 2 for half of it and 3 for panic ;) > > Will see what happens with stable tomorrow though :) > Good luck Steve, Look forward to hearing the result. If you are happy with the result you get from stable/8 I would reccommend patching to v15 which is much more stable than the v14 code. The specific patches you would want are: (in order) http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch and then the needfree.patch I already posted. The maxusers.patch being optional. -- jhell,v From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 05:33:54 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EFF210656CC for ; Mon, 6 Sep 2010 05:33:54 +0000 (UTC) (envelope-from mgamsjager@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 49FF58FC08 for ; Mon, 6 Sep 2010 05:33:53 +0000 (UTC) Received: by qwg5 with SMTP id 5so3882678qwg.13 for ; Sun, 05 Sep 2010 22:33:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=SQqFCI9MbP1BYmFT86gNaW2u4MMR9SNOZePhXiKf+Bw=; b=Yw0/sKXE/dc4yWSR8ncaB8kHlFSW0V8ODIo8lEd+/9jNICqlKe0OEkCsWpORJFVE0M B6eTm8JGI1/nicxkooczbIEQY6mGvOErVWOOZKyCoyAfRJbFX1W89uT/iAT702an7pP9 bpBDOBhEH7QB57YioFh3cyly3YK30RyMfm/t0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=Ll3rNIZK269pCZsQYPb/2MspERSUkWiqTMtZpu6DzUuF2Cl5BWf0PE2Z8l5BnTI/QS sc3+ipzEcrg7+bZbTggYAvtBSrrqDKsAWjkKoQwRELGORZ8B+agwX86by/ff983R2+MK b72DSOPnpEqf+1doz25VBpWthkJOISZWGE5UY= Received: by 10.229.251.197 with SMTP id mt5mr3060522qcb.131.1283751233190; Sun, 05 Sep 2010 22:33:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.241.213 with HTTP; Sun, 5 Sep 2010 22:33:23 -0700 (PDT) In-Reply-To: <4C84341D.8060708@DataIX.net> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84341D.8060708@DataIX.net> From: Matthias Gamsjager Date: Mon, 6 Sep 2010 07:33:23 +0200 Message-ID: To: jhell Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 05:33:54 -0000 All this patches should be posted on the zfs wiki. Make it much easier for ppl to find it rather then digging through all the mailing lists. On Mon, Sep 6, 2010 at 2:21 AM, jhell wrote: > On 09/05/2010 19:57, Steven Hartland wrote: >> >>> On 09/05/2010 16:13, Steven Hartland wrote: >>>>> 3656: =A0uint64_t available_memory =3D >>>>> ptoa((uintmax_t)cnt.v_free_count 3657: =A0 =A0 =A0+ >>>>> cnt.v_cache_count); >>> >>>> earlier at 3614 I have what I think your after which is: uint64_t >>>> available_memory =3D ptoa((uintmax_t)cnt.v_free_count); >>> >>> Alright change this to the above, recompile and re-run your tests. >>> Effectively before this change that apparently still needs to be >>> MFC'd or MFS'd would not allow ZFS to look at or use >>> cnt.v_cache_count. Pretty much to sum it up "available mem =3D cache >>> + free" >>> >>> This possibly could cause what your seeing but there might be >>> other changes still yet TBD. Ill look into what else has changed >>> from RELEASE -> STABLE. >>> >>> Also do you check out your sources with svn(1) or csup(1) ? >> >> Based on Jeremy's comments I'm updating the box the stable. Its >> building now but will be the morning before I can reboot to activate >> changes as I need to deactivate the stream instance and wait for all >> active connections to finish. >> >> That said the problem doesn't seem to be cache + free but more cache >> + free + inactive with inactive being the large chunk, so not sure >> this change would make any difference? >> > > If I remember correctly I thought that was already calculated into the > mix but I could be wrong. I remember a discussion about it before that > free was inactive + free, and for ARC the cache was never being > accounted for so not enough paging was happening which would result in a > situation like the one you have now. MAYBE! > >> How does ufs deal with this, does it take inactive into account? >> Seems a bit silly for inactive pages to prevent reuse for extended >> periods when the memory could be better used as cache. >> > > I agree commented above. > >> As an experiment I compiled a little app which malloced a large block >> of memory, 1.3G in this case and then freed it. This does indeed pull >> the memory out of inactive and back into the free pool where zfs is >> which happy to re-expand arc and once again cache large files. Seems >> a bit extreme to have to do this though. > > Maybe we should add that code to zfs(1) and call it with > gimme-my-mem-back 1 for all of it 2 for half of it and 3 for panic ;) > >> >> Will see what happens with stable tomorrow though :) >> > > Good luck Steve, Look forward to hearing the result. If you are happy > with the result you get from stable/8 I would reccommend patching to v15 > which is much more stable than the v14 code. > > The specific patches you would want are: (in order) > http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch > and then the needfree.patch I already posted. > > The maxusers.patch being optional. > > > -- > > =A0jhell,v > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 07:08:07 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 283171065695; Mon, 6 Sep 2010 07:08:07 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F23928FC15; Mon, 6 Sep 2010 07:08:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o86786SZ033414; Mon, 6 Sep 2010 07:08:06 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o867864G033410; Mon, 6 Sep 2010 07:08:06 GMT (envelope-from linimon) Date: Mon, 6 Sep 2010 07:08:06 GMT Message-Id: <201009060708.o867864G033410@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 07:08:07 -0000 Synopsis: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Mon Sep 6 07:07:55 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=150143 From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 10:49:27 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8FB810656E7 for ; Mon, 6 Sep 2010 10:49:27 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 2BE558FC13 for ; Mon, 6 Sep 2010 10:49:26 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA00362; Mon, 06 Sep 2010 13:49:15 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4C84C72A.3020506@icyb.net.ua> Date: Mon, 06 Sep 2010 13:49:14 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100823 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> In-Reply-To: <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 10:49:27 -0000 on 05/09/2010 16:19 Steven Hartland said the following: >> ----- Original Message ----- From: "jhell" >>> >>> Attached is the needfree patch mentioned in the URL alongside a local >>> system patch to adjust kern.maxusers to no more than 512 on systems that >>> can support it... > > No joy, still drops down to arc_min even with those two patches and changing > to vm_paging_needed from the post Artem mentioned: > http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html > > So I suspect if I hadn't put in a high arc_min as well it would be back down > at silly low levels. But we don't really know this, do we? I think that it would be useful for you and perhaps for us, if you'd set up monitoring (and graphing) of key memory-related parameters. E.g. at least the following sysctls: kstat.zfs.misc.arcstats.size vm.stats.vm.v_pdwakeups vm.stats.vm.v_cache_count vm.stats.vm.v_inactive_count vm.stats.vm.v_active_count vm.stats.vm.v_wire_count vm.stats.vm.v_free_count This would allow to see dynamics of memory consumption and correlation with pagedaemon events. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 10:54:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B28B10656AE for ; Mon, 6 Sep 2010 10:54:21 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 57A8D8FC16 for ; Mon, 6 Sep 2010 10:54:19 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA00410; Mon, 06 Sep 2010 13:54:16 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4C84C857.1070306@icyb.net.ua> Date: Mon, 06 Sep 2010 13:54:15 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100823 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> In-Reply-To: <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 10:54:22 -0000 on 06/09/2010 02:57 Steven Hartland said the following: > Based on Jeremy's comments I'm updating the box the stable. Its building now > but will be the morning before I can reboot to activate changes as I need to > deactivate the stream instance and wait for all active connections to finish. > > That said the problem doesn't seem to be cache + free but more cache + free > + inactive with inactive being the large chunk, so not sure this change > would make any difference? > > How does ufs deal with this, does it take inactive into account? Seems a bit > silly for inactive pages to prevent reuse for extended periods when the > memory could be better used as cache. Inactive pages are also a cache, just a different kind. > As an experiment I compiled a little app which malloced a large block of > memory, 1.3G in this case and then freed it. This does indeed pull the memory > out of inactive and back into the free pool where zfs is which happy to > re-expand arc and once again cache large files. Seems a bit extreme to have to > do this though. > > Will see what happens with stable tomorrow though :) Don't forget the change that I suggested (from Artem's link). You may want to read the whole post too, I tried to explain what's going on with inactive and what the change tries to accomplish. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 10:55:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D30810656BD for ; Mon, 6 Sep 2010 10:55:48 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B70D98FC18 for ; Mon, 6 Sep 2010 10:55:47 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 11:45:23 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 11:45:23 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011185977.msg for ; Mon, 06 Sep 2010 11:45:22 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "jhell" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84341D.8060708@DataIX.net> Date: Mon, 6 Sep 2010 11:45:21 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 10:55:48 -0000 ----- Original Message ----- From: "jhell" >> Will see what happens with stable tomorrow though :) > > Good luck Steve, Look forward to hearing the result. If you are happy > with the result you get from stable/8 I would reccommend patching to v15 > which is much more stable than the v14 code. No joy I'm afraid still the same:- ARC Size: Current Size: 57.14% 2047.91M (arcsize) Target Size: (Adaptive) 57.14% 2048.00M (c) Min Size (Hard Limit): 57.14% 2048.00M (c_min) Max Size (High Water): ~1:1 3584.00M (c_max) Mem: 104M Active, 1185M Inact, 2452M Wired, 268K Cache, 418M Buf, 202M Free Swap: 4096M Total, 1112K Used, 4095M Free > The specific patches you would want are: (in order) > http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch > and then the needfree.patch I already posted. > > The maxusers.patch being optional. Can do this but do we believe this will have any impact on the problem? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 11:04:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C66010656A6 for ; Mon, 6 Sep 2010 11:04:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id B85AF8FC08 for ; Mon, 6 Sep 2010 11:04:21 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o86B46Gi041975 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Sep 2010 14:04:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o86B46dE013894; Mon, 6 Sep 2010 14:04:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o86B46fN013893; Mon, 6 Sep 2010 14:04:06 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 6 Sep 2010 14:04:06 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20100906110406.GC2396@deviant.kiev.zoral.com.ua> References: <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84C857.1070306@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8OiOAIKaPG+pF/lZ" Content-Disposition: inline In-Reply-To: <4C84C857.1070306@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_40, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 11:04:22 -0000 --8OiOAIKaPG+pF/lZ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 06, 2010 at 01:54:15PM +0300, Andriy Gapon wrote: > on 06/09/2010 02:57 Steven Hartland said the following: > > Based on Jeremy's comments I'm updating the box the stable. Its buildin= g now > > but will be the morning before I can reboot to activate changes as I ne= ed to > > deactivate the stream instance and wait for all active connections to f= inish. > >=20 > > That said the problem doesn't seem to be cache + free but more cache + = free > > + inactive with inactive being the large chunk, so not sure this change > > would make any difference? > >=20 > > How does ufs deal with this, does it take inactive into account? Seems = a bit > > silly for inactive pages to prevent reuse for extended periods when the > > memory could be better used as cache. >=20 > Inactive pages are also a cache, just a different kind. Not quite. Inactive pages may be dirty. Such pages cannot be freed or reused without pageout. --8OiOAIKaPG+pF/lZ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkyEyqUACgkQC3+MBN1Mb4ihNwCg4VmgCRYh0+IOcXXcNBdz0yp3 PykAoJUB8210nDg466U5UMszRamFluej =V4aD -----END PGP SIGNATURE----- --8OiOAIKaPG+pF/lZ-- From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 11:06:56 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBE3D10656B7 for ; Mon, 6 Sep 2010 11:06:55 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AF3F88FC21 for ; Mon, 6 Sep 2010 11:06:55 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o86B6tIo011768 for ; Mon, 6 Sep 2010 11:06:55 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o86B6tja011766 for freebsd-fs@FreeBSD.org; Mon, 6 Sep 2010 11:06:55 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Sep 2010 11:06:55 GMT Message-Id: <201009061106.o86B6tja011766@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 11:06:56 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/150207 fs zpool import -d /dev tries to open weird devices o kern/150143 fs [patch][tmpfs] Source directory vnode can disappear be o kern/149855 fs [gvinum] growfs causes fsck to report errors in Filesy o kern/149495 fs [zfs] chflags sappend on zfs not working right o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149022 fs [hang] File system operations hangs with suspfs state o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o bin/148296 fs [zfs] [loader] [patch] Very slow probe in /usr/src/sys o kern/148204 fs [nfs] UDP NFS causes overload o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147790 fs [zfs] zfs set acl(mode|inherit) fails on existing zfs o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/147292 fs [nfs] [patch] readahead missing in nfs client options o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server o kern/146375 fs [nfs] [patch] Typos in macro variables names in sys/fs s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an o kern/145309 fs [disklabel] Editing disk label invalidates the whole d o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c o kern/144458 fs [nfs] [patch] nfsd fails as a kld p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] [iconv] mount_msdosfs: msdosfs_iconv: Operat o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr a docs/61716 fs [patch] newfs(8) code and manpage are out of sync o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 190 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 11:10:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2A9510656C6 for ; Mon, 6 Sep 2010 11:10:16 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 057008FC1C for ; Mon, 6 Sep 2010 11:10:15 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA00703; Mon, 06 Sep 2010 14:10:13 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4C84CC15.5000907@icyb.net.ua> Date: Mon, 06 Sep 2010 14:10:13 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100823 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Kostik Belousov References: <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84C857.1070306@icyb.net.ua> <20100906110406.GC2396@deviant.kiev.zoral.com.ua> In-Reply-To: <20100906110406.GC2396@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 11:10:16 -0000 on 06/09/2010 14:04 Kostik Belousov said the following: > On Mon, Sep 06, 2010 at 01:54:15PM +0300, Andriy Gapon wrote: >> on 06/09/2010 02:57 Steven Hartland said the following: >>> Based on Jeremy's comments I'm updating the box the stable. Its building now >>> but will be the morning before I can reboot to activate changes as I need to >>> deactivate the stream instance and wait for all active connections to finish. >>> >>> That said the problem doesn't seem to be cache + free but more cache + free >>> + inactive with inactive being the large chunk, so not sure this change >>> would make any difference? >>> >>> How does ufs deal with this, does it take inactive into account? Seems a bit >>> silly for inactive pages to prevent reuse for extended periods when the >>> memory could be better used as cache. >> >> Inactive pages are also a cache, just a different kind. > Not quite. Inactive pages may be dirty. Such pages cannot be freed or reused > without pageout. Yes, they can be dirty, but I didn't say that inactive was a "clean-only" cache. ARC entries can also be dirty and it's still a cache. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 11:50:22 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39EAF10656AB; Mon, 6 Sep 2010 11:50:22 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (core.vx.sk [IPv6:2a01:4f8:100:1043::2]) by mx1.freebsd.org (Postfix) with ESMTP id C420A8FC16; Mon, 6 Sep 2010 11:50:21 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id D98A111410A; Mon, 6 Sep 2010 13:50:20 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id F7nZcdZZHzy9; Mon, 6 Sep 2010 13:50:19 +0200 (CEST) Received: from [10.0.3.17] (188-167-67-67.dynamic.chello.sk [188.167.67.67]) by mail.vx.sk (Postfix) with ESMTPSA id 0CC7A114101; Mon, 6 Sep 2010 13:50:19 +0200 (CEST) Message-ID: <4C84D581.6060205@FreeBSD.org> Date: Mon, 06 Sep 2010 13:50:25 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sk; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org References: <20100831215915.GE1932@garage.freebsd.pl> In-Reply-To: <20100831215915.GE1932@garage.freebsd.pl> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=windows-1250 Content-Transfer-Encoding: 7bit Cc: freebsd-current@FreeBSD.org Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 11:50:22 -0000 Hi everyone, I have put together a slightly improved patch of Pawel's that compiles correctly and supports booting from ZFS v19 pools. You can download the patch here: http://people.freebsd.org/~mm/patches/zfs/head-zfs-v28-20100831.patch For users who don't want to compile I have created a mfsBSD ISO image with a zfs-only install of 9-CURRENT-amd64: http://mfsbsd.vx.sk/iso/mfsbsd-se-head-zfsv28-amd64.iso You can read more about all of this here: http://blog.vx.sk/archives/9-Help-testing-ZFS-v28-in-FreeBSD.html More information about ZFS pool and filesystem versions: http://blog.vx.sk/archives/4-ZFS-pool-and-filesystem-versions.html Thanks to everybody participating in testing, mm From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 12:35:15 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E84A10656C7 for ; Mon, 6 Sep 2010 12:35:15 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 95AC78FC1A for ; Mon, 6 Sep 2010 12:35:14 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 13:35:09 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 13:35:09 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011186407.msg for ; Mon, 06 Sep 2010 13:35:08 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <640F8FCE649448A4A8FF22214C993B10@multiplay.co.uk> From: "Steven Hartland" To: "Kostik Belousov" , "Andriy Gapon" References: <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84C857.1070306@icyb.net.ua> <20100906110406.GC2396@deviant.kiev.zoral.com.ua> Date: Mon, 6 Sep 2010 13:35:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 12:35:15 -0000 On Mon, Sep 06, 2010 at 01:54:15PM +0300, Andriy Gapon wrote: >>> How does ufs deal with this, does it take inactive into account? Seems a bit >>> silly for inactive pages to prevent reuse for extended periods when the >>> memory could be better used as cache. >> >> Inactive pages are also a cache, just a different kind. > Not quite. Inactive pages may be dirty. Such pages cannot be freed or reused > without pageout. I assume my little test that allocates ram then frees it, which then returned to "free" rules this out in this case though? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 13:23:28 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EBAC10656C0 for ; Mon, 6 Sep 2010 13:23:28 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id B869B8FC19 for ; Mon, 6 Sep 2010 13:23:27 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 14:23:23 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 14:23:23 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011186617.msg for ; Mon, 06 Sep 2010 14:23:23 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Andriy Gapon" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C84C72A.3020506@icyb.net.ua> Date: Mon, 6 Sep 2010 14:23:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 13:23:28 -0000 ----- Original Message ----- From: "Andriy Gapon" >> No joy, still drops down to arc_min even with those two patches and changing >> to vm_paging_needed from the post Artem mentioned: >> http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html >> >> So I suspect if I hadn't put in a high arc_min as well it would be back down >> at silly low levels. > > But we don't really know this, do we? > > I think that it would be useful for you and perhaps for us, if you'd set up > monitoring (and graphing) of key memory-related parameters. > E.g. at least the following sysctls: > kstat.zfs.misc.arcstats.size > vm.stats.vm.v_pdwakeups > vm.stats.vm.v_cache_count > vm.stats.vm.v_inactive_count > vm.stats.vm.v_active_count > vm.stats.vm.v_wire_count > vm.stats.vm.v_free_count > > This would allow to see dynamics of memory consumption and correlation with > pagedaemon events. Now monitoring these each minute to an rrd and text file and updated 8-STABLE with the following patches: http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch and then the needfree.patch I already posted. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 13:26:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5714D10656BB for ; Mon, 6 Sep 2010 13:26:29 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 9B9198FC15 for ; Mon, 6 Sep 2010 13:26:28 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA02948; Mon, 06 Sep 2010 16:26:22 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4C84EBFE.7030408@icyb.net.ua> Date: Mon, 06 Sep 2010 16:26:22 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100823 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Steven Hartland References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C84C72A.3020506@icyb.net.ua> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, jhell Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 13:26:29 -0000 on 06/09/2010 16:23 Steven Hartland said the following: > ----- Original Message ----- From: "Andriy Gapon" >>> No joy, still drops down to arc_min even with those two patches and changing >>> to vm_paging_needed from the post Artem mentioned: >>> http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html >>> >>> So I suspect if I hadn't put in a high arc_min as well it would be back down >>> at silly low levels. >> >> But we don't really know this, do we? >> >> I think that it would be useful for you and perhaps for us, if you'd set up >> monitoring (and graphing) of key memory-related parameters. >> E.g. at least the following sysctls: >> kstat.zfs.misc.arcstats.size >> vm.stats.vm.v_pdwakeups >> vm.stats.vm.v_cache_count >> vm.stats.vm.v_inactive_count >> vm.stats.vm.v_active_count >> vm.stats.vm.v_wire_count >> vm.stats.vm.v_free_count >> >> This would allow to see dynamics of memory consumption and correlation with >> pagedaemon events. > > Now monitoring these each minute to an rrd and text file and updated 8-STABLE > with the following patches: > http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch > http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch > and then the needfree.patch I already posted. Cool! What about vm_paging_needed() patch? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 13:29:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBDF0106567A for ; Mon, 6 Sep 2010 13:29:18 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 684218FC14 for ; Mon, 6 Sep 2010 13:29:18 +0000 (UTC) Received: by ywt2 with SMTP id 2so1875610ywt.13 for ; Mon, 06 Sep 2010 06:29:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=c8GZZQLdGvQQU28KMls76z83/ANXdn10hPImLUW3TVE=; b=EfD/f+8EdfxXsUrhJscoMSiFN0Nw0dBA82AvteZaBTvob3ke7zofQ3IFRM+qrgpBrp X5iO/2gZgGpPlK60/nlk4MCbT//KD77WIp+h4zTOUS6t2vJdFAy/AVAccvX4pt/H+ENL 0vYOf00UhYHIDQO4PSt1bEFZ/QRtDZLWRzQP8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=DMGmFOiN5WGHLa+K1o8osOcAuqUEBEom7g+5AmrWnBENlLz0fzf8ZtwyzmLDpUZ9Fj f3Z5yq9Z5x3WGPeQOUtOSRbAXyLEZz7iVDKjOlbd0ySmNx1p10AjoX1Zok0y4ig88347 7u4WF6/PkZ0KSgzigM1RHYKVUXZpq9OMEg8rY= Received: by 10.150.226.11 with SMTP id y11mr267155ybg.419.1283779757350; Mon, 06 Sep 2010 06:29:17 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id u42sm5977865yba.12.2010.09.06.06.29.15 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 06:29:16 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C84ECA8.8080807@DataIX.net> Date: Mon, 06 Sep 2010 09:29:12 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Matthias Gamsjager References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C83A214.1080204@DataIX.net> <06B9D23F202D4DB88D69B7C4507986B7@multiplay.co.uk> <4C842905.2080602@DataIX.net> <330B5DB2215F43899ABAEC2CF71C2EE0@multiplay.co.uk> <4C84341D.8060708@DataIX.net> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 13:29:18 -0000 On 09/06/2010 01:33, Matthias Gamsjager wrote: > All this patches should be posted on the zfs wiki. Make it much easier > for ppl to find it rather then digging through all the mailing lists. > Please see below comments. > >> http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch >> http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch >> http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch These patches will end up being MFC'd quoting mm@ sometime around ~September / Novemberish. All the patches mentioned are already in HEAD so its just a matter of time before it gets down to another branch. ZFSv15 had quite the number of bug fixes too so it would be wise to update your source to this and gain stability. The patches mentioned above also have bug fix descriptions along side of them in the same directory if you happen to be browsing through. -- jhell,v From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 13:39:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 41136106567A for ; Mon, 6 Sep 2010 13:39:44 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id EC2FB8FC13 for ; Mon, 6 Sep 2010 13:39:43 +0000 (UTC) Received: by iwn34 with SMTP id 34so4861513iwn.13 for ; Mon, 06 Sep 2010 06:39:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type; bh=SeCxFiKd4uKV33DJ/ZEijj+6MeFFDGjjJMr8o7DtHTg=; b=aTeRhhqsiMLa2B16r74GyLzqoTBK/vsaumLEbcLVxoJaptJZVI/ESviOkbiEDIuG35 K/U3mksqt70TgXLiGuU751s9Q7pBCblh4rZsq4FdJpBa1fSgWo6mT0AJE6DnVIiNIWg4 g/xSmAVPu/x4Q07vfD5eP/ZnzzKwe2GGj95F8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type; b=ssW6DgV/OxYH9IYYsdMXtt/TTX92HNFvUGDWBmz1BhtB/hyFi33I6FlFs4IJotonL/ aEc0mgcTNcMg8s+s8XDDFX7bBu6l/GiznSyGQjtpaY1UWEPezRM9exvF37ufVWN4oRLN LQLGU0OT4qLC+6iuugvzIu8qkM1/nbgB9Lq2s= Received: by 10.231.161.73 with SMTP id q9mr6326547ibx.70.1283780383264; Mon, 06 Sep 2010 06:39:43 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id h8sm5915455ibk.21.2010.09.06.06.39.41 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 06:39:42 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C84EF1B.5040004@DataIX.net> Date: Mon, 06 Sep 2010 09:39:39 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.8) Gecko/20100806 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Andriy Gapon References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk><4C825D65.3040004@DataIX.net> <7EA7AD058C0143B2BF2471CC121C1687@multiplay.co.uk> <1F64110BFBD5468B8B26879A9D8C94EF@multiplay.co.uk> <4C84C72A.3020506@icyb.net.ua> <4C84EBFE.7030408@icyb.net.ua> In-Reply-To: <4C84EBFE.7030408@icyb.net.ua> X-Enigmail-Version: 1.1.2 Content-Type: multipart/mixed; boundary="------------070309050609030205070100" Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 13:39:44 -0000 This is a multi-part message in MIME format. --------------070309050609030205070100 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 09/06/2010 09:26, Andriy Gapon wrote: > on 06/09/2010 16:23 Steven Hartland said the following: >> ----- Original Message ----- From: "Andriy Gapon" >>>> No joy, still drops down to arc_min even with those two patches and changing >>>> to vm_paging_needed from the post Artem mentioned: >>>> http://lists.freebsd.org/pipermail/freebsd-hackers/2010-August/032731.html >>>> >>>> So I suspect if I hadn't put in a high arc_min as well it would be back down >>>> at silly low levels. >>> >>> But we don't really know this, do we? >>> >>> I think that it would be useful for you and perhaps for us, if you'd set up >>> monitoring (and graphing) of key memory-related parameters. >>> E.g. at least the following sysctls: >>> kstat.zfs.misc.arcstats.size >>> vm.stats.vm.v_pdwakeups >>> vm.stats.vm.v_cache_count >>> vm.stats.vm.v_inactive_count >>> vm.stats.vm.v_active_count >>> vm.stats.vm.v_wire_count >>> vm.stats.vm.v_free_count >>> >>> This would allow to see dynamics of memory consumption and correlation with >>> pagedaemon events. >> >> Now monitoring these each minute to an rrd and text file and updated 8-STABLE >> with the following patches: >> http://people.freebsd.org/~mm/patches/zfs/v15/stable-8-v15.patch >> http://people.freebsd.org/~mm/patches/zfs/zfs_metaslab_v2.patch >> http://people.freebsd.org/~mm/patches/zfs/zfs_abe_stat_rrwlock.patch >> and then the needfree.patch I already posted. > > Cool! > > What about vm_paging_needed() patch? > Ugh! I keep forgetting about this one! and confusing it for the needfree change. /Ugh! It is attached as well. -- jhell,v --------------070309050609030205070100 Content-Type: text/plain; name="vm_paging_needed.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="vm_paging_needed.patch" diff -r 1f75c636f0d4 sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Mon Sep 06 09:32:32 2010 -0400 +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c Mon Sep 06 09:36:57 2010 -0400 @@ -2164,7 +2164,7 @@ * If pages are needed or we're within 2048 pages * of needing to page need to reclaim */ - if (vm_pages_needed || (vm_paging_target() > -2048)) + if (vm_paging_needed()) return (1); #if 0 --------------070309050609030205070100-- From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 19:01:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E4B9210656BE for ; Mon, 6 Sep 2010 19:01:04 +0000 (UTC) (envelope-from bsd@vink.pl) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id A34648FC21 for ; Mon, 6 Sep 2010 19:01:04 +0000 (UTC) Received: by qyk4 with SMTP id 4so4777121qyk.13 for ; Mon, 06 Sep 2010 12:01:04 -0700 (PDT) Received: by 10.224.89.11 with SMTP id c11mr797843qam.182.1283798104592; Mon, 06 Sep 2010 11:35:04 -0700 (PDT) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx.google.com with ESMTPS id r1sm5908272qcq.34.2010.09.06.11.35.04 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 11:35:04 -0700 (PDT) Received: by qwg5 with SMTP id 5so4402448qwg.13 for ; Mon, 06 Sep 2010 11:35:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.224.11.18 with SMTP id r18mr394734qar.195.1283798099497; Mon, 06 Sep 2010 11:34:59 -0700 (PDT) Received: by 10.229.100.131 with HTTP; Mon, 6 Sep 2010 11:34:59 -0700 (PDT) In-Reply-To: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> Date: Mon, 6 Sep 2010 20:34:59 +0200 Message-ID: From: Wiktor Niesiobedzki To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 19:01:05 -0000 2010/9/4 Steven Hartland : > When upgrading from 8.0 on our stream server to 8.1 we decided to go > for zfs to eliminate the costly fsck times should we experience > any unexpected reboots on the machine as it has a sizable RAID of > 1.6TB. > > After doing this all seemed good till after our latest event which > generated a significant amount of interest and hence the stream > server started to get quite a few requests. > > Basic install is 8.1 amd64 on a dual 2.8 Xeon with 4GB RAM and > areca controller with 6 disk in RAID 6. > > On that the machine runs nginx with the mp4 module to sudo stream > files. Hi, As far as I have check recently, nginx is using sendfile by default. There is already a reported bug against ZFS+sendfile (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results in bad performance. The quickest workaround is to set: sendfile off; In http {} sectio of nginx.conf. What I personally have observed, is that memory, that is used by sendfile, once freed lands in Inact group. And ARC is not able to force free of this memory. In my case, where I have 1G of ARC, then after sending 2G of file, my ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. If I remove the file that was sent through sendfile, memory is moved from Inact to free, from where ARC happly grabs what it wants, and ARC hit ratio comes back to normal (~99%). Cheers, Wiktor Niesiobedzki From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 19:07:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C4B710656B6 for ; Mon, 6 Sep 2010 19:07:46 +0000 (UTC) (envelope-from bsd@vink.pl) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 325938FC15 for ; Mon, 6 Sep 2010 19:07:46 +0000 (UTC) Received: by qwg5 with SMTP id 5so4423531qwg.13 for ; Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: by 10.224.11.20 with SMTP id r20mr200581qar.223.1283800065337; Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx.google.com with ESMTPS id t1sm5950642qcs.9.2010.09.06.12.07.44 (version=SSLv3 cipher=RC4-MD5); Mon, 06 Sep 2010 12:07:45 -0700 (PDT) Received: by qyk4 with SMTP id 4so4782445qyk.13 for ; Mon, 06 Sep 2010 12:07:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.248.84 with SMTP id mf20mr3828211qcb.16.1283800064508; Mon, 06 Sep 2010 12:07:44 -0700 (PDT) Received: by 10.229.100.131 with HTTP; Mon, 6 Sep 2010 12:07:44 -0700 (PDT) In-Reply-To: References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> Date: Mon, 6 Sep 2010 21:07:44 +0200 Message-ID: From: Wiktor Niesiobedzki To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 19:07:46 -0000 2010/9/6 Wiktor Niesiobedzki : > Hi, > > As far as I have check recently, nginx is using sendfile by default. > There is already a reported bug against ZFS+sendfile > (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results > in bad performance. > > What I personally have observed, is that memory, that is used by > sendfile, once freed lands in Inact group. And ARC is not able to > force free of this memory. > > In my case, where I have 1G of ARC, then after sending 2G of file, my > ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. if did some further tests with nginx and sendfile setting. The setup is following: nginx have a 1GB file to be served to client ARC_MIN=256M, ARC_MAX=1G KMEM_SIZE=1.5G With sendfile off: 1. first download leads to read from disk (as suspected) 2. second download, mostly from ARC (some minor activity on disk, like 1-10% of actual transfer) measured by gstat With sendfile on (after doing previous tests): 1. first download - leads to read from disk (suspicous - this file should be already in ARC) 2. second download - leads to read from disk (suspicous - this file should be in both ARC and Inactive memory) After that the memory looks like: Mem: 58M Active, 1032M Inact, 723M Wired, 121M Free arc size: 670M With a 512MB file, with sendfile on, starting with memory like: Mem: 59M Active, 8948K Inact, 816M Wired, 1050M Free arc size: 780M ARC warmed with test file with: cat tfile512M > /dev/null 1. first fownload - no disk activity Mem: 51M Active, 517M Inact, 822M Wired, 545M Free arc size: 790M 2. second download - no disk activity Mem: 51M Active, 517M Inact, 822M Wired, 544M Free arc size: 790M The test is taking about 90 seconds, and kstat.zfs.misc.arcstats.hits goes up by ~2M In normal situation (no download activity) it goes up by: 200 During subsequent cat tfile512M > /dev/null it goes up by: 131k During nginx download (with sendfile off) it goes up by: 23k So my gut feelings about this situation are: 1. Sendfile on ZFS is not a "zero-copy" solution (we copy from ARC to some other memory before sending a file) 2. Whatever sendfile puts in some "cache", it's never used, as we will use ARC anyway (see big number of arc hits) Some other side observerations are: - nginx is faster (by 50%) with sendfile turned off (not benchmarked, just a feeling) - arcstat hits is growing extermely fast with sendfile (small request from sendfile to ARC?) - nginx is using quite small number of accesses to ARC, even compared to simple cat, to get the same file size (if I at all properly understand what does kstat.zfs.misc.arcstats.hits mean) Hope that helps, Cheers, Wiktor Niesiobedzki From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 19:19:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DBAC10656DB for ; Mon, 6 Sep 2010 19:19:17 +0000 (UTC) (envelope-from prvs=186543ef20=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 102A38FC1B for ; Mon, 6 Sep 2010 19:19:16 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 20:19:12 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Mon, 06 Sep 2010 20:19:12 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011188227.msg for ; Mon, 06 Sep 2010 20:19:12 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=186543ef20=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: From: "Steven Hartland" To: "Wiktor Niesiobedzki" , "jhell" , "Andriy Gapon" References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> Date: Mon, 6 Sep 2010 20:19:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 19:19:17 -0000 ----- Original Message ----- From: "Wiktor Niesiobedzki" To: "Steven Hartland" Cc: Sent: Monday, September 06, 2010 8:07 PM Subject: Re: zfs very poor performance compared to ufs due to lack of cache? > 2010/9/6 Wiktor Niesiobedzki : >> Hi, >> >> As far as I have check recently, nginx is using sendfile by default. >> There is already a reported bug against ZFS+sendfile >> (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results >> in bad performance. >> >> What I personally have observed, is that memory, that is used by >> sendfile, once freed lands in Inact group. And ARC is not able to >> force free of this memory. >> >> In my case, where I have 1G of ARC, then after sending 2G of file, my >> ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. > > if did some further tests with nginx and sendfile setting. The setup > is following: > nginx have a 1GB file to be served to client > ARC_MIN=256M, ARC_MAX=1G > KMEM_SIZE=1.5G > > With sendfile off: > 1. first download leads to read from disk (as suspected) > 2. second download, mostly from ARC (some minor activity on disk, like > 1-10% of actual transfer) measured by gstat > > With sendfile on (after doing previous tests): > 1. first download - leads to read from disk (suspicous - this file > should be already in ARC) > 2. second download - leads to read from disk (suspicous - this file > should be in both ARC and Inactive memory) > > After that the memory looks like: > Mem: 58M Active, 1032M Inact, 723M Wired, 121M Free > arc size: 670M > > > With a 512MB file, with sendfile on, starting with memory like: > > Mem: 59M Active, 8948K Inact, 816M Wired, 1050M Free > arc size: 780M > > ARC warmed with test file with: > cat tfile512M > /dev/null > > 1. first fownload - no disk activity > Mem: 51M Active, 517M Inact, 822M Wired, 545M Free > arc size: 790M > 2. second download - no disk activity > > Mem: 51M Active, 517M Inact, 822M Wired, 544M Free > arc size: 790M > > The test is taking about 90 seconds, and > kstat.zfs.misc.arcstats.hits goes up by ~2M > > In normal situation (no download activity) it goes up by: 200 > > During subsequent cat tfile512M > /dev/null it goes up by: 131k > > During nginx download (with sendfile off) it goes up by: 23k > > So my gut feelings about this situation are: > 1. Sendfile on ZFS is not a "zero-copy" solution (we copy from ARC to > some other memory before sending a file) > 2. Whatever sendfile puts in some "cache", it's never used, as we will > use ARC anyway (see big number of arc hits) > > Some other side observerations are: > - nginx is faster (by 50%) with sendfile turned off (not benchmarked, > just a feeling) > - arcstat hits is growing extermely fast with sendfile (small request > from sendfile to ARC?) > - nginx is using quite small number of accesses to ARC, even compared > to simple cat, to get the same file size (if I at all properly > understand what does kstat.zfs.misc.arcstats.hits mean) > > Hope that helps, Oooo, thanks for that Wiktor sounds like you might have nailed the issue there. So guys what do you recon is this the main issue I'm seeing. Would you like me to do any testing to confirm this? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 6 22:08:57 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6757010656A8; Mon, 6 Sep 2010 22:08:57 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (core.vx.sk [IPv6:2a01:4f8:100:1043::2]) by mx1.freebsd.org (Postfix) with ESMTP id 26DA88FC0C; Mon, 6 Sep 2010 22:08:57 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id 83EB11165A9; Tue, 7 Sep 2010 00:08:56 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk ([127.0.0.1]) by core.vx.sk (mail.vx.sk [127.0.0.1]) (amavisd-new, port 10024) with LMTP id VRMIP7zR+tUL; Tue, 7 Sep 2010 00:08:54 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-139.dynamic.chello.sk [188.167.78.139]) by mail.vx.sk (Postfix) with ESMTPSA id 5272A1165A3; Tue, 7 Sep 2010 00:08:54 +0200 (CEST) Message-ID: <4C856677.6020004@FreeBSD.org> Date: Tue, 07 Sep 2010 00:08:55 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; sk; rv:1.8.1.23) Gecko/20090812 Lightning/0.9 Thunderbird/2.0.0.23 Mnenhy/0.7.5.0 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org References: <20100831215915.GE1932@garage.freebsd.pl> <4C84D581.6060205@FreeBSD.org> In-Reply-To: <4C84D581.6060205@FreeBSD.org> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=windows-1250 Content-Transfer-Encoding: 8bit Cc: freebsd-current@FreeBSD.org Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 22:08:57 -0000 To avoid user and developer confusion, my patch was just a chain of pjd's patch + pjd's atomic.h fix + my v19 boot patch. I have removed (= split) the chained patch in my posting and altered my blog article with updated build instructions that actually just summarize what has been written on this list. Thanks for understanding! Dòa 6. 9. 2010 13:50, Martin Matuska wrote / napísal(a): > Hi everyone, > I have put together a slightly improved patch of Pawel's that compiles > correctly and supports booting from ZFS v19 pools. > > You can download the patch here: > http://people.freebsd.org/~mm/patches/zfs/head-zfs-v28-20100831.patch > > For users who don't want to compile I have created a mfsBSD ISO image > with a zfs-only install of 9-CURRENT-amd64: > http://mfsbsd.vx.sk/iso/mfsbsd-se-head-zfsv28-amd64.iso > > You can read more about all of this here: > http://blog.vx.sk/archives/9-Help-testing-ZFS-v28-in-FreeBSD.html > > More information about ZFS pool and filesystem versions: > http://blog.vx.sk/archives/4-ZFS-pool-and-filesystem-versions.html > > Thanks to everybody participating in testing, > mm From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 00:21:27 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA3D110656A6; Tue, 7 Sep 2010 00:21:27 +0000 (UTC) (envelope-from arundel@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 90B0E8FC15; Tue, 7 Sep 2010 00:21:27 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o870LRbe042155; Tue, 7 Sep 2010 00:21:27 GMT (envelope-from arundel@freefall.freebsd.org) Received: (from arundel@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o870LRp3042151; Tue, 7 Sep 2010 00:21:27 GMT (envelope-from arundel) Date: Tue, 7 Sep 2010 00:21:27 GMT Message-Id: <201009070021.o870LRp3042151@freefall.freebsd.org> To: arundel@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: arundel@FreeBSD.org Cc: Subject: Re: kern/127787: [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfslock, user map/vfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 00:21:27 -0000 Synopsis: [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfslock, user map/vfs Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: arundel Responsible-Changed-When: Tue Sep 7 00:20:41 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=127787 From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 00:22:39 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0F0810656B2; Tue, 7 Sep 2010 00:22:39 +0000 (UTC) (envelope-from arundel@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A74F68FC0A; Tue, 7 Sep 2010 00:22:39 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o870Mdlc042245; Tue, 7 Sep 2010 00:22:39 GMT (envelope-from arundel@freefall.freebsd.org) Received: (from arundel@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o870MdpH042241; Tue, 7 Sep 2010 00:22:39 GMT (envelope-from arundel) Date: Tue, 7 Sep 2010 00:22:39 GMT Message-Id: <201009070022.o870MdpH042241@freefall.freebsd.org> To: arundel@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: arundel@FreeBSD.org Cc: Subject: Re: kern/117954: [ufs] dirhash on very large directories blocks the machine for tens of seconds X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 00:22:39 -0000 Synopsis: [ufs] dirhash on very large directories blocks the machine for tens of seconds Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: arundel Responsible-Changed-When: Tue Sep 7 00:21:40 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=117954 From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 07:26:28 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 67FD01065695; Tue, 7 Sep 2010 07:26:28 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3FF918FC08; Tue, 7 Sep 2010 07:26:26 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA16381; Tue, 07 Sep 2010 10:26:23 +0300 (EEST) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OssZT-000MiY-9N; Tue, 07 Sep 2010 10:26:23 +0300 Message-ID: <4C85E91E.1010602@icyb.net.ua> Date: Tue, 07 Sep 2010 10:26:22 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Wiktor Niesiobedzki , Pawel Jakub Dawidek , Konstantin Belousov References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 07:26:28 -0000 on 06/09/2010 21:34 Wiktor Niesiobedzki said the following: > As far as I have check recently, nginx is using sendfile by default. > There is already a reported bug against ZFS+sendfile > (http://www.freebsd.org/cgi/query-pr.cgi?pr=141305&cat=) which results > in bad performance. > > The quickest workaround is to set: > sendfile off; > > In http {} sectio of nginx.conf. > > What I personally have observed, is that memory, that is used by > sendfile, once freed lands in Inact group. And ARC is not able to > force free of this memory. Well, there is a patch for this, but that's besides the point of the sendfile issue. > In my case, where I have 1G of ARC, then after sending 2G of file, my > ARC is barerly on minimum level, and my ARC hit ratio drops to ~50%. > > If I remove the file that was sent through sendfile, memory is moved > from Inact to free, from where ARC happly grabs what it wants, and ARC > hit ratio comes back to normal (~99%). Interesting. I briefly looked at the code in mappedread(), zfs_vnops.c, and I have a VM question. Shouldn't we mark the corresponding page bits as valid after reading data into the page? I specifically speak of the block that starts with the following line: } else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) { I am taking mdstart_swap as an example and it does m->valid = VM_PAGE_BITS_ALL. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 10:07:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70BCB10656CE for ; Tue, 7 Sep 2010 10:07:42 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 25B478FC20 for ; Tue, 7 Sep 2010 10:07:41 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Osv5Y-0006yx-6D for freebsd-fs@freebsd.org; Tue, 07 Sep 2010 12:07:40 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 07 Sep 2010 12:07:40 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 07 Sep 2010 12:07:40 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 07 Sep 2010 12:07:31 +0200 Lines: 16 Message-ID: References: <201009060708.o867864G033410@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.9) Gecko/20100518 Thunderbird/3.0.4 In-Reply-To: <201009060708.o867864G033410@freefall.freebsd.org> X-Enigmail-Version: 1.0.1 Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 10:07:42 -0000 On 09/06/10 09:08, linimon@FreeBSD.org wrote: > Synopsis: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename > > Responsible-Changed-From-To: freebsd-bugs->freebsd-fs > Responsible-Changed-By: linimon > Responsible-Changed-When: Mon Sep 6 07:07:55 UTC 2010 > Responsible-Changed-Why: > Over to maintainer(s). > > http://www.freebsd.org/cgi/query-pr.cgi?pr=150143 Can someone look at the patch in this PR, please? I've tested it but don't want to commit it without someone reviewing it (though I could commit it if the potential reviewer is busy with other things). From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 10:10:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FB0010656B5; Tue, 7 Sep 2010 10:10:55 +0000 (UTC) (envelope-from k0802647@telus.net) Received: from outbound03.telus.net (outbound03.telus.net [199.185.220.222]) by mx1.freebsd.org (Postfix) with ESMTP id E51F98FC1A; Tue, 7 Sep 2010 10:10:54 +0000 (UTC) Received: from edtnaa01.telusplanet.net ([66.183.53.162]) by priv-edtnes29.telusplanet.net (InterMail vM.7.09.01.00 201-2219-108-20080618) with ESMTP id <20100907101054.UGUP11932.priv-edtnes29.telusplanet.net@edtnaa01.telusplanet.net>; Tue, 7 Sep 2010 04:10:54 -0600 Received: from oliver.bc.lan (d66-183-53-162.bchsia.telus.net [66.183.53.162]) by edtnaa01.telusplanet.net (BorderWare Security Platform) with ESMTP id 18C083C1EBA851DA; Tue, 7 Sep 2010 04:10:53 -0600 (MDT) Received: from [10.111.111.112] (unknown [10.111.111.112]) by oliver.bc.lan (Postfix) with ESMTP id 5E3B862A4; Tue, 7 Sep 2010 03:10:53 -0700 (PDT) Message-ID: <4C860FAC.5070700@telus.net> Date: Tue, 07 Sep 2010 03:10:52 -0700 From: Carl User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.7) Gecko/20100713 Lightning/1.0b2 Thunderbird/3.1.1 MIME-Version: 1.0 To: Pawel Jakub Dawidek , freebsd-fs@freebsd.org References: <4C834D1A.2010405@telus.net> <20100905150344.GD1900@garage.freebsd.pl> In-Reply-To: <20100905150344.GD1900@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Cloudmark-Analysis: v=1.1 cv=+EoSOrAnwUvjIXL1uzjT8lcUSwg7SnZy9aakk+NQyVs= c=1 sm=0 a=pl1w12ep0ysA:10 a=8nJEP1OIZ-IA:10 a=HNgjH8kF64GtJ7EcXKEMsQ==:17 a=mnuoJeblHigEZOc-yEoA:9 a=B87ekR2KspwKf56rLXRvfhBGXK8A:4 a=wPNLvfGTeEIA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 Cc: Subject: Re: geli'd swap and core dumps X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 10:10:55 -0000 On 2010-09-05 8:03 AM, Pawel Jakub Dawidek wrote: >> What are best practices for achieving encrypted swap and functional core >> dump recovery? Or are these mutually exclusive goals? > > Well, the idea to encrypt swap is to prevent any sensitive data to be > stored on disk unencrypted where it might last for a long time. > If you configure to dump kernel memory to a disk (kernel dumps are not > encrypted) you kinda miss the point, as kernel memory can contain a lot > of sensitive data. It makes sense that best practice would be to disable the dump device, yet it appears dumpdev is set to AUTO as the default on current versions of FreeBSD. Does AUTO imply a behaviour that will intelligently recognize the lack of a functional dumpdev in the case of a geli'd swap or do I need to explicitly set dumpdev to NO to avoid errors on normal startups or even bad behaviour during a kernel panic? Carl / K0802647 From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 10:33:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5BAA10656A4 for ; Tue, 7 Sep 2010 10:33:29 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id 801BC8FC20 for ; Tue, 7 Sep 2010 10:33:29 +0000 (UTC) Received: from omta23.westchester.pa.mail.comcast.net ([76.96.62.74]) by qmta14.westchester.pa.mail.comcast.net with comcast id 3lvr1f0021c6gX85EmZVEP; Tue, 07 Sep 2010 10:33:29 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta23.westchester.pa.mail.comcast.net with comcast id 3mZU1f0083LrwQ23jmZVy4; Tue, 07 Sep 2010 10:33:29 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5C0329B425; Tue, 7 Sep 2010 03:33:27 -0700 (PDT) Date: Tue, 7 Sep 2010 03:33:27 -0700 From: Jeremy Chadwick To: Carl Message-ID: <20100907103327.GA12222@icarus.home.lan> References: <4C834D1A.2010405@telus.net> <20100905150344.GD1900@garage.freebsd.pl> <4C860FAC.5070700@telus.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C860FAC.5070700@telus.net> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: geli'd swap and core dumps X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 10:33:30 -0000 On Tue, Sep 07, 2010 at 03:10:52AM -0700, Carl wrote: > On 2010-09-05 8:03 AM, Pawel Jakub Dawidek wrote: > >>What are best practices for achieving encrypted swap and functional core > >>dump recovery? Or are these mutually exclusive goals? > > > >Well, the idea to encrypt swap is to prevent any sensitive data to be > >stored on disk unencrypted where it might last for a long time. > >If you configure to dump kernel memory to a disk (kernel dumps are not > >encrypted) you kinda miss the point, as kernel memory can contain a lot > >of sensitive data. > > It makes sense that best practice would be to disable the dump > device, yet it appears dumpdev is set to AUTO as the default on > current versions of FreeBSD. Does AUTO imply a behaviour that will > intelligently recognize the lack of a functional dumpdev in the case > of a geli'd swap or do I need to explicitly set dumpdev to NO to > avoid errors on normal startups or even bad behaviour during a > kernel panic? dumpdev="auto" results in the system rc scripts examining /etc/fstab to look for any swap slices you've defined there. If there are none listed in /etc/fstab, then you should set the dump device explicitly using dumpdev="/dev/xxx" syntax. If there are some in /etc/fstab which you don't want to use, apply the same advice. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 16:58:09 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3862410656E8 for ; Tue, 7 Sep 2010 16:58:09 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from forward12.mail.yandex.net (forward12.mail.yandex.net [95.108.130.94]) by mx1.freebsd.org (Postfix) with ESMTP id DAE178FC12 for ; Tue, 7 Sep 2010 16:58:08 +0000 (UTC) Received: from smtp13.mail.yandex.net (smtp13.mail.yandex.net [95.108.130.68]) by forward12.mail.yandex.net (Yandex) with ESMTP id 127892210976; Tue, 7 Sep 2010 20:42:07 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1283877727; bh=9gue2XPFOGAAwD1LpZEJ1m7uvuM+PFkAET0B07FSKe0=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type; b=qs1f4KTku6198rfYNgUBtNE/nFXOMicRFwRSLMZ1g4oS2FSvmVMwA2y7+DO/6sg5j Ayj+jIr4UqA6BAn3PJNdW9BnexeLp8a3MYP/ISi2UXgP0Crs35qK+hnSlcppjkJaG+ GEL+6TBuJJZcdlG1evssx3jX7skvz7f4qfNhAZ4E= Received: from nautilus (unknown [178.155.56.14]) by smtp13.mail.yandex.net (Yandex) with ESMTPA id D6F7741580A8; Tue, 7 Sep 2010 20:42:06 +0400 (MSD) Received: by nautilus (Postfix, from userid 1001) id 33E1D1DD43D; Tue, 7 Sep 2010 20:42:04 +0400 (MSD) Date: Tue, 7 Sep 2010 20:42:04 +0400 From: Sergey Zaharchenko To: freebsd-fs@freebsd.org Message-ID: <20100907164204.GA2571@nautilus.vmks.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="HlL+5n6rz5pIUxbD" Content-Disposition: inline X-Listening-To: Silence User-Agent: Mutt/1.5.20 (2009-06-14) X-Yandex-TimeMark: 1283877727 X-Yandex-Spam: 1 X-Yandex-Front: smtp13.mail.yandex.net Cc: Pawel Jakub Dawidek Subject: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 16:58:09 -0000 --HlL+5n6rz5pIUxbD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello list, hello Pawel, I'm observing a reproducible (a matter of seconds) ZFS `deadlock'. This is on an 8.1-RELEASE amd64 quad-core system with 8G of RAM. It takes a few writes/syncs to the filesystem to make it unusable ('bonnie++ -f' works). Processes are stuck in states like zio->..., zil->..., etc. Looks like I'm not the only one: http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058362.html My problem is described more detailedly in: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058665.html Please tell me if there's anything else that you need. I can file a PR if you prefer it that way... Thanks, --=20 Sergey Zaharchenko --HlL+5n6rz5pIUxbD Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyGa1sACgkQwo7hT/9lVdxfawCffJ7Ls9Ai2lesiHIqSWHka1pl XxIAnidI6joPfyli+sTyA2Jn9goz7iwp =f7vQ -----END PGP SIGNATURE----- --HlL+5n6rz5pIUxbD-- From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 17:17:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C627E10656D5; Tue, 7 Sep 2010 17:17:58 +0000 (UTC) (envelope-from kitchetech@gmail.com) Received: from mail-px0-f182.google.com (mail-px0-f182.google.com [209.85.212.182]) by mx1.freebsd.org (Postfix) with ESMTP id 882EE8FC17; Tue, 7 Sep 2010 17:17:58 +0000 (UTC) Received: by pxi17 with SMTP id 17so1675973pxi.13 for ; Tue, 07 Sep 2010 10:17:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:received :in-reply-to:references:date:message-id:subject:from:to:cc :content-type; bh=URvQOmw3WfFWL2bmRSEiIRdkWH5bcirb8UBnY2XqNY4=; b=iYytzxNsJq2vKRC519ti0gJkRWXQk1boNUxIkPUhqpkW+gsVCcVwy8zY47asWqSUqu zeAfzEj7t7RcTadvVrIVmyD1net20rDoT/ILruY+ZWvIDYgZ4SO2gO/n6rl5VkE6Yh/v qErwJWWU7jgsp8fX2X0SghiSIQF+46xzB4ffc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=ptoXcOSIr8So+3WwR2dK71f1hI1LTjbTDN4FKHbg4Q9g/zH2JlhahntYSGpzXPNPkB qOUzgFSvFe8dZBUBFnqWfeModujVfG53jbKcJ5ZK/MltGgb6684JM5JCE6TJyYimpbn5 +3cOEHRA83QlkeMtlzNLcjZNttURir6viDlvQ= MIME-Version: 1.0 Received: by 10.142.9.35 with SMTP id 35mr39553wfi.169.1283878067388; Tue, 07 Sep 2010 09:47:47 -0700 (PDT) Received: by 10.231.12.6 with HTTP; Tue, 7 Sep 2010 09:47:46 -0700 (PDT) Received: by 10.231.12.6 with HTTP; Tue, 7 Sep 2010 09:47:46 -0700 (PDT) In-Reply-To: <4C856677.6020004@FreeBSD.org> References: <20100831215915.GE1932@garage.freebsd.pl> <4C84D581.6060205@FreeBSD.org> <4C856677.6020004@FreeBSD.org> Date: Tue, 7 Sep 2010 12:47:46 -0400 Message-ID: From: matt donovan To: Martin Matuska Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 17:17:58 -0000 The atomic.h patch does not apply anymore and buildworld fails without it. At work so can't post results On Sep 6, 2010 6:09 PM, "Martin Matuska" wrote: > To avoid user and developer confusion, my patch was just a chain of > pjd's patch + pjd's atomic.h fix + my v19 boot patch. > > I have removed (=3D split) the chained patch in my posting and altered my > blog article with updated build instructions that actually just > summarize what has been written on this list. > > Thanks for understanding! > > D=C5=88a 6. 9. 2010 13:50, Martin Matuska wrote / nap=C3=ADsal(a): >> Hi everyone, >> I have put together a slightly improved patch of Pawel's that compiles >> correctly and supports booting from ZFS v19 pools. >> >> You can download the patch here: >> http://people.freebsd.org/~mm/patches/zfs/head-zfs-v28-20100831.patch >> >> For users who don't want to compile I have created a mfsBSD ISO image >> with a zfs-only install of 9-CURRENT-amd64: >> http://mfsbsd.vx.sk/iso/mfsbsd-se-head-zfsv28-amd64.iso >> >> You can read more about all of this here: >> http://blog.vx.sk/archives/9-Help-testing-ZFS-v28-in-FreeBSD.html >> >> More information about ZFS pool and filesystem versions: >> http://blog.vx.sk/archives/4-ZFS-pool-and-filesystem-versions.html >> >> Thanks to everybody participating in testing, >> mm > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org= " From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 19:32:06 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47F4210656D9 for ; Tue, 7 Sep 2010 19:32:06 +0000 (UTC) (envelope-from mgamsjager@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id F21738FC24 for ; Tue, 7 Sep 2010 19:32:05 +0000 (UTC) Received: by qyk4 with SMTP id 4so5953677qyk.13 for ; Tue, 07 Sep 2010 12:32:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:received :in-reply-to:references:date:message-id:subject:from:to:cc :content-type; bh=QY58Vh9n24yQZiYNhJjhY+nlmjCOx2hm5UnpmhVU/Og=; b=TxnqLxwaZuwxZa+E6TCia040000pVePI7ZtHb1uFlFUnvupwc5aCDW7NCgJEMSydli uCJBvYIz3HIuMUR7I545EMpD2OkHtFLfZHkYBgkYRSDUlG93Rs4Fax3nMr6nfLyT+BbH 4Xutzq4V768e+Zw9ASUG3nBFVdEHO1G0o2ldw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=yBrDp48ptz6oxXtZB1tv0HRrL1N6UiYH/4iMAyUiWAhDibv0UqRTVbfjKqNvgUMeoW rLj1Hllt5dM4cLlcKAOUp1UXLPKMjUm0zYqZFn3uJJ8H5laY+YTb51vkjK4f54ojvpym jAi5RpPQMt+6FCofIwwE/ulonBxGGYFpwgbJE= MIME-Version: 1.0 Received: by 10.229.230.75 with SMTP id jl11mr1293979qcb.153.1283887924663; Tue, 07 Sep 2010 12:32:04 -0700 (PDT) Received: by 10.229.241.213 with HTTP; Tue, 7 Sep 2010 12:32:04 -0700 (PDT) Received: by 10.229.241.213 with HTTP; Tue, 7 Sep 2010 12:32:04 -0700 (PDT) In-Reply-To: <20100907164204.GA2571@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> Date: Tue, 7 Sep 2010 21:32:04 +0200 Message-ID: From: Matthias Gamsjager To: Sergey Zaharchenko Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 19:32:06 -0000 Did you try 8 stable to see if it makes any difference? On 7 Sep 2010 18:58, "Sergey Zaharchenko" wrote: Hello list, hello Pawel, I'm observing a reproducible (a matter of seconds) ZFS `deadlock'. This is on an 8.1-RELEASE amd64 quad-core system with 8G of RAM. It takes a few writes/syncs to the filesystem to make it unusable ('bonnie++ -f' works). Processes are stuck in states like zio->..., zil->..., etc. Looks like I'm not the only one: http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058362.html My problem is described more detailedly in: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058665.html Please tell me if there's anything else that you need. I can file a PR if you prefer it that way... Thanks, -- Sergey Zaharchenko From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 21:17:12 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70D911065694 for ; Tue, 7 Sep 2010 21:17:12 +0000 (UTC) (envelope-from gull@gull.us) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3016A8FC15 for ; Tue, 7 Sep 2010 21:17:11 +0000 (UTC) Received: by qwg5 with SMTP id 5so5516914qwg.13 for ; Tue, 07 Sep 2010 14:17:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.222.8 with SMTP id ie8mr5208901qcb.132.1283894231018; Tue, 07 Sep 2010 14:17:11 -0700 (PDT) Received: by 10.229.214.199 with HTTP; Tue, 7 Sep 2010 14:17:10 -0700 (PDT) X-Originating-IP: [69.91.158.216] In-Reply-To: <929991361.487274.1283561707352.JavaMail.root@erie.cs.uoguelph.ca> References: <929991361.487274.1283561707352.JavaMail.root@erie.cs.uoguelph.ca> Date: Tue, 7 Sep 2010 14:17:10 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 21:17:12 -0000 On Fri, Sep 3, 2010 at 5:55 PM, Rick Macklem wrote: >> >> Here's a capture for you. This is a Linux client (192.168.30.30) >> working against a FreeBSD 8.1-RELEASE server (192.168.30.102). If you >> need a capture using FreeBSD 9, that'll require some more setup on my >> part and I won't be able to get to it until Monday. > > Ok, The server is returning NFS4ERR_OLD_STATEID, which means that the > seqid# inside the stateid is out of date. I think the original intent > of this was to be a debugging aid for client writers, but a discussion > on the nfsv4 mailing list pretty much concluded that it was useless. > (It is definitely the case that the server doesn't care. A separate > seqd# is used to order the open/lock/close ops.) > > So, please try the attached little patch that just disables the server > check for this case for Close. (The line #s will be different than > 8.1, but all it is doing is commenting out the test for this case in > nfsrv_openupdate(), if the patch won't apply directly.) This seems to have changed the problem. Now I get the following error during checkouts: svn: In directory 'documentation/software-notes' svn: Can't copy 'documentation/software-notes/.svn/tmp/text-base/directory-server-install-notes.txt.svn-base' to 'documentation/software-notes/.svn/tmp/directory-server-install-notes.txt.tmp.tmp': Input/output error I don't see any related errors in the syslog on the client or server. This is on FreeBSD 8.1-RELEASE; I'm going to try to test with 9 later today. From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 21:23:59 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 953D910656DD for ; Tue, 7 Sep 2010 21:23:59 +0000 (UTC) (envelope-from gull@gull.us) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 584B78FC1B for ; Tue, 7 Sep 2010 21:23:59 +0000 (UTC) Received: by qyk4 with SMTP id 4so6077815qyk.13 for ; Tue, 07 Sep 2010 14:23:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.237.129 with SMTP id ko1mr5367346qcb.4.1283894638423; Tue, 07 Sep 2010 14:23:58 -0700 (PDT) Received: by 10.229.214.199 with HTTP; Tue, 7 Sep 2010 14:23:58 -0700 (PDT) X-Originating-IP: [69.91.158.216] In-Reply-To: <20100905174901.GE1900@garage.freebsd.pl> References: <4C812CEC.8080309@elischer.org> <4C814B9A.5030109@elischer.org> <20100905174901.GE1900@garage.freebsd.pl> Date: Tue, 7 Sep 2010 14:23:58 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 21:23:59 -0000 On Sun, Sep 5, 2010 at 10:49 AM, Pawel Jakub Dawidek wrot= e: > On Fri, Sep 03, 2010 at 12:25:14PM -0700, Julian Elischer wrote: >> >I just grabbed it and slotted it into my VirtualBox test network. =A0I >> >need a little help getting nfs going, though. =A0I'm getting the >> >following error: >> > >> >KLD nfslockd.ko: depends on nfs - not available or version mismatch >> >KLD nfsd.ko: depends on nfslockd - not available or version mismatch >> >mountd: NFS server is not available >> > >> >Do I need to grab the source and build a new kernel with NFS enabled? >> >> you should be able to find a kernel module to do it.. >> maybe pjd has it lying around in his build tree, or maybe it's in >> /boot/kernel on the image? > > All needed kernel modules are there, I just tested it. I wanted to turn > off NFS server by default in the image itself, but forgot... > > Doing the following should be enough: > > =A0 =A0 =A0 =A0# touch /etc/exports > =A0 =A0 =A0 =A0# echo 'nfs_server_enable=3D"YES"' >> /etc/rc.conf > =A0 =A0 =A0 =A0# echo 'mountd_enable=3D"YES"' >> /etc/rc.conf > =A0 =A0 =A0 =A0# echo 'rpcbind_enable=3D"YES"' >> /etc/rc.conf > =A0 =A0 =A0 =A0# /etc/rc.d/rpcbind start > =A0 =A0 =A0 =A0# /etc/rc.d/nfsd start > =A0 =A0 =A0 =A0# /etc/rc.d/mountd start > =A0 =A0 =A0 =A0# zfs set sharenfs=3Don tank Thanks, Pawel. That works to get the NFSv3 server going, but I need to test against the NFSv4 experimental server. When I add 'nfsv4_server_enable=3D"YES"' to /etc/rc.conf, I get the errors mentioned earlier. From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 22:21:49 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA97310656C7; Tue, 7 Sep 2010 22:21:49 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [64.81.247.49]) by mx1.freebsd.org (Postfix) with ESMTP id CB5E78FC12; Tue, 7 Sep 2010 22:21:49 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id o87MB4Tg020746; Tue, 7 Sep 2010 15:11:04 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201009072211.o87MB4Tg020746@chez.mckusick.com> To: Ivan Voras In-reply-to: Date: Tue, 07 Sep 2010 15:11:04 -0700 From: Kirk McKusick Cc: freebsd-fs@freebsd.org Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 22:21:50 -0000 > To: freebsd-fs@freebsd.org > From: Ivan Voras > Date: Tue, 07 Sep 2010 12:07:31 +0200 > Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear > before locking it in tmpfs_rename > > On 09/06/10 09:08, linimon@FreeBSD.org wrote: > > Synopsis: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename > > > > Responsible-Changed-From-To: freebsd-bugs->freebsd-fs > > Responsible-Changed-By: linimon > > Responsible-Changed-When: Mon Sep 6 07:07:55 UTC 2010 > > Responsible-Changed-Why: > > Over to maintainer(s). > > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=150143 > > Can someone look at the patch in this PR, please? > > I've tested it but don't want to commit it without someone reviewing it > (though I could commit it if the potential reviewer is busy with other > things). I have not worked with tmpfs, so may not be the best person to review this patch. But I have spent considerable time in ufs_rename, so am familiar with the problems associated with rename. In reviewing your patch, it looks like a reasonable approach to the problem. While I do not have enough context to convince myself that it will fully fix the problem, it certainly should help rename work better. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 22:49:36 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 817D510656A6; Tue, 7 Sep 2010 22:49:36 +0000 (UTC) (envelope-from ivoras@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 581DD8FC0A; Tue, 7 Sep 2010 22:49:36 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o87MnaH8059478; Tue, 7 Sep 2010 22:49:36 GMT (envelope-from ivoras@freefall.freebsd.org) Received: (from ivoras@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o87Mnac8059474; Tue, 7 Sep 2010 22:49:36 GMT (envelope-from ivoras) Date: Tue, 7 Sep 2010 22:49:36 GMT Message-Id: <201009072249.o87Mnac8059474@freefall.freebsd.org> To: gk@FreeBSD.org, ivoras@FreeBSD.org, freebsd-fs@FreeBSD.org From: ivoras@FreeBSD.org Cc: Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 22:49:36 -0000 Synopsis: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename State-Changed-From-To: open->closed State-Changed-By: ivoras State-Changed-When: Tue Sep 7 22:48:23 UTC 2010 State-Changed-Why: Committed. r212305. http://www.freebsd.org/cgi/query-pr.cgi?pr=150143 From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 22:50:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 425E610656AA for ; Tue, 7 Sep 2010 22:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 182D08FC1B for ; Tue, 7 Sep 2010 22:50:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o87Mo2Qm059557 for ; Tue, 7 Sep 2010 22:50:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o87Mo2KY059556; Tue, 7 Sep 2010 22:50:02 GMT (envelope-from gnats) Date: Tue, 7 Sep 2010 22:50:02 GMT Message-Id: <201009072250.o87Mo2KY059556@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/150143: commit references a PR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 22:50:03 -0000 The following reply was made to PR kern/150143; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/150143: commit references a PR Date: Tue, 7 Sep 2010 22:41:09 +0000 (UTC) Author: ivoras Date: Tue Sep 7 22:40:45 2010 New Revision: 212305 URL: http://svn.freebsd.org/changeset/base/212305 Log: Avoid "Entry can disappear before we lock fdvp" panic. PR: 150143 Submitted by: Gleb Kurtsou Pretty sure it won't blow up: mckusick MFC after: 2 weeks Modified: head/sys/fs/tmpfs/tmpfs_vnops.c Modified: head/sys/fs/tmpfs/tmpfs_vnops.c ============================================================================== --- head/sys/fs/tmpfs/tmpfs_vnops.c Tue Sep 7 21:28:45 2010 (r212304) +++ head/sys/fs/tmpfs/tmpfs_vnops.c Tue Sep 7 22:40:45 2010 (r212305) @@ -981,10 +981,14 @@ tmpfs_rename(struct vop_rename_args *v) fnode = VP_TO_TMPFS_NODE(fvp); de = tmpfs_dir_lookup(fdnode, fnode, fcnp); - /* Avoid manipulating '.' and '..' entries. */ + /* Entry can disappear before we lock fdvp, + * also avoid manipulating '.' and '..' entries. */ if (de == NULL) { - MPASS(fvp->v_type == VDIR); - error = EINVAL; + if ((fcnp->cn_flags & ISDOTDOT) != 0 || + (fcnp->cn_namelen == 1 && fcnp->cn_nameptr[0] == '.')) + error = EINVAL; + else + error = ENOENT; goto out_locked; } MPASS(de->td_node == fnode); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Sep 7 23:22:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98C4410656DC; Tue, 7 Sep 2010 23:22:21 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id F380C8FC13; Tue, 7 Sep 2010 23:22:19 +0000 (UTC) Received: by eyx24 with SMTP id 24so3055053eyx.13 for ; Tue, 07 Sep 2010 16:22:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject :message-id:references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=e1qz+V83viNnLpUfUwzy1calGQQmlbWcnEdJ3GwaYuM=; b=s9nOlgBv6/qPD3N5CHvbFOLmxEcq65r0P5bO2Ep2mhh4ObqRjKarZZ5XE/fz1xKhf6 43Nqn7Yc+qp033+TwvVRYtVWUHO8Zj1GnkVEDP2uJslQWZUKT5/MwXEGr7lz+rQS3wop rGEPArDpwh+SSYTDWJTeMSZLshCYQM8xFcoM8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=bgEVEEn8gggTOujMhLPtfGt3k3YckwFzBmjitXYX2xYXqxff5w69JpkKgpS2cbvuyY /uNkjoLgdwNKZmrnAAGDV5BpUje2WkF7gvSRyZRLoM1OhB9Y7d6zVVKfAvVWpj0+c80w +0mdzHhSYTg2aJgHDk3MRWnllvh9rPKwCn/x4= Received: by 10.213.101.10 with SMTP id a10mr11229ebo.85.1283900679619; Tue, 07 Sep 2010 16:04:39 -0700 (PDT) Received: from localhost ([212.98.186.134]) by mx.google.com with ESMTPS id u9sm10768183eeh.17.2010.09.07.16.04.38 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 07 Sep 2010 16:04:38 -0700 (PDT) Date: Wed, 8 Sep 2010 02:04:33 +0300 From: Gleb Kurtsou To: Kirk McKusick Message-ID: <20100907230433.GA3938@tops> References: <201009072211.o87MB4Tg020746@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <201009072211.o87MB4Tg020746@chez.mckusick.com> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Sep 2010 23:22:21 -0000 On (07/09/2010 15:11), Kirk McKusick wrote: > > To: freebsd-fs@freebsd.org > > From: Ivan Voras > > Date: Tue, 07 Sep 2010 12:07:31 +0200 > > Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear > > before locking it in tmpfs_rename > > > > On 09/06/10 09:08, linimon@FreeBSD.org wrote: > > > Synopsis: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename > > > > > > Responsible-Changed-From-To: freebsd-bugs->freebsd-fs > > > Responsible-Changed-By: linimon > > > Responsible-Changed-When: Mon Sep 6 07:07:55 UTC 2010 > > > Responsible-Changed-Why: > > > Over to maintainer(s). > > > > > > http://www.freebsd.org/cgi/query-pr.cgi?pr=150143 > > > > Can someone look at the patch in this PR, please? > > > > I've tested it but don't want to commit it without someone reviewing it > > (though I could commit it if the potential reviewer is busy with other > > things). > > I have not worked with tmpfs, so may not be the best person to review > this patch. But I have spent considerable time in ufs_rename, so am > familiar with the problems associated with rename. > > In reviewing your patch, it looks like a reasonable approach to the > problem. While I do not have enough context to convince myself that > it will fully fix the problem, it certainly should help rename work > better. Hello Kirk, I was working on improving namecache during this summer, and I have to admit rename with the biggest problem of all, and it still remains. There are several common approaches taken by filesystems. UFS locks all vnodes involved in rename, unlocking, trying to lock vnodes and check for races, tmpfs does something similar (although vnode locking is incorrect, I'm going to fix it a bit later). Some others (like ext2fs and msdosfs if I'm not mistaken) keep locking at minimum, it seems to work, but honestly I don't see why it can't race. ZFS is somewhat unique in this respect. It uses name locking, keeps per directory table of locked file names, i.e. names that can't change while in table. So that destination file won't be added during rename, source file can't disappear, etc. What do you think about name locking approach taken by ZFS? Are there any drawbacks you are aware of? I was thinking of trying to unify rename locking, either make UFS approach standard, i.e. lock all vnodes outside of rename or use name locking similar to ZFS. UFS way may not fit well into existing VOP API (extra vnode lookups to check for races) besides vnode locking order remains an important issue. ZFS style locks may be interesting in a way that they would allow to reduce scope of vnode locks, especially considering merging with ongoing work on rangelocks (just a guess). Thanks, Gleb. > > Kirk McKusick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 01:27:31 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78D6D106566B for ; Wed, 8 Sep 2010 01:27:31 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id F097F8FC12 for ; Wed, 8 Sep 2010 01:27:30 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAEKDhkyDaFvO/2dsb2JhbACDGZ5ir1GRd4Eigyd0BIR4hSA X-IronPort-AV: E=Sophos;i="4.56,331,1280721600"; d="scan'208";a="93112937" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 07 Sep 2010 21:27:29 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id BDB9AB3EA3; Tue, 7 Sep 2010 21:27:29 -0400 (EDT) Date: Tue, 7 Sep 2010 21:27:29 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <3705884.579377.1283909249721.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [24.65.230.102] X-Mailer: Zimbra 6.0.7_GA_2476.RHEL4 (ZimbraWebClient - SAF3 (Mac)/6.0.7_GA_2473.RHEL4_64) Cc: freebsd-fs@freebsd.org Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 01:27:31 -0000 > On Fri, Sep 3, 2010 at 5:55 PM, Rick Macklem > wrote: > >> > >> Here's a capture for you. This is a Linux client (192.168.30.30) > >> working against a FreeBSD 8.1-RELEASE server (192.168.30.102). If > >> you > >> need a capture using FreeBSD 9, that'll require some more setup on > >> my > >> part and I won't be able to get to it until Monday. > > > > Ok, The server is returning NFS4ERR_OLD_STATEID, which means that > > the > > seqid# inside the stateid is out of date. I think the original > > intent > > of this was to be a debugging aid for client writers, but a > > discussion > > on the nfsv4 mailing list pretty much concluded that it was useless. > > (It is definitely the case that the server doesn't care. A separate > > seqd# is used to order the open/lock/close ops.) > > > > So, please try the attached little patch that just disables the > > server > > check for this case for Close. (The line #s will be different than > > 8.1, but all it is doing is commenting out the test for this case in > > nfsrv_openupdate(), if the patch won't apply directly.) > > This seems to have changed the problem. Now I get the following error > during checkouts: > > svn: In directory 'documentation/software-notes' > svn: Can't copy > 'documentation/software-notes/.svn/tmp/text-base/directory-server-install-notes.txt.svn-base' > to > 'documentation/software-notes/.svn/tmp/directory-server-install-notes.txt.tmp.tmp': > Input/output error > > I don't see any related errors in the syslog on the client or server. > > This is on FreeBSD 8.1-RELEASE; I'm going to try to test with 9 later > today. > Ok, if you feel like emailing me another packet capture, I can look and see where the EIO is being returned. And good luck with testing the newer ZFS, rick From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 06:52:27 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72A9F1065697 for ; Wed, 8 Sep 2010 06:52:27 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from forward11.mail.yandex.net (forward11.mail.yandex.net [95.108.130.93]) by mx1.freebsd.org (Postfix) with ESMTP id 1D27F8FC1D for ; Wed, 8 Sep 2010 06:52:27 +0000 (UTC) Received: from smtp14.mail.yandex.net (smtp14.mail.yandex.net [95.108.131.192]) by forward11.mail.yandex.net (Yandex) with ESMTP id 7B3083ED0349; Wed, 8 Sep 2010 10:52:25 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1283928745; bh=NLI2XmjbEs3Kr7/ShHZFAem2WRGaPuJ2e932TK/8/K4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=j9cO50bbZa1iHDBSzEBL0mqgTW/69GLk/JEcI1pDtphdTJZxldTPdewGu4QR+3bNC /BnKBfJzBQVdaiq1TWTJZPd3SsPi7cHUMMx1yWDzTT5hPhSK3qDvoMijubxuQR7uHc Y6BfiWgNDa7E498hL9TNm1Q+podHrBCDGL7qjewU= Received: from nautilus (unknown [178.155.61.42]) by smtp14.mail.yandex.net (Yandex) with ESMTPA id 4519E19B8075; Wed, 8 Sep 2010 10:52:25 +0400 (MSD) Received: by nautilus (Postfix, from userid 1001) id 86D141DD43D; Wed, 8 Sep 2010 10:52:22 +0400 (MSD) Date: Wed, 8 Sep 2010 10:52:22 +0400 From: Sergey Zaharchenko To: Matthias Gamsjager Message-ID: <20100908065222.GA2522@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5mCyUwZo2JvN/JJP" Content-Disposition: inline In-Reply-To: X-Listening-To: Silence User-Agent: Mutt/1.5.20 (2009-06-14) X-Yandex-TimeMark: 1283928745 X-Yandex-Spam: 1 X-Yandex-Front: smtp14.mail.yandex.net Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 06:52:27 -0000 --5mCyUwZo2JvN/JJP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello Matthias! Tue, Sep 07, 2010 at 09:32:04PM +0200 you wrote: > Did you try 8 stable to see if it makes any difference? This server is to be going into production soon, so I'm not in the best position to try things. I've however tried disabling prefetch and ZIL. This helped ZFS pass bonnie++, but it deadlocked later on a repeated make -j 4 buildworld. top shows: CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 194M Active, 32M Inact, 1844M Wired, 48M Cache, 523M Buf, 5799M Free 19285 root 1 76 0 13060K 8876K zfs 2 0:00 0.00% cc1 19141 root 1 76 0 13060K 8876K tx->tx 1 0:00 0.00% cc1 19745 root 1 76 0 13052K 9116K tx->tx 1 0:00 0.00% cc1 1154 root 1 44 0 6896K 980K tx->tx 0 0:00 0.00% syslogd 2136 root 1 44 0 5808K 812K zio->i 2 0:00 0.00% newsyslog etc. This, apart from anything else, may indicate that disabling the features only masks the problems with ZFS, but doesn't eliminate them completely. Thanks for your input, -- Sergey Zaharchenko --5mCyUwZo2JvN/JJP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyHMqUACgkQwo7hT/9lVdyjfQCfdSv30OU9GCmyYgBWJapk1h7c ei8AniEXJaQUM+lw01PmQR38WSbtOoEH =iAd7 -----END PGP SIGNATURE----- --5mCyUwZo2JvN/JJP-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 07:36:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9A8110656DF for ; Wed, 8 Sep 2010 07:36:29 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 370CA8FC12 for ; Wed, 8 Sep 2010 07:36:28 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA07020; Wed, 08 Sep 2010 10:19:49 +0300 (EEST) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OtEwf-0000PF-6j; Wed, 08 Sep 2010 10:19:49 +0300 Message-ID: <4C873914.40404@freebsd.org> Date: Wed, 08 Sep 2010 10:19:48 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Wiktor Niesiobedzki , Pawel Jakub Dawidek , Konstantin Belousov References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C85E91E.1010602@icyb.net.ua> In-Reply-To: <4C85E91E.1010602@icyb.net.ua> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 07:36:30 -0000 on 07/09/2010 10:26 Andriy Gapon said the following: > Interesting. I briefly looked at the code in mappedread(), zfs_vnops.c, and I > have a VM question. > Shouldn't we mark the corresponding page bits as valid after reading data into > the page? > I specifically speak of the block that starts with the following line: > } else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) { > I am taking mdstart_swap as an example and it does m->valid = VM_PAGE_BITS_ALL. > I've chatted with and conclusion seems to be that vm_page_set_validclean() call should be added at the end of the block. Perhaps, something like this: --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c @@ -500,6 +500,7 @@ again: sched_unpin(); } VM_OBJECT_LOCK(obj); + vm_page_set_validclean(m, off, bytes); vm_page_wakeup(m); if (error == 0) uio->uio_resid -= bytes; BTW, I think that 'off' variable can be made of 'int' type, as it is an offset within a page; no need for int64. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 08:07:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1C2E1065672 for ; Wed, 8 Sep 2010 08:07:58 +0000 (UTC) (envelope-from mgamsjager@gmail.com) Received: from mail-qy0-f175.google.com (mail-qy0-f175.google.com [209.85.216.175]) by mx1.freebsd.org (Postfix) with ESMTP id 535378FC1D for ; Wed, 8 Sep 2010 08:07:57 +0000 (UTC) Received: by qyk31 with SMTP id 31so4526175qyk.13 for ; Wed, 08 Sep 2010 01:07:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=yjNey5BcP21kuPiJFHRvPsZaqX8yfQoM4UEwqUuOkP0=; b=hZ7pDFtHczbImx3ZtPFwPcAdnQBzO1ryR8BB24wVwo85T0b9NIIkF7V/yiJunZ54QY aB0egX4o8lPfkTUm8/E+0hCGTPQfSvcdjkxORvtjcmkJw/rxxIjFAo4ubB08eZGYCxa8 qtz3w9qv87EkspKYHLQnvUTuEhPXNQIEmbi0w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; b=x3lGCW8pfim4RdKs5U+lK3MywUGOBuTyOtnZmK21dd+F6Nlr4Wroy7qP0HTX3Ok/ZF No4T9wKux7Gwal1YznE0YKbcUtpBmrJPGarapKUlmDwZSv57tQuuZo3c/f3UgwrEfK30 0RST79ZEyR1HGsvcGXNKZPphU5N2SGZoiKRmM= Received: by 10.229.2.35 with SMTP id 35mr3958503qch.271.1283933277318; Wed, 08 Sep 2010 01:07:57 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.241.213 with HTTP; Wed, 8 Sep 2010 01:07:27 -0700 (PDT) In-Reply-To: <20100908065222.GA2522@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> From: Matthias Gamsjager Date: Wed, 8 Sep 2010 10:07:27 +0200 Message-ID: To: Sergey Zaharchenko Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 08:07:58 -0000 On Wed, Sep 8, 2010 at 8:52 AM, Sergey Zaharchenko wrote: > Hello Matthias! > > Tue, Sep 07, 2010 at 09:32:04PM +0200 you wrote: > >> Did you try 8 stable to see if it makes any difference? > > This server is to be going into production soon, so I'm not in the best > position to try things. I've however tried disabling prefetch and ZIL. > This helped ZFS pass bonnie++, but it deadlocked later on a repeated > make -j 4 buildworld. top shows: > > CPU: =9A0.0% user, =9A0.0% nice, =9A0.0% system, =9A0.0% interrupt, =9A10= 0% idle > Mem: 194M Active, 32M Inact, 1844M Wired, 48M Cache, 523M Buf, 5799M Free > > 19285 root =9A =9A =9A =9A =9A =9A 1 =9A76 =9A =9A0 13060K =9A8876K zfs = =9A =9A 2 =9A 0:00 =9A0.00% cc1 > 19141 root =9A =9A =9A =9A =9A =9A 1 =9A76 =9A =9A0 13060K =9A8876K tx->t= x =9A1 =9A 0:00 =9A0.00% cc1 > 19745 root =9A =9A =9A =9A =9A =9A 1 =9A76 =9A =9A0 13052K =9A9116K tx->t= x =9A1 =9A 0:00 =9A0.00% cc1 > =9A1154 root =9A =9A =9A =9A =9A =9A 1 =9A44 =9A =9A0 =9A6896K =9A 980K t= x->tx =9A0 =9A 0:00 =9A0.00% syslogd > =9A2136 root =9A =9A =9A =9A =9A =9A 1 =9A44 =9A =9A0 =9A5808K =9A 812K z= io->i =9A2 =9A 0:00 =9A0.00% newsyslog > > etc. > > This, apart from anything else, may indicate that disabling the features > only masks the problems with ZFS, but doesn't eliminate them completely. > > Thanks for your input, > > -- > Sergey Zaharchenko > Could you give more information about the hardware you are using (disk controller in particular) and /boot/loader.conf output. From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 08:49:04 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78EAC10656C2; Wed, 8 Sep 2010 08:49:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id C67578FC28; Wed, 8 Sep 2010 08:49:03 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id o888muvF041050 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 8 Sep 2010 11:48:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id o888mugG005788; Wed, 8 Sep 2010 11:48:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id o888mtZB005787; Wed, 8 Sep 2010 11:48:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 8 Sep 2010 11:48:55 +0300 From: Kostik Belousov To: Andriy Gapon Message-ID: <20100908084855.GF2465@deviant.kiev.zoral.com.ua> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C85E91E.1010602@icyb.net.ua> <4C873914.40404@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="E7i4zwmWs5DOuDSH" Content-Disposition: inline In-Reply-To: <4C873914.40404@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_20, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 08:49:04 -0000 --E7i4zwmWs5DOuDSH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 08, 2010 at 10:19:48AM +0300, Andriy Gapon wrote: > on 07/09/2010 10:26 Andriy Gapon said the following: > > Interesting. I briefly looked at the code in mappedread(), zfs_vnops.c= , and I > > have a VM question. > > Shouldn't we mark the corresponding page bits as valid after reading da= ta into > > the page? > > I specifically speak of the block that starts with the following line: > > } else if (m !=3D NULL && uio->uio_segflg =3D=3D UIO_NOCOPY) { > > I am taking mdstart_swap as an example and it does m->valid =3D VM_PAGE= _BITS_ALL. > >=20 >=20 > I've chatted with and conclusion seems to be that vm_page_set_validclean(= ) call > should be added at the end of the block. >=20 > Perhaps, something like this: >=20 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > @@ -500,6 +500,7 @@ again: > sched_unpin(); > } > VM_OBJECT_LOCK(obj); > + vm_page_set_validclean(m, off, bytes); Only if error =3D=3D 0, perhaps ? > vm_page_wakeup(m); > if (error =3D=3D 0) > uio->uio_resid -=3D bytes; >=20 >=20 > BTW, I think that 'off' variable can be made of 'int' type, as it is an o= ffset > within a page; no need for int64. >=20 > --=20 > Andriy Gapon --E7i4zwmWs5DOuDSH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkyHTfcACgkQC3+MBN1Mb4gIeACgm4qHkuJVwqfQNcsYuzgpSkcQ tvoAoO0JAwtgqTdovoyGUmYgX6UDJaPO =GcfI -----END PGP SIGNATURE----- --E7i4zwmWs5DOuDSH-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 08:49:10 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90EF01065674 for ; Wed, 8 Sep 2010 08:49:10 +0000 (UTC) (envelope-from spork@bway.net) Received: from xena.bway.net (xena.bway.net [216.220.96.26]) by mx1.freebsd.org (Postfix) with ESMTP id 34B2F8FC15 for ; Wed, 8 Sep 2010 08:49:10 +0000 (UTC) Received: (qmail 26881 invoked by uid 0); 8 Sep 2010 08:27:32 -0000 Received: from unknown (HELO ?10.3.2.41?) (spork@96.57.144.66) by smtp.bway.net with (DHE-RSA-AES256-SHA encrypted) SMTP; 8 Sep 2010 08:27:32 -0000 Date: Wed, 8 Sep 2010 04:22:19 -0400 (EDT) From: Charles Sprickman X-X-Sender: spork@hotlap.local To: freebsd-fs@freebsd.org, freebsd-scsi@freebsd.org Message-ID: User-Agent: Alpine 2.00 (OSX 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: Subject: 7.3 or 7-STABLE? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 08:49:10 -0000 Hello, Quick question... I'm still having issues with an amd64 box and corruption on a largish (1.3T) filesystem. http://www.listware.net/201007/freebsd-fs/13111-72-ufs2-corruption.html - sorry for the crappy archive I've let the RAID controller run a few full verifies, and that comes back clean, the hardware is fine (pre-production it went through insanely long memtest runs, a week of build/installworld, etc.). Only after putting some real mail load on it did this start happening. For all I know, it could have been one bad panic that mangled things and fsck simply cannot fix it properly. That said, before even taking it down for a long rsync off to another host, newfs, then rsync back, I'd like to upgrade to 7.3 or 7-stable. The reason for choosing both -fs and -scsi is that I'd like some feedback from both groups on the two things I'm most suspicious of - UFS itself and the mfi driver. If neither has had any interesting changes in -stable, I'll go to 7.3 (at 7.2 now). If there are, then I might give -stable a shot... If the upgrade doesn't help, then I'll spend the time moving the data off, newfs'ing /spool and bringing the data back. If that doesn't work, then it's 8.1 (unless someone has a really compelling argument on making that step 1). If this were my box, I'd be putting more time into this, but the client is having numerous issues with cashflow, and I'm not looking to put out more hours than I may get paid for. Sad but true, and perhaps a lame excuse for sloppy administration... Thanks, Charles From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 08:52:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1FD9C10656DD; Wed, 8 Sep 2010 08:52:42 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5239A8FC21; Wed, 8 Sep 2010 08:52:40 +0000 (UTC) Received: by ewy4 with SMTP id 4so3237731ewy.13 for ; Wed, 08 Sep 2010 01:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject :message-id:references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=7QmUhPQ/AbClF2mkeCZaj80aMt4dGHA32ded1kVslNM=; b=ETs1166WHgXlV1qnfKBRN1h4FXzMOvA9MiXS2IU/U2gtuXaKGLY0kCTSzgJMQDM1UK AGAyjSdNmQDuiBGFmfDGtX8iiJiG+V44ZFQhBP69Z2TDcEhwNy+C1QRmexWdCCVJAsDJ jtBtG09+/WZuuGKrRZLg6w5Tct28gbGC3ggv0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=W4s4+4SyZW9/78OPmg11qMXSzME8tnJdeXRVu93RUGpZ1QrZeLWNJmuOk3ZmjFt4GT oTjEaHz2kJAKCEQuuk94uoAw3LFoLeLoVCg8gAx7QyPdbxc9IfdVuIxBLhEZHH4I7qlb ok8cwU4DP0wkGDyLDmU+jWtqdH8YDgHoKuLxE= Received: by 10.213.2.207 with SMTP id 15mr617222ebk.9.1283935960099; Wed, 08 Sep 2010 01:52:40 -0700 (PDT) Received: from localhost ([212.98.186.134]) by mx.google.com with ESMTPS id v8sm11526816eeh.2.2010.09.08.01.52.38 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 08 Sep 2010 01:52:39 -0700 (PDT) Date: Wed, 8 Sep 2010 11:52:33 +0300 From: Gleb Kurtsou To: Andriy Gapon Message-ID: <20100908085233.GA1850@tops> References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C85E91E.1010602@icyb.net.ua> <4C873914.40404@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <4C873914.40404@freebsd.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek , Konstantin Belousov Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 08:52:42 -0000 On (08/09/2010 10:19), Andriy Gapon wrote: > on 07/09/2010 10:26 Andriy Gapon said the following: > > Interesting. I briefly looked at the code in mappedread(), zfs_vnops.c, and I > > have a VM question. > > Shouldn't we mark the corresponding page bits as valid after reading data into > > the page? > > I specifically speak of the block that starts with the following line: > > } else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) { > > I am taking mdstart_swap as an example and it does m->valid = VM_PAGE_BITS_ALL. > > > > I've chatted with and conclusion seems to be that vm_page_set_validclean() call > should be added at the end of the block. > > Perhaps, something like this: > > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c > @@ -500,6 +500,7 @@ again: > sched_unpin(); > } > VM_OBJECT_LOCK(obj); > + vm_page_set_validclean(m, off, bytes); > vm_page_wakeup(m); > if (error == 0) > uio->uio_resid -= bytes; > > > BTW, I think that 'off' variable can be made of 'int' type, as it is an offset > within a page; no need for int64. I had something similar to this patch but I've backed it out for unknown reason, can't find any details. Did you try running fsx with it? You might also be interested in recent patches to mapped read/write in tmpfs by Alan Cox the code is very similar to ZFS. Thanks, Gleb. > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 08:53:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D3EE10656E0; Wed, 8 Sep 2010 08:53:26 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 037D18FC13; Wed, 8 Sep 2010 08:53:24 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA08721; Wed, 08 Sep 2010 11:53:21 +0300 (EEST) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OtGPA-0000WR-Ug; Wed, 08 Sep 2010 11:53:20 +0300 Message-ID: <4C874F00.3050605@freebsd.org> Date: Wed, 08 Sep 2010 11:53:20 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Kostik Belousov References: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> <4C85E91E.1010602@icyb.net.ua> <4C873914.40404@freebsd.org> <20100908084855.GF2465@deviant.kiev.zoral.com.ua> In-Reply-To: <20100908084855.GF2465@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 08:53:26 -0000 on 08/09/2010 11:48 Kostik Belousov said the following: > On Wed, Sep 08, 2010 at 10:19:48AM +0300, Andriy Gapon wrote: >> on 07/09/2010 10:26 Andriy Gapon said the following: >>> Interesting. I briefly looked at the code in mappedread(), zfs_vnops.c, and I >>> have a VM question. >>> Shouldn't we mark the corresponding page bits as valid after reading data into >>> the page? >>> I specifically speak of the block that starts with the following line: >>> } else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) { >>> I am taking mdstart_swap as an example and it does m->valid = VM_PAGE_BITS_ALL. >>> >> >> I've chatted with and conclusion seems to be that vm_page_set_validclean() call >> should be added at the end of the block. >> >> Perhaps, something like this: >> >> --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >> +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c >> @@ -500,6 +500,7 @@ again: >> sched_unpin(); >> } >> VM_OBJECT_LOCK(obj); >> + vm_page_set_validclean(m, off, bytes); > Only if error == 0, perhaps ? Yes, I agree, thanks! -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 09:08:50 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6934710656E0 for ; Wed, 8 Sep 2010 09:08:50 +0000 (UTC) (envelope-from roberto@keltia.freenix.fr) Received: from keltia.net (unknown [IPv6:2a01:240:fe5c::41]) by mx1.freebsd.org (Postfix) with ESMTP id 1A9A58FC0A for ; Wed, 8 Sep 2010 09:08:50 +0000 (UTC) Received: from roberto-al.eurocontrol.fr (aran.keltia.net [88.191.250.24]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: roberto) by keltia.net (Postfix/TLS) with ESMTPSA id 874FC9558 for ; Wed, 8 Sep 2010 11:08:47 +0200 (CEST) Date: Wed, 8 Sep 2010 11:08:42 +0200 From: Ollivier Robert To: freebsd-fs@freebsd.org Message-ID: <20100908090842.GA87388@roberto-al.eurocontrol.fr> References: <20100831215915.GE1932@garage.freebsd.pl> <20100902214823.GB2542@garage.freebsd.pl> <4C811954.5040602@3mail4.co.uk> <20100903170241.GF1780@garage.freebsd.pl> <4C8232C6.9020108@DataIX.net> <4C8294B7.3010507@bellanet.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C8294B7.3010507@bellanet.org> X-Operating-System: MacOS X / Macbook Pro - FreeBSD 7.2 / Dell D820 SMP User-Agent: Mutt/1.5.20 (2009-06-14) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (keltia.net); Wed, 08 Sep 2010 11:08:47 +0200 (CEST) Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 09:08:50 -0000 According to Graham Todd: > Thanks to you and other devs for all your careful work integrating ZFS > into FreeBSD. It seems amazing that the patch for v28 *almost* even > applies to 8-STABLE. Do you think you could generate a 8-STABLE patch or at least tell us where it is failling please? -- Ollivier ROBERT -=- FreeBSD: The Power to Serve! -=- roberto@keltia.net In memoriam to Ondine, our 2nd child: http://ondine.keltia.net/ From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 10:33:43 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 453F41065696 for ; Wed, 8 Sep 2010 10:33:43 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from forward15.mail.yandex.net (forward15.mail.yandex.net [95.108.130.119]) by mx1.freebsd.org (Postfix) with ESMTP id E2C1A8FC08 for ; Wed, 8 Sep 2010 10:33:42 +0000 (UTC) Received: from smtp13.mail.yandex.net (smtp13.mail.yandex.net [95.108.130.68]) by forward15.mail.yandex.net (Yandex) with ESMTP id 24C4C44591CE; Wed, 8 Sep 2010 14:33:41 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1283942021; bh=Rpzbyp0xhhzNp5u3kwqQcewQBn+3fknoQxThhr5MWQQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=WE0Y3LHiJKUNXCqiYK0ly6RRT8zx+pJ1JkDwXKyyJ+zc3PlPDCIGgtchhe+pyhlP4 5QVvtyftIX16f1XUVID9Y4Qm1e3ajjg2tx9QqfavC2lmbkpUv/dfX8l2UxqoH3TIoD ujDjMZHs80qonD77v0XKKifSZ4h3w/pksJGjzC+A= Received: from nautilus (unknown [178.155.61.42]) by smtp13.mail.yandex.net (Yandex) with ESMTPA id E6D7E41580A7; Wed, 8 Sep 2010 14:33:40 +0400 (MSD) Received: by nautilus (Postfix, from userid 1001) id 672A01DD43D; Wed, 8 Sep 2010 14:33:38 +0400 (MSD) Date: Wed, 8 Sep 2010 14:33:38 +0400 From: Sergey Zaharchenko To: Matthias Gamsjager Message-ID: <20100908103338.GA5091@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VbJkn9YxBvnuCH5J" Content-Disposition: inline In-Reply-To: X-Listening-To: Silence User-Agent: Mutt/1.5.20 (2009-06-14) X-Yandex-TimeMark: 1283942021 X-Yandex-Spam: 1 X-Yandex-Front: smtp13.mail.yandex.net Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 10:33:43 -0000 --VbJkn9YxBvnuCH5J Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello Matthias! Wed, Sep 08, 2010 at 10:07:27AM +0200 you wrote: > Could you give more information about the hardware you are using (disk > controller in particular) and /boot/loader.conf output. The controller is a RocketRAID 3540 SATA Controller in single-disk mode: hptiop0: adapter at PCI 5:0:0, IRQ 16 hptiop0: mem 0xdd800000-0xddffffff irq 16= at device 0.0 on pci5 hptiop0: 0 RocketRAID 3xxx/4xxx controller driver v1.3 (010208) hptiop0: [GIANT-LOCKED] hptiop0: [ITHREAD] The disks are 8 identical disks like: da0 at hptiop0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device=20 da0: 1907348MB (3906250000 512 byte sectors: 255H 63S/T 243152C) CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2822.53-MHz K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0xf41 Family =3D f Model =3D 4 Stepp= ing =3D 1 Features=3D0xbfebfbff Features2=3D0x641d AMD Features=3D0x20100800 TSC: P-state invariant real memory =3D 8589934592 (8192 MB) avail memory =3D 8259899392 (7877 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) x 2 HTT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP/HT): APIC ID: 1 cpu2 (AP): APIC ID: 6 cpu3 (AP/HT): APIC ID: 7 FWIW most of it is in my original email... /bot/loader.conf has: mac_bsdextended_load=3D"YES" vfs.zfs.zil_disable=3D1 vfs.zfs.prefetch_disable=3D1 No, mac_bsdextended has nothing to do with it. Thanks, --=20 Sergey Zaharchenko --VbJkn9YxBvnuCH5J Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyHZoEACgkQwo7hT/9lVdx5LwCfVdBN9vZbOUJfH3Ga6rAWe9t1 HVEAn38jwbsyEVAEdP25r/RTQSAi4WXv =xXl6 -----END PGP SIGNATURE----- --VbJkn9YxBvnuCH5J-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 10:42:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97B2010656D2 for ; Wed, 8 Sep 2010 10:42:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta13.emeryville.ca.mail.comcast.net (qmta13.emeryville.ca.mail.comcast.net [76.96.27.243]) by mx1.freebsd.org (Postfix) with ESMTP id 7B9B58FC0C for ; Wed, 8 Sep 2010 10:42:01 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta13.emeryville.ca.mail.comcast.net with comcast id 4AY31f0021vN32cADAi1Um; Wed, 08 Sep 2010 10:42:01 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta22.emeryville.ca.mail.comcast.net with comcast id 4Ai01f0043LrwQ28iAi0J9; Wed, 08 Sep 2010 10:42:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 084919B423; Wed, 8 Sep 2010 03:42:00 -0700 (PDT) Date: Wed, 8 Sep 2010 03:42:00 -0700 From: Jeremy Chadwick To: Sergey Zaharchenko Message-ID: <20100908104200.GA36566@icarus.home.lan> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> <20100908103338.GA5091@nautilus.vmks.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100908103338.GA5091@nautilus.vmks.ru> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 10:42:01 -0000 On Wed, Sep 08, 2010 at 02:33:38PM +0400, Sergey Zaharchenko wrote: > Wed, Sep 08, 2010 at 10:07:27AM +0200 you wrote: > > Could you give more information about the hardware you are using (disk > > controller in particular) and /boot/loader.conf output. > > The controller is a RocketRAID 3540 SATA Controller in single-disk mode: > > hptiop0: adapter at PCI 5:0:0, IRQ 16 > hptiop0: mem 0xdd800000-0xddffffff irq 16 at device 0.0 on pci5 > hptiop0: 0 RocketRAID 3xxx/4xxx controller driver v1.3 (010208) > hptiop0: [GIANT-LOCKED] > hptiop0: [ITHREAD] If you get rid of hptiop(4) (meaning move the disks to the on-board controller, which presumably has 4 or 6 ports on it?), does the situation improve? I realise this isn't an effective long-term solution, but it would help rule out hptiop(4) being the source of the problem. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 13:12:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C60C110656CE for ; Wed, 8 Sep 2010 13:12:03 +0000 (UTC) (envelope-from gtodd@bellanet.org) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 863F78FC1A for ; Wed, 8 Sep 2010 13:12:03 +0000 (UTC) Received: by yxn35 with SMTP id 35so7945yxn.13 for ; Wed, 08 Sep 2010 06:12:02 -0700 (PDT) Received: by 10.100.205.16 with SMTP id c16mr1614308ang.57.1283951522679; Wed, 08 Sep 2010 06:12:02 -0700 (PDT) Received: from wawanesa.iciti.ca (CPE0080c8f208a5-CM001371173cf8.cpe.net.cable.rogers.com [99.246.61.82]) by mx.google.com with ESMTPS id i30sm13502816anh.29.2010.09.08.06.11.58 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 08 Sep 2010 06:11:59 -0700 (PDT) Message-ID: <4C878B9A.2000909@bellanet.org> Date: Wed, 08 Sep 2010 09:11:54 -0400 From: Graham Todd User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.10) Gecko/20100721 Thunderbird/3.0.5 MIME-Version: 1.0 To: Ollivier Robert , freebsd-fs@freebsd.org References: <20100831215915.GE1932@garage.freebsd.pl> <20100902214823.GB2542@garage.freebsd.pl> <4C811954.5040602@3mail4.co.uk> <20100903170241.GF1780@garage.freebsd.pl> <4C8232C6.9020108@DataIX.net> <4C8294B7.3010507@bellanet.org> <20100908090842.GA87388@roberto-al.eurocontrol.fr> In-Reply-To: <20100908090842.GA87388@roberto-al.eurocontrol.fr> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 13:12:03 -0000 On 09/08/10 05:08, Ollivier Robert wrote: > According to Graham Todd: >> Thanks to you and other devs for all your careful work integrating ZFS >> into FreeBSD. It seems amazing that the patch for v28 *almost* even >> applies to 8-STABLE. > > Do you think you could generate a 8-STABLE patch or at least tell us where > it is failling please? thanks to jhell,v - who has already has attempted this and reported back :) see: http://bit.ly/d20tML and lists.freebsd.org/pipermail/freebsd-current/2010-September/019576.html "only 14 of the 293 files that were changed had rejects against a stable/8 that was patched to ZFSv15 + metaslab_v2 + abe_stat_rrwlock" I was unable to quite that far - I think because I applied the earlier patches in the wrong order. But I am impressed that adding a patch from CURRENT to go from ZFSv15 to ZFSv28 on the STABLE branch was that close to applying. From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 15:36:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5747910656D0 for ; Wed, 8 Sep 2010 15:36:21 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 013FA8FC13 for ; Wed, 8 Sep 2010 15:36:20 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 6CC1145C9C; Wed, 8 Sep 2010 17:36:18 +0200 (CEST) Received: from localhost (pdawidek.whl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9F47645683; Wed, 8 Sep 2010 17:36:13 +0200 (CEST) Date: Wed, 8 Sep 2010 17:36:02 +0200 From: Pawel Jakub Dawidek To: Sergey Zaharchenko Message-ID: <20100908153602.GB1751@garage.freebsd.pl> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> <20100908103338.GA5091@nautilus.vmks.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="L6iaP+gRLNZHKoI4" Content-Disposition: inline In-Reply-To: <20100908103338.GA5091@nautilus.vmks.ru> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 15:36:21 -0000 --L6iaP+gRLNZHKoI4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 08, 2010 at 02:33:38PM +0400, Sergey Zaharchenko wrote: > real memory =3D 8589934592 (8192 MB) > avail memory =3D 8259899392 (7877 MB) [...] > /bot/loader.conf has: >=20 > mac_bsdextended_load=3D"YES" > vfs.zfs.zil_disable=3D1 > vfs.zfs.prefetch_disable=3D1 No vm.kmem_size? Try adding: vm.kmem_size=3D"8G" And try leaving ZIL and prefetch alone (eventually turn prefetch off, but keep ZIL on). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --L6iaP+gRLNZHKoI4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyHrWEACgkQForvXbEpPzS+wQCeOidSxIqyaOKk0bupwpeCGaXD uZIAoMqQRYn4AbPOfFDDKjpvbYF00BST =+jj/ -----END PGP SIGNATURE----- --L6iaP+gRLNZHKoI4-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 16:49:09 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1981510656CB for ; Wed, 8 Sep 2010 16:49:09 +0000 (UTC) (envelope-from appdebgr@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id CBD3C8FC13 for ; Wed, 8 Sep 2010 16:49:08 +0000 (UTC) Received: by ywt2 with SMTP id 2so163616ywt.13 for ; Wed, 08 Sep 2010 09:49:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=MwNAKUBC8C4K7DU2rQA8ieAthpPh4MhcQt8OA/zkx9Y=; b=X8kXZGcjuRC2r/Etnupj6QFqAC/DdItMadJwvYK3UnTafMbNWM/3CqjqPQ2DKbDO7B eboX4EIIJKIwGBT5z8pCdZsYxl2xSbS+u433mVt3MQndGbQNM8s4QQ2+F7GAgtup+pea PkpnI6l6nu9zujvCueQmWnddvxeuXoAPBLzME= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=nHdj8CpfPUVBgEHdT7wGF6Z+OtzI52w57B/3GX7xuP2CQYc83i8rfkYUyqdvvOiuWb WOGJVsg8srSVVST2fxc6Siw+P2wSdt+0yYmwJKmN7ctQNjuF6nMk/Tx1ubQIR41eox/U krBGsfO4i0+SFJqEe0b8W2VqnqTyXbBIeq4wA= MIME-Version: 1.0 Received: by 10.229.212.11 with SMTP id gq11mr195619qcb.78.1283962781003; Wed, 08 Sep 2010 09:19:41 -0700 (PDT) Received: by 10.229.189.81 with HTTP; Wed, 8 Sep 2010 09:19:40 -0700 (PDT) Date: Wed, 8 Sep 2010 19:19:40 +0300 Message-ID: From: App Deb To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 16:49:09 -0000 Reading the wiki guides they mention the official way to add swap volumes on ZFS (set org.freebsd:swap=on etc..), So I thought that it should work now, But I just got a panic in 8.1-RELEASE after a heavy memory situation that touched swap for the first time. It is a hassle to add gmirror volumes for swap on full zfs systems, is there any workaround for this, or any news when a fix is coming or if it is coming? The guides also mention to turn checksums=off on the swap volume, I don't understand why, but can this be related to the panics because I had the checksums enabled in my case. If the current code procudes guaranteed panics with zfs swap, I think that every mention of swap on zfs should be removed from the semi-official wiki guides. Thanks. From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 18:22:37 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B46EE10656AA; Wed, 8 Sep 2010 18:22:37 +0000 (UTC) (envelope-from bright@elvis.mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 92E148FC0A; Wed, 8 Sep 2010 18:22:37 +0000 (UTC) Received: by elvis.mu.org (Postfix, from userid 1192) id 885131A3FBB; Wed, 8 Sep 2010 11:03:33 -0700 (PDT) Date: Wed, 8 Sep 2010 11:03:33 -0700 From: Alfred Perlstein To: fs@freebsd.org, des@freebsd.org Message-ID: <20100908180333.GG69795@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: leverage pseudofs for autofs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 18:22:37 -0000 fs & DES, I'm looking at reimplementing autofs. I've had my eye on psuedofs as a base for autofs since it seems like I can override most of the methods. At a glance what has me wondering is that there seems to be some code in the pseudofs base that checks for processes and traverses allproc. For instance "pfs_readdir()" implies some sort of tie between the number of dir entries and the reading of the directory, obviously for autofs this would be different. I do not need this in autofs and can probably get around this by overloading those vnops entries in my filesystem switch. The basic requirements of autofs are: 1) In-memory filesystem. 2) if a lookup is done for a non-existant dir entry then block the process. (unless it's "special" ie. the actual automounter). 3) for readdir, do some magic to send requests back to userspace to "fill in" a dynamic listing. basically, vop_readdir will optionally dispatch back to userland for this. I'd like to reduce code duplication. Pseudofs looks like a great starting point. Am I missing some tightly coupled issue that makes this a non-starter? Would you be open to adding hooks if autofs needs them? Should I be looking elsewhere or just starting from scratch? thank you! -- - Alfred Perlstein .- AMA, VMOA #5191, 03 vmax, 92 gs500, 85 ch250, 07 zx10 .- FreeBSD committer From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 18:29:57 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E30010656C6 for ; Wed, 8 Sep 2010 18:29:57 +0000 (UTC) (envelope-from numisemis@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id BCDED8FC08 for ; Wed, 8 Sep 2010 18:29:56 +0000 (UTC) Received: by fxm4 with SMTP id 4so383072fxm.13 for ; Wed, 08 Sep 2010 11:29:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=5cAaUMQ093/cPtedq5D/5ujCzFbnxz9A+m8o12j3tq4=; b=keHKZQs0SAacrfoMsrLMH4lXfXLFcLSz9Y02S4yr1xPH25XIn9S9vCpHHIvi4wwW+l sQedhj489pM08tlq+5BdSYhnbgATKHF+hzomXlqIap2+r84LC1F7LVrmUirZ6DhAwJar sxMcUNxET2dmSRoZz9pMI7kIhUrjRiMMGH/J8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=d23QsmV1eDxsvw4asJNPsHrBGAFA/OeUjhywf6gSYZOcYc0BocCPa/X4BngvWmYTir JEetdPAKADeAQFTtLraW/cZGjYH7OVj7sSsw+wHV1EFRUHMQ779KV+aSknnCXTvf9mB0 5Y+IJ2abZ9fzaoo84ZXf3e5Waw0C9+DFv8b+Y= Received: by 10.223.104.13 with SMTP id m13mr189127fao.25.1283969279723; Wed, 08 Sep 2010 11:07:59 -0700 (PDT) Received: from [172.10.20.2] ([178.160.76.147]) by mx.google.com with ESMTPS id 14sm213157fav.26.2010.09.08.11.07.55 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 08 Sep 2010 11:07:58 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: text/plain; charset=us-ascii From: =?iso-8859-2?Q?=A9imun_Mikecin?= In-Reply-To: Date: Wed, 8 Sep 2010 20:07:44 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: App Deb X-Mailer: Apple Mail (2.1081) Cc: freebsd-fs@freebsd.org Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 18:29:57 -0000 On 8. ruj. 2010., at 18:19, App Deb wrote: > Reading the wiki guides they mention the official way to add swap = volumes on > ZFS (set org.freebsd:swap=3Don etc..), >=20 > So I thought that it should work now, But I just got a panic in = 8.1-RELEASE > after a heavy memory situation that touched swap for the first time. >=20 Last time I tried swap on ZFS (long time ago) I needed: zfs set recordsize=3D4k tank/swap Without it, kernel would panic as soon as swap was touched. I suppose = this helped because it matches processor page size. From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 22:32:59 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CFA6910656AA for ; Wed, 8 Sep 2010 22:32:59 +0000 (UTC) (envelope-from des@des.no) Received: from smtp.des.no (smtp.des.no [194.63.250.102]) by mx1.freebsd.org (Postfix) with ESMTP id 94D0F8FC1E for ; Wed, 8 Sep 2010 22:32:59 +0000 (UTC) Received: from ds4.des.no (des.no [84.49.246.2]) by smtp.des.no (Postfix) with ESMTP id 009351FFC36; Wed, 8 Sep 2010 22:13:19 +0000 (UTC) Received: by ds4.des.no (Postfix, from userid 1001) id AA55684525; Thu, 9 Sep 2010 00:13:19 +0200 (CEST) From: =?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?= To: Alfred Perlstein References: <20100908180333.GG69795@elvis.mu.org> Date: Thu, 09 Sep 2010 00:13:19 +0200 In-Reply-To: <20100908180333.GG69795@elvis.mu.org> (Alfred Perlstein's message of "Wed, 8 Sep 2010 11:03:33 -0700") Message-ID: <86fwxjlw8w.fsf@ds4.des.no> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: fs@freebsd.org Subject: Re: leverage pseudofs for autofs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 22:32:59 -0000 Alfred Perlstein writes: > The basic requirements of autofs are: > > 1) In-memory filesystem. > 2) if a lookup is done for a non-existant dir > entry then block the process. (unless it's > "special" ie. the actual automounter). > 3) for readdir, do some magic to send requests > back to userspace to "fill in" a dynamic listing. > basically, vop_readdir will optionally dispatch > back to userland for this. > > I'd like to reduce code duplication. Pseudofs looks > like a great starting point.=20 You can probably use the pseudofs code as a starting point for autofs, but I don't think it's a good idea to build autofs on top of pseudofs, unless you start by greatly generalizing pseudofs: instead of pfstype_procdir, you'd have pfstype_dynamic and custom callbacks to handle readdir and lookup. That actually sounds like a good idea... a lot of work, though. DES --=20 Dag-Erling Sm=C3=B8rgrav - des@des.no From owner-freebsd-fs@FreeBSD.ORG Wed Sep 8 22:41:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E27110656EB for ; Wed, 8 Sep 2010 22:41:45 +0000 (UTC) (envelope-from gull@gull.us) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8DCE38FC08 for ; Wed, 8 Sep 2010 22:41:44 +0000 (UTC) Received: by gwb15 with SMTP id 15so103124gwb.13 for ; Wed, 08 Sep 2010 15:41:40 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.73.142 with SMTP id q14mr494952qcj.26.1283985699887; Wed, 08 Sep 2010 15:41:39 -0700 (PDT) Received: by 10.229.214.199 with HTTP; Wed, 8 Sep 2010 15:41:39 -0700 (PDT) X-Originating-IP: [69.91.158.216] In-Reply-To: <3705884.579377.1283909249721.JavaMail.root@erie.cs.uoguelph.ca> References: <3705884.579377.1283909249721.JavaMail.root@erie.cs.uoguelph.ca> Date: Wed, 8 Sep 2010 15:41:39 -0700 Message-ID: From: David Brodbeck To: Rick Macklem Content-Type: multipart/mixed; boundary=00163646d8e289dff0048fc737d3 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Sep 2010 22:41:45 -0000 --00163646d8e289dff0048fc737d3 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Sep 7, 2010 at 6:27 PM, Rick Macklem wrote: >> On Fri, Sep 3, 2010 at 5:55 PM, Rick Macklem >> wrote: >> > So, please try the attached little patch that just disables the >> > server >> > check for this case for Close. (The line #s will be different than >> > 8.1, but all it is doing is commenting out the test for this case in >> > nfsrv_openupdate(), if the patch won't apply directly.) >> >> This seems to have changed the problem. Now I get the following error >> during checkouts: >> >> svn: In directory 'documentation/software-notes' >> svn: Can't copy >> 'documentation/software-notes/.svn/tmp/text-base/directory-server-install-notes.txt.svn-base' >> to >> 'documentation/software-notes/.svn/tmp/directory-server-install-notes.txt.tmp.tmp': >> Input/output error >> >> I don't see any related errors in the syslog on the client or server. >> >> This is on FreeBSD 8.1-RELEASE; I'm going to try to test with 9 later >> today. >> > Ok, if you feel like emailing me another packet capture, I can look and > see where the EIO is being returned. OK, here's another packet capture. I spent some time trying to find the minimal checkout that would demonstrate the problem, in order to minimize the size of the capture. It turns out checking out several zero-length files will not trigger the problem. (It would, however, trigger the original issue the patch was trying to solve.) Checking out a single file with some text in it works, though, so this is a checkout of a single file called "a.txt" inside a directory called "bsd-test". --00163646d8e289dff0048fc737d3-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 04:28:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B6D6C10656AE; Thu, 9 Sep 2010 04:28:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5A5438FC15; Thu, 9 Sep 2010 04:28:05 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAC//h0yDaFvO/2dsb2JhbACDGZ8DrRSRWYRJdASEeoUl X-IronPort-AV: E=Sophos;i="4.56,337,1280721600"; d="scan'208";a="91231302" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 09 Sep 2010 00:28:03 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 84BB9B3EA8; Thu, 9 Sep 2010 00:28:03 -0400 (EDT) Date: Thu, 9 Sep 2010 00:28:03 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <139628434.645718.1284006483409.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_645717_56228502.1284006483407" X-Originating-IP: [24.65.230.102] X-Mailer: Zimbra 6.0.7_GA_2476.RHEL4 (ZimbraWebClient - SAF3 (Mac)/6.0.7_GA_2473.RHEL4_64) Cc: freebsd-fs@freebsd.org, pjd@freebsd.org Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 04:28:05 -0000 ------=_Part_645717_56228502.1284006483407 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit > > This seems to have changed the problem. Now I get the following error > during checkouts: > > svn: In directory 'documentation/software-notes' > svn: Can't copy > 'documentation/software-notes/.svn/tmp/text-base/directory-server-install-notes.txt.svn-base' > to > 'documentation/software-notes/.svn/tmp/directory-server-install-notes.txt.tmp.tmp': > Input/output error > > I don't see any related errors in the syslog on the client or server. > Ok, your packet trace for this one pointed me at what I think is the problem. Thanks for sending the packet traces. I wouldn't have figured this out without them. Here's what seems to be happening: - At packets #543-546 in the trace, there were two opens of different files like the above, (one ended in .tmp.tmp and the other .svn-base). The two files have different file handles (which they should), but were assigned the same open stateid (which shouldn't be the case). My code only assigns the same stateid if it thinks they have the same file handle. Here's the macro that does this... #define NFSVNO_CMPFH(f1, f2) \ ((f1)->fh_fsid.val[0] == (f2)->fh_fsid.val[0] && \ (f1)->fh_fsid.val[1] == (f2)->fh_fsid.val[1] && \ !bcmp((f1)->fh_fid.fid_data, (f2)->fh_fid.fid_data, \ (f1)->fh_fid.fid_len)) struct fid is... struct fid { u_short fid_len; /* length of data in bytes */ u_short fid_reserved; /* force longword alignment */ char fid_data[MAXFIDSZ]; /* data (variable length) */ }; and the zfs variant of this is... typedef struct zfid_short { uint16_t zf_len; uint8_t zf_object[6]; /* obj[i] = obj >> (8 * i) */ uint8_t zf_gen[4]; /* gen[i] = gen >> (8 * i) */ } zfid_short_t; can you spot the problem? It turned out that, for these two files, the only difference in the fids fell in the fid_reserved field of the generic one, so the macro erroneously said they were the same file handle. Although it would be nice if ZFS filled all of its fid into fid_data, I suspect they needed the extra two fid_reserved bytes. Please try the attached patch, which changes the macro so that it compares the entire "struct fid". (I think that is safe, since I bzero() the entire structure before calling VOP_VPTOFH().) Again, thanks for sending the packet traces and, hopefully, this patch will do the trick, rick ------=_Part_645717_56228502.1284006483407 Content-Type: text/x-patch; name=nfsdport.patch Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=nfsdport.patch LS0tIGZzL25mcy9uZnNkcG9ydC5oLnNhdgkyMDEwLTA5LTA4IDIyOjIzOjEwLjAwMDAwMDAwMCAt MDQwMAorKysgZnMvbmZzL25mc2Rwb3J0LmgJMjAxMC0wOS0wOCAyMjo0Nzo1OC4wMDAwMDAwMDAg LTA0MDAKQEAgLTcwLDggKzcwLDcgQEAKICNkZWZpbmUJTkZTVk5PX0NNUEZIKGYxLCBmMikJCQkJ CQlcCiAgICAgKChmMSktPmZoX2ZzaWQudmFsWzBdID09IChmMiktPmZoX2ZzaWQudmFsWzBdICYm CQkJXAogICAgICAoZjEpLT5maF9mc2lkLnZhbFsxXSA9PSAoZjIpLT5maF9mc2lkLnZhbFsxXSAm JgkJCVwKLSAgICAgIWJjbXAoKGYxKS0+ZmhfZmlkLmZpZF9kYXRhLCAoZjIpLT5maF9maWQuZmlk X2RhdGEsCQlcCi0gICAgICAgICAgICAoZjEpLT5maF9maWQuZmlkX2xlbikpCisgICAgICFiY21w KCYoZjEpLT5maF9maWQsICYoZjIpLT5maF9maWQsIHNpemVvZihzdHJ1Y3QgZmlkKSkpCiAKICNk ZWZpbmUJTkZTTE9DS0hBU0goZikgCQkJCQkJCVwKIAkoJm5mc2xvY2toYXNoWygqKCh1X2ludDMy X3QgKikoKGYpLT5maF9maWQuZmlkX2RhdGEpKSkgJSBORlNMT0NLSEFTSFNJWkVdKQo= ------=_Part_645717_56228502.1284006483407-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 05:42:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31C9010656C6; Thu, 9 Sep 2010 05:42:21 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [64.81.247.49]) by mx1.freebsd.org (Postfix) with ESMTP id F2AE48FC14; Thu, 9 Sep 2010 05:42:20 +0000 (UTC) Received: from chez.mckusick.com (localhost [127.0.0.1]) by chez.mckusick.com (8.14.3/8.14.3) with ESMTP id o895gD7f096872; Wed, 8 Sep 2010 22:42:14 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201009090542.o895gD7f096872@chez.mckusick.com> To: Gleb Kurtsou In-reply-to: <20100907230433.GA3938@tops> Date: Wed, 08 Sep 2010 22:42:13 -0700 From: Kirk McKusick Cc: freebsd-fs@freebsd.org, Jeff Roberson , Ivan Voras Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can disappear before locking it in tmpfs_rename X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 05:42:21 -0000 > Date: Wed, 8 Sep 2010 02:04:33 +0300 > From: Gleb Kurtsou > To: Kirk McKusick > Cc: Ivan Voras , freebsd-fs@freebsd.org > Subject: Re: kern/150143: [patch][tmpfs] Source directory vnode can > disappear before locking it in tmpfs_rename > X-ASK-Info: Message Queued (2010/09/07 16:22:21) > X-ASK-Info: Confirmed by User (2010/09/07 17:06:38) > > Hello Kirk, > > I was working on improving namecache during this summer, and I have to > admit rename with the biggest problem of all, and it still remains. Rename is very complex and hard to get right. I have made at least five attempts to implement it and I am not convinced that it is yet right. > There are several common approaches taken by filesystems. > > UFS locks all vnodes involved in rename, unlocking, trying to lock > vnodes and check for races, tmpfs does something similar (although vnode > locking is incorrect, I'm going to fix it a bit later). > > Some others (like ext2fs and msdosfs if I'm not mistaken) keep locking > at minimum, it seems to work, but honestly I don't see why it can't > race. Ext2fs (and most others) have a filesystem-wide lock that is held whenever one is doing a rename which means that only one rename at a time can take place. That greatly reduces the set of possible races that one has to deal with. While this will obviously limit rename intensive applications, I don't know of any practical examples where this serialization matters. I am about ready to throw in the towel and use this approach for UFS rename. > ZFS is somewhat unique in this respect. It uses name locking, keeps > per directory table of locked file names, i.e. names that can't change > while in table. So that destination file won't be added during rename, > source file can't disappear, etc. > > What do you think about name locking approach taken by ZFS? Are there > any drawbacks you are aware of? A parallel rename can still move one of your parents in such a way that you can end up loping off a branch of the tree: a / \ b d / \ c e rename(b, a/d/e) & rename(d, a/b/c) could end up with b->c->d->e->b in a loop divorced from the tree rooted by a. The current UFS locking will catch this, but one merely tracking names may not. Note that serializing these two renames ensure that this cannot happen as the second rename will recognize that it is about to do something bad as the first one will have finished before it starts. I have not studied the ZFS solution, so they may in fact catch this possible race. > I was thinking of trying to unify rename locking, either make UFS > approach standard, i.e. lock all vnodes outside of rename or use name > locking similar to ZFS. UFS way may not fit well into existing VOP API > (extra vnode lookups to check for races) besides vnode locking order > remains an important issue. ZFS style locks may be interesting in a way > that they would allow to reduce scope of vnode locks, especially > considering merging with ongoing work on rangelocks (just a guess). > > Thanks, > Gleb. I do think that coming up with a common rename solution would be good. If the ZFS code catches the known races, then that would be a good one on which to standardize. Further scrutiny of the current UFS code may show that we have indeed found all the races. But I fear that the only implementable solution is to single-thread rename per filesystem. I have copied Jeff Roberson on this email as he is likely to have some insight on an optimal solution. Kirk McKusick From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 07:54:42 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03D2A10656A6; Thu, 9 Sep 2010 07:54:42 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from forward11.mail.yandex.net (forward11.mail.yandex.net [95.108.130.93]) by mx1.freebsd.org (Postfix) with ESMTP id A137F8FC17; Thu, 9 Sep 2010 07:54:41 +0000 (UTC) Received: from smtp13.mail.yandex.net (smtp13.mail.yandex.net [95.108.130.68]) by forward11.mail.yandex.net (Yandex) with ESMTP id AE9853ED085E; Thu, 9 Sep 2010 11:54:39 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1284018879; bh=lJDMs9xiNRhpO6izgTZAFA4pJUpPnUFa8zOz6ThXMdQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=sDhX9tghjL3BeBAGqyfF/qmmrVt/k+iwVTZYN0Pdt5ip8zuWJ1+YY1a2cWe6FrjBc yQKQRYjJcUbejzKoxzeaT+c+TW0pLFUi3vYwCj9ByA3c9kWgfkiXGGXeKjwfkWBUA8 27RESAPzzCDxlOBJSjOKK2sgxw+DcPi/1EHeLMjA= Received: from nautilus (unknown [178.155.56.98]) by smtp13.mail.yandex.net (Yandex) with ESMTPA id 7FD014158062; Thu, 9 Sep 2010 11:54:39 +0400 (MSD) Received: by nautilus (Postfix, from userid 1001) id 0B2CB1DD43D; Thu, 9 Sep 2010 11:54:37 +0400 (MSD) Date: Thu, 9 Sep 2010 11:54:36 +0400 From: Sergey Zaharchenko To: Pawel Jakub Dawidek Message-ID: <20100909075436.GA2789@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> <20100908103338.GA5091@nautilus.vmks.ru> <20100908153602.GB1751@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <20100908153602.GB1751@garage.freebsd.pl> X-Listening-To: Silence User-Agent: Mutt/1.5.20 (2009-06-14) X-Yandex-TimeMark: 1284018879 X-Yandex-Spam: 1 X-Yandex-Front: smtp13.mail.yandex.net Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 07:54:42 -0000 --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello Pawel! Wed, Sep 08, 2010 at 05:36:02PM +0200 you wrote: > On Wed, Sep 08, 2010 at 02:33:38PM +0400, Sergey Zaharchenko wrote: > > real memory =3D 8589934592 (8192 MB) > > avail memory =3D 8259899392 (7877 MB) > [...] > > /bot/loader.conf has: > >=20 > > mac_bsdextended_load=3D"YES" > > vfs.zfs.zil_disable=3D1 > > vfs.zfs.prefetch_disable=3D1 >=20 > No vm.kmem_size? Try adding: >=20 > vm.kmem_size=3D"8G" > And try leaving ZIL and prefetch alone (eventually turn prefetch off, > but keep ZIL on). Done. Still dies to a repeated make -j 4 buildworld (not sure how many times it takes, mut not more than 5). Thanks, --=20 Sergey Zaharchenko --3MwIy2ne0vdjdPXF Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyIkrwACgkQwo7hT/9lVdyhSACbB8Fp+Hjt27hfzzSJgs4ulMMP cfkAniVFw4/yM0VJ9DfMZ3vn+A4iC9zX =lzT9 -----END PGP SIGNATURE----- --3MwIy2ne0vdjdPXF-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 10:42:46 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59F2910656E3 for ; Thu, 9 Sep 2010 10:42:46 +0000 (UTC) (envelope-from numisemis@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id DD6098FC08 for ; Thu, 9 Sep 2010 10:42:44 +0000 (UTC) Received: by eyx24 with SMTP id 24so833021eyx.13 for ; Thu, 09 Sep 2010 03:42:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:content-type:mime-version :subject:from:in-reply-to:date:content-transfer-encoding:message-id :references:to:x-mailer; bh=Nh47j8cJCFOM5edLZaU0+pyJpdcSoeEUsuHYYTBT8VM=; b=FoiHA7qZ4A7MPPqYMrl1xwIrA4wkoyN3QQz3oqQlcQsikPvYq900KH5rX7KSvlvD4V 5ji6tW1FOl95ZXFxB3HH0Bz/0m1kFGjaqbF8xVAGUYn1pYgPPFu2R2zoRhdAeV3JIxbb qGv7GUBejIP6D/y2W1F965Qaus99xjadW1as4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; b=bpslyA2pFv5osDJB3ETpjK3cx3w4xOcbr/unLbg83QTZQLamQyayvhTGq+oH4Z2Zg1 RLjEmcrmcnQaXrrSyFhamsmAt57UuCjlS1fy44L7SKp6xMW+vLITo+gVHsuVcoVVdpG0 eR53SXo1SSKtQiWpRur63K32WCIC32RzwvpC0= Received: by 10.213.35.6 with SMTP id n6mr55490ebd.0.1284028963706; Thu, 09 Sep 2010 03:42:43 -0700 (PDT) Received: from imac.logos.hr ([213.147.110.159]) by mx.google.com with ESMTPS id a48sm1778419eei.19.2010.09.09.03.42.42 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 09 Sep 2010 03:42:42 -0700 (PDT) Content-Type: text/plain; charset=iso-8859-2 Mime-Version: 1.0 (Apple Message framework v1081) From: =?iso-8859-2?Q?=A9imun_Mikecin?= In-Reply-To: Date: Thu, 9 Sep 2010 12:42:40 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: freebsd-fs X-Mailer: Apple Mail (2.1081) Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 10:42:46 -0000 On 8. ruj. 2010., at 20:07, =A9imun Mikecin wrote: > On 8. ruj. 2010., at 18:19, App Deb wrote: >=20 >> Reading the wiki guides they mention the official way to add swap = volumes on >> ZFS (set org.freebsd:swap=3Don etc..), >>=20 >> So I thought that it should work now, But I just got a panic in = 8.1-RELEASE >> after a heavy memory situation that touched swap for the first time. >>=20 >=20 > Last time I tried swap on ZFS (long time ago) I needed: >=20 > zfs set recordsize=3D4k tank/swap >=20 > Without it, kernel would panic as soon as swap was touched. I suppose = this helped because it matches processor page size. >=20 I have to correct myself: property that I was setting was blocksize, not = recordsize. So it would be somethink like: zfs create -b 4k -V 2G tank/swap From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 13:18:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C61310656CA for ; Thu, 9 Sep 2010 13:18:33 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 21C0B8FC12 for ; Thu, 9 Sep 2010 13:18:32 +0000 (UTC) Received: from [89.21.28.38] (port=28497 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net envelope-to freebsd-fs@freebsd.org id 1OtgjQ-000Lyr-BG; Thu, 09 Sep 2010 14:00:00 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Thu, 09 Sep 2010 14:00:00 +0100 Message-ID: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> Date: Thu, 09 Sep 2010 14:00:00 +0100 From: a.smith@ukgrid.net To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Subject: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 13:18:33 -0000 Hi, I am experiencing kernel panics when running a zfs receive. And unfortunately Im also unable to get a core dump written as its freezing at the point where its meant to write the dump to disk. Initially I was on a FreeBSD 8.1-RELEASE i386 server with 2GB RAM running with an SIIS eSATA card and two eSATA disks attached (using a single cable using port multiplier). The server with the primary copy of the data (that does the zfs sends) is running on amd64 OS with 4GB RAM, so I decided as this was ZFS related the quickest probable fix would be to go to amd64 with 4GB RAM on the receiving server too. I have now done that, Ive moved the eSATA cards and disks to a brand new server, and I am still seeing kernel panics, and Im still unable to write the dump to disk. I have a Dell graphical remote console (no serial console), and the info on screen when the system crashes is like: siisch0: Timeout on slot 23 siisch0: siis_timeout is 00040000 ss 7fffffff rs 7fffffff es 00000000 sts 801d20 siisch0 ..waiting for slots 7f7fffff Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault code = supervisor read, page not present instruction pointer = 0x20:0xffffffff80e6ba05 (always the same) stack pointer = 0x28:0xffffff80000a5ab0 (always the same) frame pointer = 0x28:0xffffff80000a5b00 (always the same) code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current proecess = 12 (irq256: siis0) trap number = 12 panic: page fault cpuid = 0 uptime 1h36m3s The crash screen on the original i386 server was basically the same. Now that Ive moved the disks to the new server I can use this for whatever testing is necessary (the original server is running production services). My boot/loader.conf is: siis_load="YES" vfs.zfs.prefetch_disable=0 Any help much appreciated, thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 13:47:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76CB110656F1 for ; Thu, 9 Sep 2010 13:47:33 +0000 (UTC) (envelope-from appdebgr@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2566B8FC17 for ; Thu, 9 Sep 2010 13:47:32 +0000 (UTC) Received: by qyk4 with SMTP id 4so1640347qyk.13 for ; Thu, 09 Sep 2010 06:47:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=/2DSKRsG1NDbvgH9dmuLQpyeHtNE0ovWFATejUodjg4=; b=dHjIfxKI64r9M4rIRxnplt4weE8lChbKMGuzzaYd0zX9ij71DoX12hMMdXMEqnPWHH BpQ2DK90R3T52tZ1JmUmk/HziO8mt9skCiLTI/77gCqlaben/ahosjeVcsm0qmYHe2Oe j6SZJuuADCB/pTWH9ld0CqkCmWbkwGJWvM+NI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=YGv8+i3O2yx6ikda3L/am04rie2+2a/phNpx3cst3JcbWvSLJ76Yt8dEH7VojowzrW QQq/FkDXaK+YeojRA6R0VFoet5d6fBesRecsrj9QJmoQEbHRxAuLarmwNXwlj5e08xKk ADZnlgPx4N7nCF87cMjuGuaideeIo8VkUjAII= MIME-Version: 1.0 Received: by 10.224.29.16 with SMTP id o16mr627231qac.343.1284040052076; Thu, 09 Sep 2010 06:47:32 -0700 (PDT) Received: by 10.229.189.81 with HTTP; Thu, 9 Sep 2010 06:47:31 -0700 (PDT) In-Reply-To: References: Date: Thu, 9 Sep 2010 16:47:31 +0300 Message-ID: From: App Deb To: =?UTF-8?Q?=C5=A0imun_Mikecin?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 13:47:33 -0000 I just tried a 4k blocksize on the swap zfs volume. It still panics once it touches swap. 2010/9/9 =C5=A0imun Mikecin > On 8. ruj. 2010., at 20:07, =C5=A0imun Mikecin wrote: > > > On 8. ruj. 2010., at 18:19, App Deb wrote: > > > >> Reading the wiki guides they mention the official way to add swap > volumes on > >> ZFS (set org.freebsd:swap=3Don etc..), > >> > >> So I thought that it should work now, But I just got a panic in > 8.1-RELEASE > >> after a heavy memory situation that touched swap for the first time. > >> > > > > Last time I tried swap on ZFS (long time ago) I needed: > > > > zfs set recordsize=3D4k tank/swap > > > > Without it, kernel would panic as soon as swap was touched. I suppose > this helped because it matches processor page size. > > > > I have to correct myself: property that I was setting was blocksize, not > recordsize. So it would be somethink like: > zfs create -b 4k -V 2G tank/swap > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 14:12:27 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D82C910656B5 for ; Thu, 9 Sep 2010 14:12:27 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 90ACB8FC0C for ; Thu, 9 Sep 2010 14:12:27 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1OthrW-0000bQ-Op for freebsd-fs@freebsd.org; Thu, 09 Sep 2010 16:12:26 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 09 Sep 2010 16:12:26 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 09 Sep 2010 16:12:26 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Thu, 09 Sep 2010 16:12:18 +0200 Lines: 19 Message-ID: References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.9) Gecko/20100518 Thunderbird/3.0.4 In-Reply-To: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> X-Enigmail-Version: 1.0.1 Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 14:12:27 -0000 On 09/09/10 15:00, a.smith@ukgrid.net wrote: > Hi, > > I am experiencing kernel panics when running a zfs receive. And Others will comment more, but... > siisch0: Timeout on slot 23 > siisch0: siis_timeout is 00040000 ss 7fffffff rs 7fffffff es 00000000 > sts 801d20 > siisch0 ..waiting for slots 7f7fffff > > Fatal trap 12: page fault while in kernel mode > current proecess = 12 (irq256: siis0) ... this looks like a hardware error. Are you using AHCI here? Try turning it off. From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 14:16:51 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0251710656E4 for ; Thu, 9 Sep 2010 14:16:51 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id A47118FC0A for ; Thu, 9 Sep 2010 14:16:50 +0000 (UTC) Received: from omta05.westchester.pa.mail.comcast.net ([76.96.62.43]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 4bZ41f0060vyq2s5BeGq4x; Thu, 09 Sep 2010 14:16:50 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta05.westchester.pa.mail.comcast.net with comcast id 4eGp1f00D3LrwQ23ReGq59; Thu, 09 Sep 2010 14:16:50 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 68B069B423; Thu, 9 Sep 2010 07:16:48 -0700 (PDT) Date: Thu, 9 Sep 2010 07:16:48 -0700 From: Jeremy Chadwick To: Ivan Voras Message-ID: <20100909141648.GA77376@icarus.home.lan> References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 14:16:51 -0000 On Thu, Sep 09, 2010 at 04:12:18PM +0200, Ivan Voras wrote: > On 09/09/10 15:00, a.smith@ukgrid.net wrote: > >Hi, > > > >I am experiencing kernel panics when running a zfs receive. And > > Others will comment more, but... > > >siisch0: Timeout on slot 23 > >siisch0: siis_timeout is 00040000 ss 7fffffff rs 7fffffff es 00000000 > >sts 801d20 > >siisch0 ..waiting for slots 7f7fffff > > > >Fatal trap 12: page fault while in kernel mode > > >current proecess = 12 (irq256: siis0) > > ... this looks like a hardware error. > > Are you using AHCI here? Try turning it off. This is a siis(4) card -- chances are he can't turn it off. :-) We really need a backtrace to see what transpired. The siis(4) driver may have crashed as a result of LUN timeouts. The driver shouldn't panic, I agree, but the port timing out is of major concern. I imagine this behaviour (the crash) could be reproduced even if UFS2 was used and the timeout took place. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 16:58:37 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED8D01065670 for ; Thu, 9 Sep 2010 16:58:37 +0000 (UTC) (envelope-from doublef-ctm@yandex.ru) Received: from forward19.mail.yandex.net (forward19.mail.yandex.net [95.108.253.144]) by mx1.freebsd.org (Postfix) with ESMTP id 95DA88FC18 for ; Thu, 9 Sep 2010 16:58:37 +0000 (UTC) Received: from smtp18.mail.yandex.net (smtp18.mail.yandex.net [95.108.252.18]) by forward19.mail.yandex.net (Yandex) with ESMTP id 9FE7F344004D; Thu, 9 Sep 2010 20:58:35 +0400 (MSD) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1284051515; bh=j5l2FYy8CD6RMZeRoO8ifJK3w+Qow0tl5VhApdGvKmc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=TOhOm/wiR4faNO/LysxSWK/UMj7E9OHJxsZAw89PLqxnPJ3JyYMJpJVMrYaO3OiRt n03mAAanFgaDhOj+NfdzdaBP3NQadZEGJENWa+GPy0XTiJHgOeNdG7uYjUusAIw0CA +aRuFH+BLwjaveZILSLJ54JpDESgwSAF53nwOPl8= Received: from nautilus (unknown [178.155.62.83]) by smtp18.mail.yandex.net (Yandex) with ESMTPA id 66EE2367807D; Thu, 9 Sep 2010 20:58:35 +0400 (MSD) Received: by nautilus (Postfix, from userid 1001) id 3EC8F1DD43D; Thu, 9 Sep 2010 20:58:29 +0400 (MSD) Date: Thu, 9 Sep 2010 20:58:29 +0400 From: Sergey Zaharchenko To: Jeremy Chadwick Message-ID: <20100909165829.GA2602@nautilus.vmks.ru> References: <20100907164204.GA2571@nautilus.vmks.ru> <20100908065222.GA2522@nautilus.vmks.ru> <20100908103338.GA5091@nautilus.vmks.ru> <20100908104200.GA36566@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="rwEMma7ioTxnRzrJ" Content-Disposition: inline In-Reply-To: <20100908104200.GA36566@icarus.home.lan> X-Listening-To: Silence User-Agent: Mutt/1.5.20 (2009-06-14) X-Yandex-TimeMark: 1284051515 X-Yandex-Spam: 1 X-Yandex-Front: smtp18.mail.yandex.net Cc: freebsd-fs@freebsd.org Subject: Re: 8.1-RELEASE ZFS hangs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 16:58:38 -0000 --rwEMma7ioTxnRzrJ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello Jeremy! Wed, Sep 08, 2010 at 03:42:00AM -0700 you wrote: > On Wed, Sep 08, 2010 at 02:33:38PM +0400, Sergey Zaharchenko wrote: > > Wed, Sep 08, 2010 at 10:07:27AM +0200 you wrote: > > > Could you give more information about the hardware you are using (disk > > > controller in particular) and /boot/loader.conf output. > >=20 > > The controller is a RocketRAID 3540 SATA Controller in single-disk mode: > >=20 > > hptiop0: adapter at PCI 5:0:0, IRQ 16 > > hptiop0: mem 0xdd800000-0xddffffff ir= q 16 at device 0.0 on pci5 > > hptiop0: 0 RocketRAID 3xxx/4xxx controller driver v1.3 (010208) > > hptiop0: [GIANT-LOCKED] > > hptiop0: [ITHREAD] >=20 > If you get rid of hptiop(4) (meaning move the disks to the on-board > controller, which presumably has 4 or 6 ports on it?), does the > situation improve? The internal controller only has 2 ports, one of which is the system disk (not in RAID). I've made 5 file-backed devices with mdconfig (with files on the system disk) and united them into a RAIDZ to rule out the controller. ZFS seems to run fine using this configuration (the resulting device is smaller, of course), e.g. it's still grinding its buildworlds OK. For testing, I set up the original disks into a GEOM stripe and newfs'd it into UFS2. This configuration also seems to be working fine, so the controller and drives are OK also! Am I stupid or what? Thanks, --=20 Sergey Zaharchenko --rwEMma7ioTxnRzrJ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyJEjQACgkQwo7hT/9lVdypvACfcoVtHWUTizSfCfTkfN/7Fkxv NgAAmwYXUfV2r/zaZat15WBiqY8YoLbd =1FmR -----END PGP SIGNATURE----- --rwEMma7ioTxnRzrJ-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 17:23:19 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA6C910656B8 for ; Thu, 9 Sep 2010 17:23:19 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 7159C8FC08 for ; Thu, 9 Sep 2010 17:23:19 +0000 (UTC) Received: from [89.21.28.38] (port=57985 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net envelope-to freebsd-fs@freebsd.org id 1OtkqE-0002OM-BN; Thu, 09 Sep 2010 18:23:18 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Thu, 09 Sep 2010 18:23:18 +0100 Message-ID: <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> Date: Thu, 09 Sep 2010 18:23:18 +0100 From: a.smith@ukgrid.net To: freebsd-fs@freebsd.org References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> In-Reply-To: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 17:23:19 -0000 Quoting jdc: > > This is a siis(4) card -- chances are he can't turn it off. :-) We > really need a backtrace to see what transpired. > > The siis(4) driver may have crashed as a result of LUN timeouts. The > driver shouldn't panic, I agree, but the port timing out is of major > concern. I imagine this behaviour (the crash) could be reproduced even > if UFS2 was used and the timeout took place. Hi, as mentioned I cant currently get any back trace as the system hangs when it starts to core dump after the panic :(. And with SIIS no I dont think I have much options to turn off any features... BTW the card is a PCI-X SiI3132. thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Thu Sep 9 17:49:08 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28F2C10656B1 for ; Thu, 9 Sep 2010 17:49:08 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id E2F608FC18 for ; Thu, 9 Sep 2010 17:49:07 +0000 (UTC) Received: from [89.21.28.38] (port=19326 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net envelope-to freebsd-fs@freebsd.org id 1OtlFD-0002qO-AA; Thu, 09 Sep 2010 18:49:07 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Thu, 09 Sep 2010 18:49:07 +0100 Message-ID: <20100909184907.83447refk019wkr4@webmail2.ukgrid.net> Date: Thu, 09 Sep 2010 18:49:07 +0100 From: a.smith@ukgrid.net To: freebsd-fs@freebsd.org References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> In-Reply-To: <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Sep 2010 17:49:08 -0000 Quoting a.smith@ukgrid.net: > BTW the card is a PCI-X SiI3132. Sorry thats a typo, its a SiI3124. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 06:56:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00C861065672 for ; Fri, 10 Sep 2010 06:56:47 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 49DD98FC15 for ; Fri, 10 Sep 2010 06:56:45 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA15504; Fri, 10 Sep 2010 09:56:42 +0300 (EEST) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OtxXN-0008DH-HD; Fri, 10 Sep 2010 09:56:41 +0300 Message-ID: <4C89D6A8.1080107@icyb.net.ua> Date: Fri, 10 Sep 2010 09:56:40 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: a.smith@ukgrid.net References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> In-Reply-To: <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 06:56:47 -0000 on 09/09/2010 20:23 a.smith@ukgrid.net said the following: > Quoting jdc: >> >> This is a siis(4) card -- chances are he can't turn it off. :-) We >> really need a backtrace to see what transpired. >> >> The siis(4) driver may have crashed as a result of LUN timeouts. The >> driver shouldn't panic, I agree, but the port timing out is of major >> concern. I imagine this behaviour (the crash) could be reproduced even >> if UFS2 was used and the timeout took place. > > Hi, > > as mentioned I cant currently get any back trace as the system hangs when it > starts to core dump after the panic :(. Try to add the following options to your kernel: options KDB options KDB_TRACE With these a backtrace should be printed before dumping and you should be able take a picture of your screen. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 07:39:31 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0988C1065694 for ; Fri, 10 Sep 2010 07:39:31 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id A7E798FC15 for ; Fri, 10 Sep 2010 07:39:30 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 3C9EF45C8A; Fri, 10 Sep 2010 09:39:29 +0200 (CEST) Received: from localhost (pdawidek.whl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 1680C45684; Fri, 10 Sep 2010 09:39:24 +0200 (CEST) Date: Fri, 10 Sep 2010 09:39:12 +0200 From: Pawel Jakub Dawidek To: App Deb Message-ID: <20100910073912.GC2007@garage.freebsd.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="96YOpH+ONegL0A3E" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT amd64 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 07:39:31 -0000 --96YOpH+ONegL0A3E Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 08, 2010 at 07:19:40PM +0300, App Deb wrote: > Reading the wiki guides they mention the official way to add swap volumes= on > ZFS (set org.freebsd:swap=3Don etc..), >=20 > So I thought that it should work now, But I just got a panic in 8.1-RELEA= SE > after a heavy memory situation that touched swap for the first time. Swap on ZVOL is still not recommended. Where did you find information that it now recommended? I can't find anything about swap on http://wiki.freebsd.org/ZFS and on http://wiki.freebsd.org/ZFSQuickStartGuide there is a note that it is not recommended. Let me know where it is advised and I'll remove it or add a note (if the documentation is mine). > It is a hassle to add gmirror volumes for swap on full zfs systems, is th= ere > any workaround for this, or any news when a fix is coming or if it is > coming? I've no plans to fix it, maybe with ZFSv28 it will be easier to fix, but this is really low priority. If you use full ZFS system the recommended layout is described here: http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-con= figuration/ > If the current code procudes guaranteed panics with zfs swap, I think that > every mention of swap on zfs should be removed from the semi-official wiki > guides. BTW. If this issue will be worked on in the future, it will be useful to actually see your panic, backtrace and other debug info. To be honest, I didn't expect it to panic, rather deadlock. Maybe panic is from deadlock resolver? Hard to say without any debug info. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --96YOpH+ONegL0A3E Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkyJ4J8ACgkQForvXbEpPzQPgACfWZ7X4E/Xz4uuGSW9NFc9H5w2 SaoAoMXMC+ctRy0S/p/Ui20bxGSVRU9H =17Pc -----END PGP SIGNATURE----- --96YOpH+ONegL0A3E-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 08:50:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CED3D106564A; Fri, 10 Sep 2010 08:50:33 +0000 (UTC) (envelope-from appdebgr@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6CE088FC2D; Fri, 10 Sep 2010 08:50:33 +0000 (UTC) Received: by qyk4 with SMTP id 4so2706056qyk.13 for ; Fri, 10 Sep 2010 01:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=7AL0bVYCSfiR0wbegOxAGyAGB1dgIaSILhQ8PXJ8IbY=; b=PB1IRod56eg1aILVPgEXpxHKHYVhwAxgVVLfox9fNnI40OUIucOXsSUcweDpnn4CSU ZSO1mRvl52+cJgj16kYt+ejAoW221O/cB77vukLsunm523rh0Nw1T4JZ0LBZaHuPkmUa JFzu94knZZu7+cwsY1MkdqfJWk7NbnuwDfs58= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Z5r1dO+NZyc0U7FVmbD3lMW/vxFXkG6mNt8hm17W1nljK7SgCDgK9vW6gvqeI/v+/h 5rghKNPQ3ZZvoinqVz8K5i69dukbJfKJIhnsUXxWAZNRE5NRm2UyfQiLMSWXZmdusxWb poOSNiGO3ShSJCw80uBixmoYXSlM30sXYAFAc= MIME-Version: 1.0 Received: by 10.224.10.211 with SMTP id q19mr151429qaq.227.1284108632493; Fri, 10 Sep 2010 01:50:32 -0700 (PDT) Received: by 10.229.189.81 with HTTP; Fri, 10 Sep 2010 01:50:32 -0700 (PDT) In-Reply-To: <20100910073912.GC2007@garage.freebsd.pl> References: <20100910073912.GC2007@garage.freebsd.pl> Date: Fri, 10 Sep 2010 11:50:32 +0300 Message-ID: From: App Deb To: Pawel Jakub Dawidek Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 08:50:33 -0000 Yes sorry for actually calling it a panic, it is a deadlock, I didn't see any messages, anyway the system becomes unusable and needs a manual hard reboot. I found the mention (without any warnings) here: http://wiki.freebsd.org/RootOnZFS I have tried this on 3 different systems (all 8.1R though) and they always deadlock once the zfs swap is touched. I think that the "not recommended" is not enough. It just doesn't work at all, except if I am doing something wrong and there are some cases that it works? I have now switched to native freebsd-swap (after a long zfs send/receive backup to resize the zpool :) ) and everything works great. Thanks. On Fri, Sep 10, 2010 at 10:39 AM, Pawel Jakub Dawidek wrote: > On Wed, Sep 08, 2010 at 07:19:40PM +0300, App Deb wrote: > > Reading the wiki guides they mention the official way to add swap volumes > on > > ZFS (set org.freebsd:swap=on etc..), > > > > So I thought that it should work now, But I just got a panic in > 8.1-RELEASE > > after a heavy memory situation that touched swap for the first time. > > Swap on ZVOL is still not recommended. Where did you find information > that it now recommended? I can't find anything about swap on > http://wiki.freebsd.org/ZFS and on > http://wiki.freebsd.org/ZFSQuickStartGuide there is a note that it is > not recommended. Let me know where it is advised and I'll remove it or > add a note (if the documentation is mine). > > > It is a hassle to add gmirror volumes for swap on full zfs systems, is > there > > any workaround for this, or any news when a fix is coming or if it is > > coming? > > I've no plans to fix it, maybe with ZFSv28 it will be easier to fix, but > this is really low priority. If you use full ZFS system the recommended > layout is described here: > > > http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/ > > > If the current code procudes guaranteed panics with zfs swap, I think > that > > every mention of swap on zfs should be removed from the semi-official > wiki > > guides. > > BTW. If this issue will be worked on in the future, it will be useful to > actually see your panic, backtrace and other debug info. To be honest, I > didn't expect it to panic, rather deadlock. Maybe panic is from deadlock > resolver? Hard to say without any debug info. > > -- > Pawel Jakub Dawidek http://www.wheelsystems.com > pjd@FreeBSD.org http://www.FreeBSD.org > FreeBSD committer Am I Evil? Yes, I Am! > From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 13:11:13 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD9AB106566B for ; Fri, 10 Sep 2010 13:11:13 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 82B998FC15 for ; Fri, 10 Sep 2010 13:11:13 +0000 (UTC) Received: from [89.21.28.38] (port=54358 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou3Nn-0003gy-9l; Fri, 10 Sep 2010 14:11:11 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 14:11:10 +0100 Message-ID: <20100910141110.205866s50jvc8zk0@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 14:11:10 +0100 From: a.smith@ukgrid.net To: Andriy Gapon References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> In-Reply-To: <4C89D6A8.1080107@icyb.net.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 13:11:13 -0000 Quoting Andriy Gapon : > > Try to add the following options to your kernel: > options KDB > options KDB_TRACE > > With these a backtrace should be printed before dumping and you > should be able > take a picture of your screen. > > -- > Andriy Gapon > Thanks Andiy, have done, but the useful info is going off the top of the screen before I can catch it. Am going to try again, problem is watching the screen every second for about an hour! thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 13:39:01 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 265F11065679 for ; Fri, 10 Sep 2010 13:39:01 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id E041B8FC1C for ; Fri, 10 Sep 2010 13:39:00 +0000 (UTC) Received: from [89.21.28.38] (port=34544 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou3oi-0004Fz-AE; Fri, 10 Sep 2010 14:39:00 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 14:39:00 +0100 Message-ID: <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 14:39:00 +0100 From: a.smith@ukgrid.net To: Andriy Gapon References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> In-Reply-To: <4C89D6A8.1080107@icyb.net.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 13:39:01 -0000 Quoting Andriy Gapon : > > Try to add the following options to your kernel: > options KDB > options KDB_TRACE > > With these a backtrace should be printed before dumping and you > should be able > take a picture of your screen. > > -- > Andriy Gapon > Hi, So far I have got this additional info from the console when the server panics: cpuid = 0 Uptime: 19m57s Sleeping thread (tid 100024, pid12) owns a non-sleepable lock panic: sleeping thread cpuid = 0 Uptime: 19m57s kernel trap 12 with interrupts disabled then follows the same text I originally posted... Im going to run this again and try check if any other info is printed but scrolled off the top of the screen... thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 14:11:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A54B106566B for ; Fri, 10 Sep 2010 14:11:30 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id E43278FC0A for ; Fri, 10 Sep 2010 14:11:29 +0000 (UTC) Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27]) by qmta15.emeryville.ca.mail.comcast.net with comcast id 51T51f0070b6N64AF2BVKT; Fri, 10 Sep 2010 14:11:29 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta03.emeryville.ca.mail.comcast.net with comcast id 52BT1f00S3LrwQ28P2BUQC; Fri, 10 Sep 2010 14:11:29 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id AD06A9B423; Fri, 10 Sep 2010 07:11:27 -0700 (PDT) Date: Fri, 10 Sep 2010 07:11:27 -0700 From: Jeremy Chadwick To: a.smith@ukgrid.net Message-ID: <20100910141127.GA13056@icarus.home.lan> References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 14:11:30 -0000 On Fri, Sep 10, 2010 at 02:39:00PM +0100, a.smith@ukgrid.net wrote: > Quoting Andriy Gapon : > > > > >Try to add the following options to your kernel: > >options KDB > >options KDB_TRACE > > > >With these a backtrace should be printed before dumping and you > >should be able > >take a picture of your screen. > > > >-- > >Andriy Gapon > > > > Hi, > > So far I have got this additional info from the console when the > server panics: > > cpuid = 0 > Uptime: 19m57s > Sleeping thread (tid 100024, pid12) owns a non-sleepable lock > panic: sleeping thread > cpuid = 0 > Uptime: 19m57s > kernel trap 12 with interrupts disabled > > then follows the same text I originally posted... > Im going to run this again and try check if any other info is > printed but scrolled off the top of the screen... What we're looking for is something that resembles what's shown here (using this as an example): http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00413.html The KDB and KDB_TRACE options should result in a backtrace automatically being shown when the system panics. Are you absolutely certain your kernel config has the options Andriy provided, and you rebuilt + reinstalled the kernel? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 14:16:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73DF6106564A for ; Fri, 10 Sep 2010 14:16:26 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 3A41A8FC17 for ; Fri, 10 Sep 2010 14:16:24 +0000 (UTC) Received: from [89.21.28.38] (port=38530 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou4Ou-0004yk-AD; Fri, 10 Sep 2010 15:16:24 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 15:16:24 +0100 Message-ID: <20100910151624.1140550i5h4mbmpk@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 15:16:24 +0100 From: a.smith@ukgrid.net To: Andriy Gapon References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> In-Reply-To: <4C89D6A8.1080107@icyb.net.ua> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_3z35x87eym0" Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 14:16:26 -0000 This message is in MIME format. --=_3z35x87eym0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit Quoting Andriy Gapon : > > Try to add the following options to your kernel: > options KDB > options KDB_TRACE > > With these a backtrace should be printed before dumping and you > should be able > take a picture of your screen. > > -- > Andriy Gapon > Ok, at the third time of asking I have everything that was logged to screen (also I noticed the last panic was put down to syncer process, not sure what that was about but this one is back to SIIS). In the end its given me zero more info than I had before :( thanks Andy. --=_3z35x87eym0-- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 14:32:00 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83A1A106566C for ; Fri, 10 Sep 2010 14:32:00 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C9C538FC17 for ; Fri, 10 Sep 2010 14:31:59 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA21524; Fri, 10 Sep 2010 17:31:56 +0300 (EEST) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Ou3eg-0008dT-TP; Fri, 10 Sep 2010 16:28:38 +0300 Message-ID: <4C8A3285.3040607@icyb.net.ua> Date: Fri, 10 Sep 2010 16:28:37 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: a.smith@ukgrid.net References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910141110.205866s50jvc8zk0@webmail2.ukgrid.net> In-Reply-To: <20100910141110.205866s50jvc8zk0@webmail2.ukgrid.net> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 14:32:00 -0000 on 10/09/2010 16:11 a.smith@ukgrid.net said the following: > Quoting Andriy Gapon : > >> >> Try to add the following options to your kernel: >> options KDB >> options KDB_TRACE >> >> With these a backtrace should be printed before dumping and you should be able >> take a picture of your screen. >> >> -- >> Andriy Gapon >> > Thanks Andiy, > > have done, but the useful info is going off the top of the screen before I can > catch it. Am going to try again, problem is watching the screen every second for > about an hour! You can add DDB option, then kernel should enter it on panic instead of automatically trying to dump and reboot. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 14:40:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F35231065674 for ; Fri, 10 Sep 2010 14:40:44 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8834D8FC27 for ; Fri, 10 Sep 2010 14:40:44 +0000 (UTC) Received: by ewy4 with SMTP id 4so1996639ewy.13 for ; Fri, 10 Sep 2010 07:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=JY48CHn8pIcufGyNvKvrE3FyNuJt7VIBoRemZNRxsc0=; b=q7WirymZh5onEFCUu/fjX8qpY8TsiGnSW7FkrHp0tEDX0lZtf3cNP6kfDJ//eOcoSw 8ZNx9UiuKpvHB3aFz9uBEHMzm/mYoqLqqAe3FkKwJGU2o75gEQnnX1v42ZsfqoylQJkX nj4+1UtRUNaiHYuhc6NPDTEKZ51zPqS+OeuaY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=NtCXi/9F/4Ka2za8Gm+BKL7cbrtdSDEzuKCaD9+tn53dyniQSC92pSBiRFRMCXXhsP qWJ6f2ISgz6LILZnHLKm+i/6QZE6FysLJ2QRfQwza4XsiqpOlccP3QaSPxpG9T+6PENS EJJUoVyyCE/b4ecLzEz6b6uKjqPgsUntnWmks= MIME-Version: 1.0 Received: by 10.216.176.83 with SMTP id a61mr992222wem.47.1284129642956; Fri, 10 Sep 2010 07:40:42 -0700 (PDT) Received: by 10.216.6.28 with HTTP; Fri, 10 Sep 2010 07:40:42 -0700 (PDT) In-Reply-To: References: <20100910073912.GC2007@garage.freebsd.pl> Date: Fri, 10 Sep 2010 07:40:42 -0700 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 14:40:45 -0000 On Fri, Sep 10, 2010 at 1:50 AM, App Deb wrote: > Yes sorry for actually calling it a panic, it is a deadlock, I didn't see > any messages, anyway the system becomes unusable and needs a manual hard > reboot. > > I found the mention (without any warnings) here: > http://wiki.freebsd.org/RootOnZFS > > I have tried this on 3 different systems (all 8.1R though) and they always > deadlock once the zfs swap is touched. > > I think that the "not recommended" is not enough. It just doesn't work at > all, except if I am doing something wrong and there are some cases that it > works? > > I have now switched to native freebsd-swap (after a long zfs send/receive > backup to resize the zpool :) ) and everything works great. An alternative to re-creating the pool to add a non-ZFS swap is to plug in a 2-8 GB USB stick and use that for swap. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 14:55:12 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10D01106564A for ; Fri, 10 Sep 2010 14:55:12 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id C863A8FC15 for ; Fri, 10 Sep 2010 14:55:11 +0000 (UTC) Received: from [89.21.28.38] (port=21628 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou50Q-0005mj-AE; Fri, 10 Sep 2010 15:55:10 +0100 Received: from voip.ukgrid.net (voip.ukgrid.net [89.107.16.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 15:55:10 +0100 Message-ID: <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 15:55:10 +0100 From: a.smith@ukgrid.net To: Jeremy Chadwick References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> In-Reply-To: <20100910141127.GA13056@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 14:55:12 -0000 Quoting Jeremy Chadwick : > What we're looking for is something that resembles what's shown here > (using this as an example): > > http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00413.html > > The KDB and KDB_TRACE options should result in a backtrace automatically > being shown when the system panics. Are you absolutely certain your > kernel config has the options Andriy provided, and you rebuilt + > reinstalled the kernel? > Hi Jeremy, well Ive not had to built a custom kernel before, but Ive done the make buildkernel and make installkernel and all seems to have gone thro as expected. Here is the evidence from after a reboot: # pwd /usr/src/sys/amd64/conf # grep KDB ZFSDEBUG options KDB options KDB_TRACE [root@kappa /usr/src/sys/amd64/conf]# uname -a FreeBSD kappa.ukgrid.net 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Fri Sep 10 12:30:24 BST 2010 root@kappa.ukgrid.net:/usr/obj/usr/src/sys/ZFSDEBUG amd64 Any ideas? thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 15:25:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46ED8106566C for ; Fri, 10 Sep 2010 15:25:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA11.westchester.pa.mail.comcast.net (qmta11.westchester.pa.mail.comcast.net [76.96.59.211]) by mx1.freebsd.org (Postfix) with ESMTP id E66F18FC15 for ; Fri, 10 Sep 2010 15:25:46 +0000 (UTC) Received: from omta20.westchester.pa.mail.comcast.net ([76.96.62.71]) by QMTA11.westchester.pa.mail.comcast.net with comcast id 50RX1f00C1YDfWL5B3RnJY; Fri, 10 Sep 2010 15:25:47 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta20.westchester.pa.mail.comcast.net with comcast id 53Rl1f00V3LrwQ23g3Rmpq; Fri, 10 Sep 2010 15:25:47 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7A2D59B423; Fri, 10 Sep 2010 08:25:44 -0700 (PDT) Date: Fri, 10 Sep 2010 08:25:44 -0700 From: Jeremy Chadwick To: a.smith@ukgrid.net Message-ID: <20100910152544.GA14636@icarus.home.lan> References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 15:25:47 -0000 On Fri, Sep 10, 2010 at 03:55:10PM +0100, a.smith@ukgrid.net wrote: > Quoting Jeremy Chadwick : > > >What we're looking for is something that resembles what's shown here > >(using this as an example): > > > >http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00413.html > > > >The KDB and KDB_TRACE options should result in a backtrace automatically > >being shown when the system panics. Are you absolutely certain your > >kernel config has the options Andriy provided, and you rebuilt + > >reinstalled the kernel? > > > > Hi Jeremy, > > well Ive not had to built a custom kernel before, but Ive done the > make buildkernel and make installkernel and all seems to have gone > thro as expected. Here is the evidence from after a reboot: > > > # pwd > /usr/src/sys/amd64/conf > # grep KDB ZFSDEBUG > options KDB > options KDB_TRACE > [root@kappa /usr/src/sys/amd64/conf]# uname -a > FreeBSD kappa.ukgrid.net 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Fri Sep > 10 12:30:24 BST 2010 > root@kappa.ukgrid.net:/usr/obj/usr/src/sys/ZFSDEBUG amd64 > > Any ideas? Not particularly. Sure looks like you should be seeing a backtrace to me. I'd recommend following Andriy's advice of adding "options DDB" to your kernel configuration, rebuild + install kernel + reboot. When the panic happens, you will be dropped to a db> prompt at the console. You should type "trace" to get a backtrace at that point. Something you might also try/do before the kernel rebuild/reboot: If you have a swap slice defined (not on ZFS!), I would would advocate setting dumpdev="auto" in your /etc/rc.conf prior to the reboot. This will allow you to do a post-crash analysis. You'll need to make sure that your /var filesystem has enough space to hold an entire memory dump (e.g. if your system RAM is 8GB, ensure you have at least 8GB free on /var). Next, depending on the state of the system (particularly the controller driver and ATA subsystem), the following may or may not work for you. After the above db> commands, type "call doadump", which should dump all the contents of kernel memory to swap. The next time the system starts up, savecore(8) should automatically extract the crash contents from swap and save them in /var/crash -- hopefully without the system crashing in the middle of writing the contents. :-) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 16:34:07 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44162106566C for ; Fri, 10 Sep 2010 16:34:07 +0000 (UTC) (envelope-from gvidals@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 99F8A8FC12 for ; Fri, 10 Sep 2010 16:34:06 +0000 (UTC) Received: by fxm4 with SMTP id 4so2174491fxm.13 for ; Fri, 10 Sep 2010 09:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:reply-to:date :message-id:subject:from:to:content-type; bh=Aumvt04ovW05loNjW18lgrvAFH3X6THazCWXND5yIOQ=; b=AAhL4Wfl0W0La2E+47igGvgxmOIz45SEB1eflWgBpDgJMS/0GtAA3fRiQBoN6ziRxE ovv9JeFrCAhxGwLJtN2f9utQ/i32uowsVWEEWH/gEGxwkOrdepKgd0ovwOaCySGOy49Y t/SeNgep9AEWvKCEEMFdLOK/x/TAJTQ/3AbAU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; b=xwS8el3ssPQFd9EhR1xh1DBp2CozmHj8p6pGTSV6YTlGEZ2bi1D/pHIPQXtWZfJf5H SJY8oet4x9ht9la4Kup5XYmZLFnwHAOScK6RHPB2vm6quaJbgF/lli1j/zK/TZOBmFE/ aBzsf/YNXPs0Fp+W1QZmpYx85Vy+vz62aDLVw= MIME-Version: 1.0 Received: by 10.239.186.3 with SMTP id e3mr91432hbh.67.1284134688046; Fri, 10 Sep 2010 09:04:48 -0700 (PDT) Received: by 10.239.153.75 with HTTP; Fri, 10 Sep 2010 09:04:48 -0700 (PDT) Date: Fri, 10 Sep 2010 09:04:48 -0700 Message-ID: From: Gil Vidals To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS write speed performance tuning (NFS) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gil@vidals.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 16:34:08 -0000 I'm building my first ZFS NAS and I'm struggling with write speeds over NFS. I want to know if it is possible to somehow set NFS to async AND keep slogging active with my SSD write cache? Here is a brief run down of my write speeds on a 1 gigabit LAN and FreeBSD 8.1 using ZFS: WRITE SPEEDS of 1 GB file: 55 MB/sec: scp copy test 28 MB/sec: NFS with async (zil_disable="1") 12 MB/sec: NFS with sync (default settings) 3 x 1TB 7200 RPM SATA HDs CPU Intel i7 Controller: LSI 1068e Controller: ICH10R (for read and write SSDs) This NAS will eventually go into my VMware environment and unfortunately ESX uses o_sync in their NFS client. AFAIK, there is no way to change the VMware NFS client, so I thought of turning off async on the server side and yet somehow still use the ZIL SSD Cache for fast writes? Looking forward to learning more about ZFS. --Gil Vidals From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 16:39:15 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8AB6F106566C for ; Fri, 10 Sep 2010 16:39:15 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 4D08B8FC17 for ; Fri, 10 Sep 2010 16:39:15 +0000 (UTC) Received: from [89.21.28.38] (port=14883 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou6d6-0008Ek-CW; Fri, 10 Sep 2010 17:39:12 +0100 Received: from 39757.net (39757.net [89.107.23.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 17:39:12 +0100 Message-ID: <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 17:39:12 +0100 From: a.smith@ukgrid.net To: Jeremy Chadwick References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> In-Reply-To: <20100910152544.GA14636@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 16:39:15 -0000 Quoting Jeremy Chadwick : > When the panic happens, you will be dropped to a db> prompt at the > console. You should type "trace" to get a backtrace at that point. > ok, I have successfully got to the db> prompt, here is the output of trace: Tracing pid 12 tid 100024 td 0xffffff0001667000 siis_end_transaction() at siis_end_transaction+0x45 siis_ch_intr() at siis_ch_intr+0x70 siis_ch_intr_locked() at siis_ch_intr_locked+0x3e siis_intr() at siis_intr+0x61 intr_event_exectute_handlers() at intr_event_execute_handlers+0xfd ithread_loop() at fork_exit+0x118 fork_exit() at fork_exit+0x118 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a5d30, rbp = 0 --- The call doadump also seems to have worked so hopefully I have some more info for you if you tell me what youŽd like me to do with it, cheers Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 16:42:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A8611065674 for ; Fri, 10 Sep 2010 16:42:18 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.emeryville.ca.mail.comcast.net (qmta08.emeryville.ca.mail.comcast.net [76.96.30.80]) by mx1.freebsd.org (Postfix) with ESMTP id E0D2D8FC0C for ; Fri, 10 Sep 2010 16:42:17 +0000 (UTC) Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52]) by qmta08.emeryville.ca.mail.comcast.net with comcast id 4zf51f00117UAYkA84iHHr; Fri, 10 Sep 2010 16:42:17 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta13.emeryville.ca.mail.comcast.net with comcast id 54iF1f00D3LrwQ28Z4iGGv; Fri, 10 Sep 2010 16:42:16 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 98D2A9B423; Fri, 10 Sep 2010 09:42:15 -0700 (PDT) Date: Fri, 10 Sep 2010 09:42:15 -0700 From: Jeremy Chadwick To: a.smith@ukgrid.net Message-ID: <20100910164215.GA16881@icarus.home.lan> References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org, mav@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 16:42:18 -0000 On Fri, Sep 10, 2010 at 05:39:12PM +0100, a.smith@ukgrid.net wrote: > Quoting Jeremy Chadwick : > > >When the panic happens, you will be dropped to a db> prompt at the > >console. You should type "trace" to get a backtrace at that point. > > ok, I have successfully got to the db> prompt, here is the output of trace: > > Tracing pid 12 tid 100024 td 0xffffff0001667000 > siis_end_transaction() at siis_end_transaction+0x45 > siis_ch_intr() at siis_ch_intr+0x70 > siis_ch_intr_locked() at siis_ch_intr_locked+0x3e > siis_intr() at siis_intr+0x61 > intr_event_exectute_handlers() at intr_event_execute_handlers+0xfd > ithread_loop() at fork_exit+0x118 > fork_exit() at fork_exit+0x118 > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff80000a5d30, rbp = 0 --- > > The call doadump also seems to have worked so hopefully I have some > more info for you if you tell me what youŽd like me to do with it, Adding mav@ to the CC list. Alexander, have any ideas? The original problem report for reading, and model of card: http://lists.freebsd.org/pipermail/freebsd-fs/2010-September/009319.html http://lists.freebsd.org/pipermail/freebsd-fs/2010-September/009325.html -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 16:48:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C3CC110657AD; Fri, 10 Sep 2010 16:48:55 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id 8523F8FC1C; Fri, 10 Sep 2010 16:48:55 +0000 (UTC) Received: from [89.21.28.38] (port=45846 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou6mU-0008TB-9k; Fri, 10 Sep 2010 17:48:54 +0100 Received: from 39757.net (39757.net [89.107.23.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 17:48:53 +0100 Message-ID: <20100910174853.3551491lc7hwe4ao@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 17:48:53 +0100 From: a.smith@ukgrid.net To: Jeremy Chadwick References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> <20100910164215.GA16881@icarus.home.lan> In-Reply-To: <20100910164215.GA16881@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org, mav@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 16:48:55 -0000 Quoting Jeremy Chadwick : > > Adding mav@ to the CC list. Alexander, have any ideas? The original > problem report for reading, and model of card: > Id planned on swapping out the SIIS PCI card tonight with an identical card to eliminate the possibility of a hardware fault on the card, is that good for you guys or would you prefer I leave all as is while we are investigating? thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 17:30:21 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73E81106564A; Fri, 10 Sep 2010 17:30:21 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 48D258FC1B; Fri, 10 Sep 2010 17:30:20 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA25309; Fri, 10 Sep 2010 20:30:15 +0300 (EEST) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Ou7QV-0008vx-HT; Fri, 10 Sep 2010 20:30:15 +0300 Message-ID: <4C8A6B26.8050305@icyb.net.ua> Date: Fri, 10 Sep 2010 20:30:14 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.8) Gecko/20100822 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: a.smith@ukgrid.net References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> In-Reply-To: <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Alexander Motin Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 17:30:21 -0000 on 10/09/2010 19:39 a.smith@ukgrid.net said the following: > Quoting Jeremy Chadwick : > >> When the panic happens, you will be dropped to a db> prompt at the >> console. You should type "trace" to get a backtrace at that point. >> > > ok, I have successfully got to the db> prompt, here is the output of trace: > > > Tracing pid 12 tid 100024 td 0xffffff0001667000 > siis_end_transaction() at siis_end_transaction+0x45 > siis_ch_intr() at siis_ch_intr+0x70 > siis_ch_intr_locked() at siis_ch_intr_locked+0x3e > siis_intr() at siis_intr+0x61 > intr_event_exectute_handlers() at intr_event_execute_handlers+0xfd > ithread_loop() at fork_exit+0x118 > fork_exit() at fork_exit+0x118 > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip = 0, rsp = 0xffffff80000a5d30, rbp = 0 --- That's useful information I think. For completeness you might also run kgdb and then do: (kgdb) info line *siis_end_transaction+0x45 and so on for all stack trace lines involving siis. Thanks. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 17:43:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD29F1065672 for ; Fri, 10 Sep 2010 17:43:24 +0000 (UTC) (envelope-from gull@gull.us) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 92F608FC17 for ; Fri, 10 Sep 2010 17:43:24 +0000 (UTC) Received: by qwg5 with SMTP id 5so1828898qwg.13 for ; Fri, 10 Sep 2010 10:43:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.224.60.136 with SMTP id p8mr558655qah.330.1284140603310; Fri, 10 Sep 2010 10:43:23 -0700 (PDT) Received: by 10.229.214.199 with HTTP; Fri, 10 Sep 2010 10:43:22 -0700 (PDT) X-Originating-IP: [64.81.163.112] In-Reply-To: <139628434.645718.1284006483409.JavaMail.root@erie.cs.uoguelph.ca> References: <139628434.645718.1284006483409.JavaMail.root@erie.cs.uoguelph.ca> Date: Fri, 10 Sep 2010 10:43:22 -0700 Message-ID: From: David Brodbeck To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 17:43:24 -0000 On Wed, Sep 8, 2010 at 9:28 PM, Rick Macklem wrote: > Please try the attached patch, which changes the macro so that it > compares the entire "struct fid". (I think that is safe, since I > bzero() the entire structure before calling VOP_VPTOFH().) > > Again, thanks for sending the packet traces and, hopefully, this > patch will do the trick, rick Rick, after doing some testing I'm pretty confident that this patch (nfsdport.patch) has solved both the original problem and the I/O error problem. I'm glad I could help. I know how tricky to debug these interoperability problems can be. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 17:49:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0515A106566B; Fri, 10 Sep 2010 17:49:24 +0000 (UTC) (envelope-from a.smith@ukgrid.net) Received: from mx0.ukgrid.net (mx0.ukgrid.net [89.21.28.37]) by mx1.freebsd.org (Postfix) with ESMTP id B94158FC1B; Fri, 10 Sep 2010 17:49:23 +0000 (UTC) Received: from [89.21.28.38] (port=12842 helo=omicron.ukgrid.net) by mx0.ukgrid.net with esmtp (Exim 4.72; FreeBSD) envelope-from a.smith@ukgrid.net id 1Ou7iz-0009Q5-Ck; Fri, 10 Sep 2010 18:49:21 +0100 Received: from 39757.net (39757.net [89.107.23.9]) by webmail2.ukgrid.net (Horde Framework) with HTTP; Fri, 10 Sep 2010 18:49:21 +0100 Message-ID: <20100910184921.16956kbaskhrsmg4@webmail2.ukgrid.net> Date: Fri, 10 Sep 2010 18:49:21 +0100 From: a.smith@ukgrid.net To: Andriy Gapon References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> <4C8A6B26.8050305@icyb.net.ua> In-Reply-To: <4C8A6B26.8050305@icyb.net.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.0 Cc: freebsd-fs@freebsd.org, Alexander Motin Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 17:49:24 -0000 Quoting Andriy Gapon : > > That's useful information I think. > For completeness you might also run kgdb and then do: > (kgdb) info line *siis_end_transaction+0x45 > and so on for all stack trace lines involving siis. > Thanks. > Ok, I got this running against the data in /var/crash, is that ok? (kgdb) info line *siis_end_transaction+0x45 Line 1188 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" starts at address 0xffffffff80ea5a05 and ends at 0xffffffff80ea5a13 . (kgdb) info line *siis_ch_intr_locked+0x3e Line 809 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" starts at address 0xffffffff80ea780e and ends at 0xffffffff80ea7811 . (kgdb) info line *siis_intr+0x61 Line 298 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" starts at address 0xffffffff80ea7a61 and ends at 0xffffffff80ea7a72 . let me know, thanks Andy. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 19:08:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 277C01065679 for ; Fri, 10 Sep 2010 19:08:36 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9CE9F8FC13 for ; Fri, 10 Sep 2010 19:08:35 +0000 (UTC) Received: by bwz20 with SMTP id 20so2986916bwz.13 for ; Fri, 10 Sep 2010 12:08:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=qn5wC2snOEB+MkCmlgwzuNs8PlcBCJ3b9I73SO9abmc=; b=AjsfT/XGVyEsr+ipCPo6c8xCtzOU5QC71Pis9W5yt0pCgjJJiJpQScAeTNaAyfqcX7 jOV+fCOkJK9lTkPLvoBsfQr9qzMAABX4DcX3LzMVd60TLuGnJbfvqwPQ1+Rk1oSRodtN PIg9bCdiYFVf1UVtJWCHHh4UwcVoHPE5NPPWw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=e3Kl3NQRxLuRA4AfkWLioLgne0JaYKlsoU5SdaeNQzgEdlS8whyL2Sk2dMhnJvQa6G FwXONxyM4xMJ4QuHc/fhyYncY/fG/yV+rWSRuTrYxfJQrtxDtyZMM1HZq7cuNJ8QRNUA cKEA1Fr6WCyaY71B8VuhSoK6DC1gJw+F52fRE= Received: by 10.204.126.92 with SMTP id b28mr833046bks.47.1284143923453; Fri, 10 Sep 2010 11:38:43 -0700 (PDT) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id x19sm2213114bkv.9.2010.09.10.11.38.39 (version=SSLv3 cipher=RC4-MD5); Fri, 10 Sep 2010 11:38:40 -0700 (PDT) Sender: Alexander Motin Message-ID: <4C8A7B20.7090408@FreeBSD.org> Date: Fri, 10 Sep 2010 21:38:24 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: a.smith@ukgrid.net References: <20100909140000.5744370gkyqv4eo0@webmail2.ukgrid.net> <20100909182318.11133lqu4q4u1mw4@webmail2.ukgrid.net> <4C89D6A8.1080107@icyb.net.ua> <20100910143900.20382xl5bl6oo9as@webmail2.ukgrid.net> <20100910141127.GA13056@icarus.home.lan> <20100910155510.11831w104qjpyc4g@webmail2.ukgrid.net> <20100910152544.GA14636@icarus.home.lan> <20100910173912.205969tzhjiovf8c@webmail2.ukgrid.net> <4C8A6B26.8050305@icyb.net.ua> <20100910184921.16956kbaskhrsmg4@webmail2.ukgrid.net> In-Reply-To: <20100910184921.16956kbaskhrsmg4@webmail2.ukgrid.net> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, Andriy Gapon Subject: Re: ZFS related kernel panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 19:08:36 -0000 Hi. a.smith@ukgrid.net wrote: > Quoting Andriy Gapon : >> That's useful information I think. >> For completeness you might also run kgdb and then do: >> (kgdb) info line *siis_end_transaction+0x45 >> and so on for all stack trace lines involving siis. > > Ok, I got this running against the data in /var/crash, is that ok? > > (kgdb) info line *siis_end_transaction+0x45 > Line 1188 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" > starts at address 0xffffffff80ea5a05 > and ends at 0xffffffff80ea5a13 . > (kgdb) info line *siis_ch_intr_locked+0x3e > Line 809 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" > starts at address 0xffffffff80ea780e > and ends at 0xffffffff80ea7811 . > (kgdb) info line *siis_intr+0x61 > Line 298 of "/usr/src/sys/modules/siis/../../dev/siis/siis.c" > starts at address 0xffffffff80ea7a61 > and ends at 0xffffffff80ea7a72 . It looks like during timeout handling (it is quite complicated process when port multiplier is used) some request was completed twice. So original problem is probably in hardware (try to check/replace cables, multiplier, ...), that caused timeout, but the fact that drive was unable to handle it is probably a siis(4) driver bug. At this moment I have no idea where to look, except dumb rereading whole error handling logic. It could help much if I could reproduce the problem in realistic time in controlled environment with access to serial (or some other) console and possibility to insert additional debugging. -- Alexander Motin From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 20:35:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC108106566B for ; Fri, 10 Sep 2010 20:35:18 +0000 (UTC) (envelope-from scottj75074@yahoo.com) Received: from web110711.mail.gq1.yahoo.com (web110711.mail.gq1.yahoo.com [67.195.13.218]) by mx1.freebsd.org (Postfix) with SMTP id 8DCC88FC08 for ; Fri, 10 Sep 2010 20:35:18 +0000 (UTC) Received: (qmail 28248 invoked by uid 60001); 10 Sep 2010 20:08:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1284149316; bh=IRD/zEmjJ/7l3YpZDFlifhDTKyla7FMFfD+BFGpn/nw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=LEGyqJZYJFA9qERNCVtU18Q9qQxqMCgzyQoTFIoILtipymlG2Ozev5/DwJL1kUYcR5WEFz+pMqAOillSDQkybgL5UGs/Tg6cql1NmYwIkglNbg2mWmCAFLq0qTMKA0z8LbFFI1T5pdJ04yjqP1FkXF/shBWZ1eiMJYoXV1Vxw3k= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=oQIX2JicF8sy/D/cL9t+l5ipP5ohRNe/o+OBY0oTwqAV3h3wrMlA3t9+X+YYQCnEj8HPfmFqN4Vfmz3GDOJ75jikTpmMEOEDkBoUMicwFoZfB0HAImHs2Cc2OSbvIt9tdzSgZz2jBIFG2d6Ynh4kJJqdgmVHh17eozxyLS9oos8=; Message-ID: <981878.92019.qm@web110711.mail.gq1.yahoo.com> X-YMail-OSG: Qjft_t8VM1kEvEVGFA8rA1VFAo5ebhKEgxu8SCXWwdiTFgW Zo1FXdSH3v1r0NIvQ..5JQWocYC0A1lM.VVHpSjiWHqp_p12HVTINkjRLQv5 xGGAKl4xqds11ZKGlXehJRXqv.08hRJqq4RTUIaeq_ImljACtHdQIyDcg.GP kFj6RPpQhaRQogNJ1SixAuQWB06mb9Q05EHZrWWN2XBvA18pKFyx353LB4a7 _pYeiFZMGJtJKESpVfRS.ezV9MahB5nCitBN5Yj7kLwrTdYfqkVEjRkrWbrL W3B7QAvzoy8Y5ZdI3q39yWlRiwMxGLOsjMsTno72Nab1.mDYk6ezkWRpyiDL HM4JZy_F_9JiDxZRtMuSHXF4G Received: from [99.189.91.206] by web110711.mail.gq1.yahoo.com via HTTP; Fri, 10 Sep 2010 13:08:36 PDT X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950 Date: Fri, 10 Sep 2010 13:08:36 -0700 (PDT) From: Scott Johnson To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: raidz: error during resilver: what next? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 20:35:18 -0000 Hi all, I'm running 8.1-RELEASE on amd64. I'm upgrading a 4-disk raidz to faster drives. While resilvering drive #3, I hit errors on drive #4. The old drives are ada{1,2,3,4}. New drives are label/hitachi{1,2,3,4}. While resilvering drive 3, I got timeouts on ada4 that caused `zpool status` to hang forever. Even `shutdown -r` did not reboot. Hard reset was required. dmesg was full of ada4 timeouts, the most recent of which was several hours old. After power cycle, ada4 appeared fine, reporting no recent smart errors. My current zpool status: pool: tank state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 label/hitachi1 ONLINE 0 0 0 label/hitachi2 ONLINE 0 0 0 replacing DEGRADED 0 0 1 ada3 OFFLINE 0 283 0 label/hitachi3 ONLINE 0 0 0 ada4 ONLINE 0 0 10 The resilvering did not complete, and did not automatically continue after the power cycle. Where do I go from here? I see two choices: 1. Continue the resilver and see what happens. (How do I restart the resilver?) 2. Cancel the replacement. Pull hitachi3 and put back in the (still good) ada3. What commands do I use? My guess: zpool offline tank label/hitachi3 zpool online tank ada3 Thanks in advance for your help. [In other news, while ada4 was timed out, booting from my other zfs pool zroot failed with all kinds of bizarre missing file errors. It seems that one ZFS pool being degraded/corrupted has affected the other ZFS pool too. I now regret going to the trouble of zfs boot.] From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 20:58:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20A661065672 for ; Fri, 10 Sep 2010 20:58:55 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id A89EF8FC13 for ; Fri, 10 Sep 2010 20:58:54 +0000 (UTC) Received: by fxm4 with SMTP id 4so2413576fxm.13 for ; Fri, 10 Sep 2010 13:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=GDmNa9+d/9ha18smn4Dx10eTmN10iC8kVfwF97NZXPA=; b=RfjDt3iLAfj4Xbv5KPU5rIkIW2daXnqHUU++vnRP5nBZfwEaKSLHfH4PSeYUoHzcHL GFzFHoqnU/veUAaK1gqiS6NxvAQoSsaxEvBg/Q9i9Wa4sVQ2sJUogOodek/ZWkUmOzU6 oP9BVxuQkA6FAbfyQgelTOn89Teirggo+pKEI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=oZIr2+QUGVAnvN1jjQtj5RKVFFgDf1DGBqQRLzYw3faAMgDdlUCCf8HAGTX9umQX2o exEbijr/FUf23b7TPKs9QY2xLYAh4a5zbXwsk0wzmpWmIc5bS++QA30yfs3Tf/IUTRjK lQ+Pk4YMbik8twd5z4p+7waj3y548WSQIvZwM= MIME-Version: 1.0 Received: by 10.223.123.79 with SMTP id o15mr776086far.95.1284152333386; Fri, 10 Sep 2010 13:58:53 -0700 (PDT) Received: by 10.223.110.69 with HTTP; Fri, 10 Sep 2010 13:58:53 -0700 (PDT) In-Reply-To: <981878.92019.qm@web110711.mail.gq1.yahoo.com> References: <981878.92019.qm@web110711.mail.gq1.yahoo.com> Date: Fri, 10 Sep 2010 13:58:53 -0700 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: raidz: error during resilver: what next? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 20:58:55 -0000 On Fri, Sep 10, 2010 at 1:08 PM, Scott Johnson wrot= e: > After power cycle, ada4 appeared fine, reporting no recent smart errors. > > My current zpool status: > > =C2=A0pool: tank > state: DEGRADED > status: One or more devices has experienced an error resulting in data > =C2=A0 =C2=A0corruption. =C2=A0Applications may be affected. > action: Restore the file in question if possible. =C2=A0Otherwise restore= the > =C2=A0 =C2=A0entire pool from backup. > =C2=A0 see: http://www.sun.com/msg/ZFS-8000-8A > scrub: none requested > config: > > =C2=A0 =C2=A0NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0STATE =C2=A0 =C2=A0 READ WRITE CKSUM > =C2=A0 =C2=A0tank =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0DEGRADED =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0raidz1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0DEGRADED =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0 =C2=A0label/hitachi1 =C2=A0 =C2=A0ONLINE =C2=A0 =C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0 =C2=A0label/hitachi2 =C2=A0 =C2=A0ONLINE =C2=A0 =C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0 =C2=A0replacing =C2=A0 =C2=A0 =C2=A0 =C2=A0 DEGRADED= =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 1 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ada3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0OFFLINE =C2=A0 =C2=A0 =C2=A00 =C2=A0 283 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0label/hitachi3 =C2=A0ONLINE =C2=A0 =C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 > =C2=A0 =C2=A0 =C2=A0 =C2=A0ada4 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A010 > > The resilvering did not complete, and did not automatically continue afte= r the > power cycle. > > Where do I go from here? I see two choices: > > 1. Continue the resilver and see what happens. (How do I restart the resi= lver?) > > 2. Cancel the replacement. Pull hitachi3 and put back in the (still good)= ada3. > What commands do I use? My guess: > =C2=A0zpool offline tank label/hitachi3 > =C2=A0 > =C2=A0zpool online tank ada3 Number 2 is what I did last time I ran into a similar issue (search -stable/-current/-fs for "cannot replace a replacing drive"). --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 21:24:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CF571065674 for ; Fri, 10 Sep 2010 21:24:45 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 06B618FC13 for ; Fri, 10 Sep 2010 21:24:44 +0000 (UTC) Received: by qwg5 with SMTP id 5so2047046qwg.13 for ; Fri, 10 Sep 2010 14:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=okXs+flvkDhA2T0uhB+zOeTp9dCwjTo813o4pqjSf34=; b=oB/B2fOlv6A5CIORikwZh3yelgyHkkcvGZSg93VaFcfA8RtaQ04xz6r7fdQ+0gm4IW 0daXxO+TRnVaL9bkqZP+KxrGeqfhl41H2L2LbcR/1C69QqHGhmXJESJbo1XKmWqfNtN4 zj25/anAf3wj3BQGmCjDPtLWpSE0o+Gr281LM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=Tv66XbN6H/JIfEl35GQqD1sMxJdjrNr1Pv+8fZPnY1NPr6xwV8lbPARNKYuIE4h1RD B1Ui+6hR+o5jyYwLMEgicNlS5FzwtV9FGIbEx1bsOjTWPZ7GVsYb5je/qB115VXB0sWl Vipy5EeRRbaqGBewUlIVfF3qsipvC6sd2tZ2g= Received: by 10.229.11.8 with SMTP id r8mr852431qcr.236.1284153884128; Fri, 10 Sep 2010 14:24:44 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id t1sm3162537qcs.21.2010.09.10.14.24.41 (version=SSLv3 cipher=RC4-MD5); Fri, 10 Sep 2010 14:24:42 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C8AA218.8090104@DataIX.net> Date: Fri, 10 Sep 2010 17:24:40 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100908 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Graham Todd References: <20100831215915.GE1932@garage.freebsd.pl> <20100902214823.GB2542@garage.freebsd.pl> <4C811954.5040602@3mail4.co.uk> <20100903170241.GF1780@garage.freebsd.pl> <4C8232C6.9020108@DataIX.net> <4C8294B7.3010507@bellanet.org> <20100908090842.GA87388@roberto-al.eurocontrol.fr> <4C878B9A.2000909@bellanet.org> In-Reply-To: <4C878B9A.2000909@bellanet.org> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS v28 is ready for wider testing. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 21:24:45 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/08/2010 09:11, Graham Todd wrote: > On 09/08/10 05:08, Ollivier Robert wrote: >> According to Graham Todd: >>> Thanks to you and other devs for all your careful work integrating ZFS >>> into FreeBSD. It seems amazing that the patch for v28 *almost* even >>> applies to 8-STABLE. >> >> Do you think you could generate a 8-STABLE patch or at least tell us where >> it is failling please? > > thanks to jhell,v - who has already has attempted this and reported back :) :-) > > see: http://bit.ly/d20tML and > > lists.freebsd.org/pipermail/freebsd-current/2010-September/019576.html > > "only 14 of the 293 files that were changed had rejects against a stable/8 > that was patched to ZFSv15 + metaslab_v2 + abe_stat_rrwlock" > > I was unable to quite that far - I think because I applied the earlier > patches in the wrong order. But I am impressed that adding a patch from > CURRENT to go from ZFSv15 to ZFSv28 on the STABLE branch was that close to > applying. Just an update to the above, I was planning on getting to this sooner but currently have higher ordered tasks to accomplish first. If I do happen to get around to it I will put a patchset up somewhere for trial. You should be advised that if this does happen and you upgrade your pool + kernel there is a possibility that 8-STABLE will never see these changes officially so you will be left to your own demise or to upgrade to a 9-CURRENT system in order to have a supported source tree with the zfsv28 patch once that gets in there. With the above 'stated', this is why it has not been my highest priority right now. Regards, - -- jhell,v -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJMiqIYAAoJEJBXh4mJ2FR+dbwH/2l0tUg0TbJrwxJOhEbZmCez kviw4TLRugd3EX/c/Gs3PFlJPaN7dK8PFMpdt3P8ule0Vd/5+b0tAzdiwP58p1g9 AcELf1N83hhwunkVPp5OYW7+dUP7ujhz/tGCdZkKVF3VHx+Tmp4oK3PY6p9yXZuO XZwbSYVq8SeJoZAANbO1/OTQxnCB/N7mCnW4pR09IUnHB4HiYuEuzgkjbWvFbbKR 2yAxLl5JwkKy9maK2E3DVS6icv3DiSGp42s6dp4faSnFbgQrJNpd9LcUbJNiiC0a r+uVgYvo4Wgj8z354CFnXpEct+LKBtXj04ufhzcyvpSoMKm9L1U3u41FeMEwWsE= =AZRS -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 10 23:32:41 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 543131065670 for ; Fri, 10 Sep 2010 23:32:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 115328FC2B for ; Fri, 10 Sep 2010 23:32:40 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApwEAOhcikyDaFvO/2dsb2JhbACDGZ8grwGRcIEigyd0BIog X-IronPort-AV: E=Sophos;i="4.56,348,1280721600"; d="scan'208";a="93507877" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 10 Sep 2010 19:32:39 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 5101BB3F28; Fri, 10 Sep 2010 19:32:39 -0400 (EDT) Date: Fri, 10 Sep 2010 19:32:39 -0400 (EDT) From: Rick Macklem To: David Brodbeck Message-ID: <1229417926.760488.1284161559264.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [24.65.230.102] X-Mailer: Zimbra 6.0.7_GA_2476.RHEL4 (ZimbraWebClient - SAF3 (Mac)/6.0.7_GA_2473.RHEL4_64) Cc: freebsd-fs@freebsd.org Subject: Re: Weird Linux - FreeBSD/ZFS NFSv4 interoperability problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 23:32:41 -0000 > On Wed, Sep 8, 2010 at 9:28 PM, Rick Macklem > wrote: > > Please try the attached patch, which changes the macro so that it > > compares the entire "struct fid". (I think that is safe, since I > > bzero() the entire structure before calling VOP_VPTOFH().) > > > > Again, thanks for sending the packet traces and, hopefully, this > > patch will do the trick, rick > > Rick, after doing some testing I'm pretty confident that this patch > (nfsdport.patch) has solved both the original problem and the I/O > error problem. > > I'm glad I could help. I know how tricky to debug these > interoperability problems can be. > Ok, thanks for testing it. I've just committed it to head. Maybe you'd like to test one other thing? (Not exactly interop. testing)? A couple of people have reported very slow read rates for the NFSv4 client (actually the experimental client, since they see it for NFSv3 too). If you could easily do the following, using a FreeBSD8.1 or newer client: # mount -t nfs -o nfsv4 :/path - cd to anywhere in the mount that has a 100Mbyte+ file # dd if=<100Mbyte+ file> of=/dev/null bs=1m and then report what read rate you see along with the client's machine-arch/# of cores/ram size/network driver used by the mount rick ps: Btw, anyone else who can do this test, it would be appreciated. If you aren't set up for NFSv4, you can do an NFSv3 mount using the exp. client instead. # mount -t newnfs -o nfsv3 :/path From owner-freebsd-fs@FreeBSD.ORG Sat Sep 11 00:50:56 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BADCB106566C for ; Sat, 11 Sep 2010 00:50:56 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 76CC68FC08 for ; Sat, 11 Sep 2010 00:50:56 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ag4FAI9vikyDaFvO/2dsb2JhbACDGZBujjKuL5FlgSKDJ3QEiiA X-IronPort-AV: E=Sophos;i="4.56,349,1280721600"; d="scan'208";a="93512035" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 10 Sep 2010 20:50:55 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 94E12B3F32; Fri, 10 Sep 2010 20:50:55 -0400 (EDT) Date: Fri, 10 Sep 2010 20:50:55 -0400 (EDT) From: Rick Macklem To: gil@vidals.net Message-ID: <1639122954.761765.1284166255552.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [24.65.230.102] X-Mailer: Zimbra 6.0.7_GA_2476.RHEL4 (ZimbraWebClient - SAF3 (Mac)/6.0.7_GA_2473.RHEL4_64) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS write speed performance tuning (NFS) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 00:50:56 -0000 > I'm building my first ZFS NAS and I'm struggling with write speeds > over NFS. > I want to know if it is possible to somehow set NFS to async AND keep > slogging active with my SSD write cache? > > Here is a brief run down of my write speeds on a 1 gigabit LAN and > FreeBSD > 8.1 using ZFS: > > WRITE SPEEDS of 1 GB file: > 55 MB/sec: scp copy test > 28 MB/sec: NFS with async (zil_disable="1") > 12 MB/sec: NFS with sync (default settings) > > 3 x 1TB 7200 RPM SATA HDs > CPU Intel i7 > Controller: LSI 1068e > Controller: ICH10R (for read and write SSDs) > > This NAS will eventually go into my VMware environment and > unfortunately ESX > uses o_sync in their NFS client. AFAIK, there is no way to change the > VMware > NFS client, so I thought of turning off async on the server side and > yet > somehow still use the ZIL SSD Cache for fast writes? > > Looking forward to learning more about ZFS. > > --Gil Vidals > It sounds like you may already be doing this, but the following email suggests that the log be on a separate SSD device. >From freebsd-fs archive:http://docs.FreeBSD.org/cgi/mid.cgi?4ABD56D6.50301 From owner-freebsd-fs@FreeBSD.ORG Sat Sep 11 07:06:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5025106564A; Sat, 11 Sep 2010 07:06:03 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 51CB38FC15; Sat, 11 Sep 2010 07:06:03 +0000 (UTC) Received: by iwn34 with SMTP id 34so3414307iwn.13 for ; Sat, 11 Sep 2010 00:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=6LvB8Uitvu4+6uDl895s5Fb7JnHRgSphYFxHiWR7l/o=; b=QqfpVpvTeqyHzCc7brbbou2yQUIoTrqrhsX6/1W4pqWuXgV7S+A9fp0HjDrA5M1Zcd qwkmZID+bKIlPuU0Mdi0/0kB/cJg/NzRM9A7T8ksf0qCcnpXx1G2h0/LXGEvQejhjrU2 aX4wRuPfCtz2F6CoXBTK0+jE5837uiGw5WvtY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=UO5C+xZa+dCOb16sIQKZ7c/c6tR2aUQNoMsDxo/1jpIxqvNIz81Fyj8Hz5FN8QGxCL 8T9xKqjy3pgb3oUk99/z1xMDnnNH0b2mVb/YemXmUu85kbsjrFOBwyQ8vuBLSRfFJB2c Jn503iUggb6o59x+Up1DR/UZMkgZfcND/KK7s= Received: by 10.231.30.76 with SMTP id t12mr1987312ibc.161.1284188762765; Sat, 11 Sep 2010 00:06:02 -0700 (PDT) Received: from centel.dataix.local (adsl-99-181-137-20.dsl.klmzmi.sbcglobal.net [99.181.137.20]) by mx.google.com with ESMTPS id i6sm3237477iba.20.2010.09.11.00.05.59 (version=SSLv3 cipher=RC4-MD5); Sat, 11 Sep 2010 00:06:00 -0700 (PDT) Sender: "J. Hellenthal" Message-ID: <4C8B2A56.8080809@DataIX.net> Date: Sat, 11 Sep 2010 03:05:58 -0400 From: jhell User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100908 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20100910073912.GC2007@garage.freebsd.pl> In-Reply-To: <20100910073912.GC2007@garage.freebsd.pl> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Swap on ZFS Volume still panics? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 07:06:03 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/10/2010 03:39, Pawel Jakub Dawidek wrote: > On Wed, Sep 08, 2010 at 07:19:40PM +0300, App Deb wrote: >> Reading the wiki guides they mention the official way to add swap volumes on >> ZFS (set org.freebsd:swap=on etc..), >> >> So I thought that it should work now, But I just got a panic in 8.1-RELEASE >> after a heavy memory situation that touched swap for the first time. > > Swap on ZVOL is still not recommended. Where did you find information > that it now recommended? I can't find anything about swap on > http://wiki.freebsd.org/ZFS and on > http://wiki.freebsd.org/ZFSQuickStartGuide there is a note that it is > not recommended. Let me know where it is advised and I'll remove it or > add a note (if the documentation is mine). > >> It is a hassle to add gmirror volumes for swap on full zfs systems, is there >> any workaround for this, or any news when a fix is coming or if it is >> coming? > > I've no plans to fix it, maybe with ZFSv28 it will be easier to fix, but > this is really low priority. If you use full ZFS system the recommended > layout is described here: > > http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/ > >> If the current code procudes guaranteed panics with zfs swap, I think that >> every mention of swap on zfs should be removed from the semi-official wiki >> guides. > > BTW. If this issue will be worked on in the future, it will be useful to > actually see your panic, backtrace and other debug info. To be honest, I > didn't expect it to panic, rather deadlock. Maybe panic is from deadlock > resolver? Hard to say without any debug info. > For reference I have been using swap on ZFS for a while. I recieve no panics when doing so and have the following properties set. exports/swap refreservation 2G local exports/swap primarycache none local exports/swap secondarycache none local exports/swap org.freebsd:swap on local As someone already mentioned, it might help to also change volblocksize property to 4k but I have never had to do that as changing the primary and secondary caches were enough to keep it from panic here. I would be interested to hear if this helped anyone else. Regards, - -- jhell,v -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJMiypWAAoJEJBXh4mJ2FR+OeYH/0jxC6aJlOHVKxL8UCeA+94s V3scx8jDH3ZiqoCno4FLgkzW41E1NUGlvZA4arLe76Fv2/QeU53ZyZD160WcRYax fhOyUQeHPJ2QecSnsmBJS2aJSvl9abdNYMyEhA1YxoApvBRcm7FEvAt3qY8Dj5uQ jJPN4zOxM6CGq6FUWEbAmmtpIKSi/TLNZqvpxN1GcJeyfyFJroswiEisWYq2rkEA MqXEe13DqVdjS0yysyMHsGNqfgpByDRQJ4M/nnGMPHyUlGdItFJR2B3htzVdeb88 69TnEw5rwiztNm/629jyBaDdynjwortP3Jw37Mij6SE/kHS/+kZ6dfCb7eZHPmA= =PAEM -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Sep 11 14:56:55 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 466931065679; Sat, 11 Sep 2010 14:56:55 +0000 (UTC) (envelope-from brucec@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1D46D8FC13; Sat, 11 Sep 2010 14:56:55 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o8BEusXI081129; Sat, 11 Sep 2010 14:56:54 GMT (envelope-from brucec@freefall.freebsd.org) Received: (from brucec@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o8BEusjQ081125; Sat, 11 Sep 2010 14:56:54 GMT (envelope-from brucec) Date: Sat, 11 Sep 2010 14:56:54 GMT Message-Id: <201009111456.o8BEusjQ081125@freefall.freebsd.org> To: brucec@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: brucec@FreeBSD.org Cc: Subject: Re: bin/107692: newfs(8): newfs -O 1 doesn't create consistent filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 14:56:55 -0000 Synopsis: newfs(8): newfs -O 1 doesn't create consistent filesystems Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: brucec Responsible-Changed-When: Sat Sep 11 14:56:34 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=107692 From owner-freebsd-fs@FreeBSD.ORG Sat Sep 11 14:58:30 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B989E106566B; Sat, 11 Sep 2010 14:58:30 +0000 (UTC) (envelope-from brucec@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 90A488FC08; Sat, 11 Sep 2010 14:58:30 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o8BEwUM3081204; Sat, 11 Sep 2010 14:58:30 GMT (envelope-from brucec@freefall.freebsd.org) Received: (from brucec@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o8BEwU1N081200; Sat, 11 Sep 2010 14:58:30 GMT (envelope-from brucec) Date: Sat, 11 Sep 2010 14:58:30 GMT Message-Id: <201009111458.o8BEwU1N081200@freefall.freebsd.org> To: brucec@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: brucec@FreeBSD.org Cc: Subject: Re: kern/150390: zfs deadlock when arcmsr reports drive faulted X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 14:58:30 -0000 Synopsis: zfs deadlock when arcmsr reports drive faulted Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: brucec Responsible-Changed-When: Sat Sep 11 14:57:59 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=150390