From owner-freebsd-fs@FreeBSD.ORG Sun Jun 16 00:59:45 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A9276C1C for ; Sun, 16 Jun 2013 00:59:45 +0000 (UTC) (envelope-from beastie@tardisi.com) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) by mx1.freebsd.org (Postfix) with ESMTP id 6EFDB1014 for ; Sun, 16 Jun 2013 00:59:45 +0000 (UTC) Received: from ip70-179-144-108.fv.ks.cox.net ([70.179.144.108] helo=zen.lhaven.homeip.net) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Uo0cr-000PnN-3T for freebsd-fs@freebsd.org; Sun, 16 Jun 2013 00:15:21 +0000 Received: from lhaven.homeip.net (localhost [127.0.0.1]) by zen.lhaven.homeip.net (8.14.7/8.14.5) with ESMTP id r5G0FESI034168 for ; Sat, 15 Jun 2013 19:15:14 -0500 (CDT) (envelope-from beastie@tardisi.com) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 70.179.144.108 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX19M2bPmYL2XPf9gD4VDaseYirXt7BgLNb4= X-Authentication-Warning: zen.lhaven.homeip.net: Host localhost [127.0.0.1] claimed to be lhaven.homeip.net MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 15 Jun 2013 19:15:09 -0500 From: The BSD Dreamer To: freebsd-fs@freebsd.org Subject: Re: /tmp: change default to mdmfs and/or =?UTF-8?Q?tmpfs=3F?= In-Reply-To: References: Message-ID: <5079bfde7e7154fd266d26c5f0e908c7@lhaven.homeip.net> X-Sender: beastie@tardisi.com User-Agent: Roundcube Webmail/0.8.6 X-Spam-Status: No, score=-2.9 required=4.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on zen.lhaven.homeip.net X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jun 2013 00:59:45 -0000 On 2013-06-10 03:43, Dmitry Morozovsky wrote: > On Mon, 10 Jun 2013, Julian Elischer wrote: > >> > > > what do you think about stop using precious disk or even SSD resources >> > > > for >> > > > /tmp? >> > > > >> > > > For last several (well, maybe over 10?) years I constantly use md >> > > > (swap-backed) >> > > > for /tmp, usually 128M in size, which is enough for most of our server >> > > > needs. >> > > > Some require more, but none more than 512M. Regarding the options, we >> > > > use >> > > > tmpmfs_flags="-S -n -o async -b 4096 -f 512" >> > > [...] >> >> I sometimes use virtual filesystems but there are cases when I am looking >> to >> store HUGE amounts of trace data in /tmp and end up cursing and remounting >> a >> real disk partition. > > Hmm, don't /var/tmp exist for such a task? Well, providing you make /var/tmp sufficiently and separate from the rest of /var....since filling up /var can be bad. Though it seems to be the trend to have /var be in the same fs as /, but its not the way I do my systems. Since before my return to FreeBSD, I had been doing primarily Solaris. I was used to /tmp being tmpfs, with /var/tmp a real fs. Since there are files that are temporary, but you want them to survive across a reboot (such as vi.recover). So, I've been using tmpfs for /tmp on all my FreeBSD servers. I had also a few years back done something similar to all my Linux servers....before they started discussing whether they should be doing such things....I don't know what they had decided. Though I may get to find out when have to setup a new Linux system at home for something that only has Linux or Windows drivers available for it. On most of my FreeBSD systems, I go with 4GB /tmp tmpfs, though for some reason I went with a 8GB /tmp tmpfs on my home workstation. Though my work workstation also has 4G mdmfs /cache filesystem (which I use by starting chromium with '--disk-cache-dir=/cache', I've also wondered about what would be involved in getting profile-sync-daemon working on FreeBSD, but I never seem to get around to it.) I knew from experience that I should set a size for tmpfs, even on my personal systems. At work, somebody would see all that free space in /tmp and fill it up and then wonder why the system ran out of memory..... But, then I ran into the problem on a home server as well, because there was a boinc project that could copy all of the boinc client directory to /tmp for no good reason at the end of a run. And, when people demanded to know why and to have them stop....and got no acceptable response....we voted with our 'feet'. What we have at work for the people that needed somewhere to store HUGE amounts of trace data....was to eventually give them a large /scratch NFS filesystem (or rather mounted as /scratch, since other groups then wanted their own /scratch for their servers...) since it went from store temporarily to view from another system, and then to use to distribute dumps to other systems... Which was intended to be temporary for the particular situation, but hasn't gone away (had to be migrated to new NAS when old NAS was decommissioned -- delete and recreate wasn't acceptable) now appears on more and more systems, and has gotten bigger.... -- Name: Lawrence "The Dreamer" Chen Call: W0LKC Snail: 1530 College Ave, A5 Email: beastie@tardisi.com Manhattan, KS 66502-2768 Blog: http://lawrencechen.net From owner-freebsd-fs@FreeBSD.ORG Sun Jun 16 08:20:13 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6E1706DD for ; Sun, 16 Jun 2013 08:20:13 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id 2963A1F10 for ; Sun, 16 Jun 2013 08:20:12 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id F3977106A7F8; Sun, 16 Jun 2013 10:20:04 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.004382, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 11.4950] X-CRM114-CacheID: sfid-20130616_10200_0F2E3633 X-CRM114-Status: Good ( pR: 11.4950 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Sun Jun 16 10:20:04 2013 X-DSPAM-Confidence: 0.7629 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 51bd7534898104177382390 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00010, >+On, 0.00059, wrote+>, 0.00218, wrote+>, 0.00218, On+Tue, 0.00242, >+>>, 0.00321, that+>, 0.00348, >+>, 0.00371, References*fsn.hu>, 0.00371, >+with, 0.00463, wrote, 0.00535, wrote, 0.00535, ZFS, 0.00555, ZFS, 0.00555, Subject*ZFS, 0.00617, From*Attila, 0.00617, get+>, 0.00922, Received*online.co.hu+[195.228.243.99]), 0.01000, Subject*An, 0.99000, that+you're, 0.01000, hu>+wrote, 0.01000, Received*[195.228.243.99]), 0.01000, >+would, 0.01000, UFS, 0.01000, 80%25, 0.99000, Subject*higher, 0.99000, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 8C83D106A7D6; Sun, 16 Jun 2013 10:20:00 +0200 (CEST) Message-ID: <51BD7530.9060000@fsn.hu> Date: Sun, 16 Jun 2013 10:20:00 +0200 From: Attila Nagy MIME-Version: 1.0 To: Mark Felder Subject: Re: An order of magnitude higher IOPS needed with ZFS than UFS References: <51B79023.5020109@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jun 2013 08:20:13 -0000 On 06/12/13 13:40, Mark Felder wrote: > On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy wrote: > >> BTW, the file systems are 77-78% full according to df (so ZFS holds >> more, because UFS is -m 8). > > ZFS write performance can begin to drop pretty badly when you get > around 80% full. I've not seen any benchmarks showing an improvement > with a very fast and large ZIL or tons of memory, but I'd expect that > would help significantly. Just note that you're right at the edge > where performance gets impacted. I'm aware of that, even thought about it, that's why I wrote the disk free percents into my mail. I see a completely normal write pattern, but a lot of reads, so I guess ZIL wouldn't help here (maybe L2ARC or more memory, if something doesn't fit into it). BTW, I think having the allowed fill value this low (UFS has the breakdown point much higher) is of a really bad design... From owner-freebsd-fs@FreeBSD.ORG Sun Jun 16 08:44:01 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id CB66EB14; Sun, 16 Jun 2013 08:44:01 +0000 (UTC) (envelope-from freebsd@psconsult.nl) Received: from mx1.psconsult.nl (unknown [IPv6:2001:7b8:30f:e0::5059:ee8a]) by mx1.freebsd.org (Postfix) with ESMTP id 87F931F8F; Sun, 16 Jun 2013 08:44:01 +0000 (UTC) Received: from mx1.psconsult.nl (mx1.hvnu.psconsult.nl [46.44.189.154]) by mx1.psconsult.nl (8.14.5/8.14.4) with ESMTP id r5G8hs3I018665 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 16 Jun 2013 10:43:59 +0200 (CEST) (envelope-from freebsd@psconsult.nl) Received: (from paul@localhost) by mx1.psconsult.nl (8.14.5/8.14.4/Submit) id r5G8hsRi018664; Sun, 16 Jun 2013 10:43:54 +0200 (CEST) (envelope-from freebsd@psconsult.nl) X-Authentication-Warning: mx1.psconsult.nl: paul set sender to freebsd@psconsult.nl using -f Date: Sun, 16 Jun 2013 10:43:54 +0200 From: Paul Schenkeveld To: freebsd-fs@freebsd.org, fs@freebsd.org Subject: Re: Changing the default for ZFS atime to off? Message-ID: <20130616084354.GA23945@psconsult.nl> References: <16FEF774EE8E4100AD2CAEC65276A49D@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16FEF774EE8E4100AD2CAEC65276A49D@multiplay.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jun 2013 08:44:01 -0000 On Sat, Jun 08, 2013 at 07:54:04PM +0100, Steven Hartland wrote: > One of the first changes we make here when installing machines > here to changing atime=off on all ZFS pool roots. > > I know there are a few apps which can rely on atime updates > such as qmail and possibly postfix, but those seem like special > cases for which admins should enable atime instead of the other > way round. > > This is going to of particular interest for flash based storage > which should avoid unnessacary writes to reduce wear, but it will > also help improve performance in general. > > So what do people think is it worth considering changing the > default from atime=on to atime=off moving forward? I'm very strongly against making atime=off the default when creating pools/datasets using zpool(8) and zfs(8): - POLA vialation: pools/datasets created with older version of the commands have atime=on by default, with newer versions of the command have atime=off by default. How can I see what version of the commands I have? So sysadmins must not only be aware of the existence of the switch but also either explicitly specify what they want on *every* invocation or check afterwards whether the default did what they wanted. (I administer dozens of FreeBSD systems at different customers, because of the independence of each customer the all run different versions of FreeBSD). - It's against POSIX (according to a reply by Bruce Evans, didn't check myself). - Sysadmins who type these commands should know what they are doing and can easily add -o atime=off if they want that behaviour. - If you write a script or program that invokes these commands you can always explicitly add the -o atime=whatever arguments to make your tool do exactly what you want. I would not be against making atime=off the default for menu driven installers (bsdinstall, PCBSD installer) and tools that take a config file for which a template exists, provided that they tell the user that this is the default and provide with a choise to switch atime to on there and then, e.g. make atime=off a checkbox questing that's checked by default of add it to he config file template with a proper comment. With kind regards, Paul Schenkeveld From owner-freebsd-fs@FreeBSD.ORG Mon Jun 17 11:06:44 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5776A1A3 for ; Mon, 17 Jun 2013 11:06:44 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 3B4511C06 for ; Mon, 17 Jun 2013 11:06:44 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5HB6ibf012704 for ; Mon, 17 Jun 2013 11:06:44 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5HB6hCW012702 for freebsd-fs@FreeBSD.org; Mon, 17 Jun 2013 11:06:43 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 17 Jun 2013 11:06:43 GMT Message-Id: <201306171106.r5HB6hCW012702@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jun 2013 11:06:44 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o bin/178996 fs [zfs] [patch] error in message with zfs mount -> there o kern/178854 fs [ufs] FreeBSD kernel crash in UFS o kern/178713 fs [nfs] [patch] Correct WebNFS support in NFS server and o kern/178412 fs [smbfs] Coredump when smbfs mounted o kern/178388 fs [zfs] [patch] allow up to 8MB recordsize o kern/178349 fs [zfs] zfs scrub on deduped data could be much less see o kern/178329 fs [zfs] extended attributes leak o kern/178238 fs [nullfs] nullfs don't release i-nodes on unlink. f kern/178231 fs [nfs] 8.3 nfsv4 client reports "nfsv4 client/server pr o kern/178103 fs [kernel] [nfs] [patch] Correct support of index files o kern/177985 fs [zfs] disk usage problem when copying from one zfs dat o kern/177971 fs [nfs] FreeBSD 9.1 nfs client dirlist problem w/ nfsv3, o kern/177966 fs [zfs] resilver completes but subsequent scrub reports o kern/177658 fs [ufs] FreeBSD panics after get full filesystem with uf o kern/177536 fs [zfs] zfs livelock (deadlock) with high write-to-disk o kern/177445 fs [hast] HAST panic o kern/177240 fs [zfs] zpool import failed with state UNAVAIL but all d o kern/176978 fs [zfs] [panic] zfs send -D causes "panic: System call i o kern/176857 fs [softupdates] [panic] 9.1-RELEASE/amd64/GENERIC panic o bin/176253 fs zpool(8): zfs pool indentation is misleading/wrong o kern/176141 fs [zfs] sharesmb=on makes errors for sharenfs, and still o kern/175950 fs [zfs] Possible deadlock in zfs after long uptime o kern/175897 fs [zfs] operations on readonly zpool hang o kern/175179 fs [zfs] ZFS may attach wrong device on move o kern/175071 fs [ufs] [panic] softdep_deallocate_dependencies: unrecov o kern/174372 fs [zfs] Pagefault appears to be related to ZFS o kern/174315 fs [zfs] chflags uchg not supported o kern/174310 fs [zfs] root point mounting broken on CURRENT with multi o kern/174279 fs [ufs] UFS2-SU+J journal and filesystem corruption o kern/173830 fs [zfs] Brain-dead simple change to ZFS error descriptio o kern/173718 fs [zfs] phantom directory in zraid2 pool f kern/173657 fs [nfs] strange UID map with nfsuserd o kern/173363 fs [zfs] [panic] Panic on 'zpool replace' on readonly poo o kern/173136 fs [unionfs] mounting above the NFS read-only share panic o kern/172942 fs [smbfs] Unmounting a smb mount when the server became o kern/172348 fs [unionfs] umount -f of filesystem in use with readonly o kern/172334 fs [unionfs] unionfs permits recursive union mounts; caus o kern/171626 fs [tmpfs] tmpfs should be noisier when the requested siz o kern/171415 fs [zfs] zfs recv fails with "cannot receive incremental o kern/170945 fs [gpt] disk layout not portable between direct connect o bin/170778 fs [zfs] [panic] FreeBSD panics randomly o kern/170680 fs [nfs] Multiple NFS Client bug in the FreeBSD 7.4-RELEA o kern/170497 fs [xfs][panic] kernel will panic whenever I ls a mounted o kern/169945 fs [zfs] [panic] Kernel panic while importing zpool (afte o kern/169480 fs [zfs] ZFS stalls on heavy I/O o kern/169398 fs [zfs] Can't remove file with permanent error o kern/169339 fs panic while " : > /etc/123" o kern/169319 fs [zfs] zfs resilver can't complete o kern/168947 fs [nfs] [zfs] .zfs/snapshot directory is messed up when o kern/168942 fs [nfs] [hang] nfsd hangs after being restarted (not -HU o kern/168158 fs [zfs] incorrect parsing of sharenfs options in zfs (fs o kern/167979 fs [ufs] DIOCGDINFO ioctl does not work on 8.2 file syste o kern/167977 fs [smbfs] mount_smbfs results are differ when utf-8 or U o kern/167688 fs [fusefs] Incorrect signal handling with direct_io o kern/167685 fs [zfs] ZFS on USB drive prevents shutdown / reboot o kern/167612 fs [portalfs] The portal file system gets stuck inside po o kern/167272 fs [zfs] ZFS Disks reordering causes ZFS to pick the wron o kern/167260 fs [msdosfs] msdosfs disk was mounted the second time whe o kern/167109 fs [zfs] [panic] zfs diff kernel panic Fatal trap 9: gene o kern/167105 fs [nfs] mount_nfs can not handle source exports wiht mor o kern/167067 fs [zfs] [panic] ZFS panics the server o kern/167065 fs [zfs] boot fails when a spare is the boot disk o kern/167048 fs [nfs] [patch] RELEASE-9 crash when using ZFS+NULLFS+NF o kern/166912 fs [ufs] [panic] Panic after converting Softupdates to jo o kern/166851 fs [zfs] [hang] Copying directory from the mounted UFS di o kern/166477 fs [nfs] NFS data corruption. o kern/165950 fs [ffs] SU+J and fsck problem o kern/165521 fs [zfs] [hang] livelock on 1 Gig of RAM with zfs when 31 o kern/165392 fs Multiple mkdir/rmdir fails with errno 31 o kern/165087 fs [unionfs] lock violation in unionfs o kern/164472 fs [ufs] fsck -B panics on particular data inconsistency o kern/164370 fs [zfs] zfs destroy for snapshot fails on i386 and sparc o kern/164261 fs [nullfs] [patch] fix panic with NFS served from NULLFS o kern/164256 fs [zfs] device entry for volume is not created after zfs o kern/164184 fs [ufs] [panic] Kernel panic with ufs_makeinode o kern/163801 fs [md] [request] allow mfsBSD legacy installed in 'swap' o kern/163770 fs [zfs] [hang] LOR between zfs&syncer + vnlru leading to o kern/163501 fs [nfs] NFS exporting a dir and a subdir in that dir to o kern/162944 fs [coda] Coda file system module looks broken in 9.0 o kern/162860 fs [zfs] Cannot share ZFS filesystem to hosts with a hyph o kern/162751 fs [zfs] [panic] kernel panics during file operations o kern/162591 fs [nullfs] cross-filesystem nullfs does not work as expe o kern/162519 fs [zfs] "zpool import" relies on buggy realpath() behavi o kern/161968 fs [zfs] [hang] renaming snapshot with -r including a zvo o kern/161864 fs [ufs] removing journaling from UFS partition fails on o bin/161807 fs [patch] add option for explicitly specifying metadata o kern/161579 fs [smbfs] FreeBSD sometimes panics when an smb share is o kern/161533 fs [zfs] [panic] zfs receive panic: system ioctl returnin o kern/161438 fs [zfs] [panic] recursed on non-recursive spa_namespace_ o kern/161424 fs [nullfs] __getcwd() calls fail when used on nullfs mou o kern/161280 fs [zfs] Stack overflow in gptzfsboot o kern/161205 fs [nfs] [pfsync] [regression] [build] Bug report freebsd o kern/161169 fs [zfs] [panic] ZFS causes kernel panic in dbuf_dirty o kern/161112 fs [ufs] [lor] filesystem LOR in FreeBSD 9.0-BETA3 o kern/160893 fs [zfs] [panic] 9.0-BETA2 kernel panic f kern/160860 fs [ufs] Random UFS root filesystem corruption with SU+J o kern/160801 fs [zfs] zfsboot on 8.2-RELEASE fails to boot from root-o o kern/160790 fs [fusefs] [panic] VPUTX: negative ref count with FUSE o kern/160777 fs [zfs] [hang] RAID-Z3 causes fatal hang upon scrub/impo o kern/160706 fs [zfs] zfs bootloader fails when a non-root vdev exists o kern/160591 fs [zfs] Fail to boot on zfs root with degraded raidz2 [r o kern/160410 fs [smbfs] [hang] smbfs hangs when transferring large fil o kern/160283 fs [zfs] [patch] 'zfs list' does abort in make_dataset_ha o kern/159930 fs [ufs] [panic] kernel core o kern/159402 fs [zfs][loader] symlinks cause I/O errors o kern/159357 fs [zfs] ZFS MAXNAMELEN macro has confusing name (off-by- o kern/159356 fs [zfs] [patch] ZFS NAME_ERR_DISKLIKE check is Solaris-s o kern/159351 fs [nfs] [patch] - divide by zero in mountnfs() o kern/159251 fs [zfs] [request]: add FLETCHER4 as DEDUP hash option o kern/159077 fs [zfs] Can't cd .. with latest zfs version o kern/159048 fs [smbfs] smb mount corrupts large files o kern/159045 fs [zfs] [hang] ZFS scrub freezes system o kern/158839 fs [zfs] ZFS Bootloader Fails if there is a Dead Disk o kern/158802 fs amd(8) ICMP storm and unkillable process. o kern/158231 fs [nullfs] panic on unmounting nullfs mounted over ufs o f kern/157929 fs [nfs] NFS slow read o kern/157399 fs [zfs] trouble with: mdconfig force delete && zfs strip o kern/157179 fs [zfs] zfs/dbuf.c: panic: solaris assert: arc_buf_remov o kern/156797 fs [zfs] [panic] Double panic with FreeBSD 9-CURRENT and o kern/156781 fs [zfs] zfs is losing the snapshot directory, p kern/156545 fs [ufs] mv could break UFS on SMP systems o kern/156193 fs [ufs] [hang] UFS snapshot hangs && deadlocks processes o kern/156039 fs [nullfs] [unionfs] nullfs + unionfs do not compose, re o kern/155615 fs [zfs] zfs v28 broken on sparc64 -current o kern/155587 fs [zfs] [panic] kernel panic with zfs p kern/155411 fs [regression] [8.2-release] [tmpfs]: mount: tmpfs : No o kern/155199 fs [ext2fs] ext3fs mounted as ext2fs gives I/O errors o bin/155104 fs [zfs][patch] use /dev prefix by default when importing o kern/154930 fs [zfs] cannot delete/unlink file from full volume -> EN o kern/154828 fs [msdosfs] Unable to create directories on external USB o kern/154491 fs [smbfs] smb_co_lock: recursive lock for object 1 p kern/154228 fs [md] md getting stuck in wdrain state o kern/153996 fs [zfs] zfs root mount error while kernel is not located o kern/153753 fs [zfs] ZFS v15 - grammatical error when attempting to u o kern/153716 fs [zfs] zpool scrub time remaining is incorrect o kern/153695 fs [patch] [zfs] Booting from zpool created on 4k-sector o kern/153680 fs [xfs] 8.1 failing to mount XFS partitions o kern/153418 fs [zfs] [panic] Kernel Panic occurred writing to zfs vol o kern/153351 fs [zfs] locking directories/files in ZFS o bin/153258 fs [patch][zfs] creating ZVOLs requires `refreservation' s kern/153173 fs [zfs] booting from a gzip-compressed dataset doesn't w o bin/153142 fs [zfs] ls -l outputs `ls: ./.zfs: Operation not support o kern/153126 fs [zfs] vdev failure, zpool=peegel type=vdev.too_small o kern/152022 fs [nfs] nfs service hangs with linux client [regression] o kern/151942 fs [zfs] panic during ls(1) zfs snapshot directory o kern/151905 fs [zfs] page fault under load in /sbin/zfs o bin/151713 fs [patch] Bug in growfs(8) with respect to 32-bit overfl o kern/151648 fs [zfs] disk wait bug o kern/151629 fs [fs] [patch] Skip empty directory entries during name o kern/151330 fs [zfs] will unshare all zfs filesystem after execute a o kern/151326 fs [nfs] nfs exports fail if netgroups contain duplicate o kern/151251 fs [ufs] Can not create files on filesystem with heavy us o kern/151226 fs [zfs] can't delete zfs snapshot o kern/150503 fs [zfs] ZFS disks are UNAVAIL and corrupted after reboot o kern/150501 fs [zfs] ZFS vdev failure vdev.bad_label on amd64 o kern/150390 fs [zfs] zfs deadlock when arcmsr reports drive faulted o kern/150336 fs [nfs] mountd/nfsd became confused; refused to reload n o kern/149208 fs mksnap_ffs(8) hang/deadlock o kern/149173 fs [patch] [zfs] make OpenSolaris installa o kern/149015 fs [zfs] [patch] misc fixes for ZFS code to build on Glib o kern/149014 fs [zfs] [patch] declarations in ZFS libraries/utilities o kern/149013 fs [zfs] [patch] make ZFS makefiles use the libraries fro o kern/148504 fs [zfs] ZFS' zpool does not allow replacing drives to be o kern/148490 fs [zfs]: zpool attach - resilver bidirectionally, and re o kern/148368 fs [zfs] ZFS hanging forever on 8.1-PRERELEASE o kern/148138 fs [zfs] zfs raidz pool commands freeze o kern/147903 fs [zfs] [panic] Kernel panics on faulty zfs device o kern/147881 fs [zfs] [patch] ZFS "sharenfs" doesn't allow different " o kern/147420 fs [ufs] [panic] ufs_dirbad, nullfs, jail panic (corrupt o kern/146941 fs [zfs] [panic] Kernel Double Fault - Happens constantly o kern/146786 fs [zfs] zpool import hangs with checksum errors o kern/146708 fs [ufs] [panic] Kernel panic in softdep_disk_write_compl o kern/146528 fs [zfs] Severe memory leak in ZFS on i386 o kern/146502 fs [nfs] FreeBSD 8 NFS Client Connection to Server s kern/145712 fs [zfs] cannot offline two drives in a raidz2 configurat o kern/145411 fs [xfs] [panic] Kernel panics shortly after mounting an f bin/145309 fs bsdlabel: Editing disk label invalidates the whole dev o kern/145272 fs [zfs] [panic] Panic during boot when accessing zfs on o kern/145246 fs [ufs] dirhash in 7.3 gratuitously frees hashes when it o kern/145238 fs [zfs] [panic] kernel panic on zpool clear tank o kern/145229 fs [zfs] Vast differences in ZFS ARC behavior between 8.0 o kern/145189 fs [nfs] nfsd performs abysmally under load o kern/144929 fs [ufs] [lor] vfs_bio.c + ufs_dirhash.c p kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization s kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o bin/143572 fs [zfs] zpool(1): [patch] The verbose output from iostat o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs p bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/138662 fs [panic] ffs_blkfree: freeing free block o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic p kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis p kern/133174 fs [msdosfs] [patch] msdosfs must support multibyte inter o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/127787 fs [lor] [ufs] Three LORs: vfslock/devfs/vfslock, ufs/vfs o bin/127270 fs fsck_msdosfs(8) may crash if BytesPerSec is zero o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file o kern/125895 fs [ffs] [panic] kernel: panic: ffs_blkfree: freeing free s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o kern/118318 fs [nfs] NFS server hangs under special circumstances o bin/118249 fs [ufs] mv(1): moving a directory changes its mtime o kern/118126 fs [nfs] [patch] Poor NFS server write performance o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o kern/117954 fs [ufs] dirhash on very large directories blocks the mac o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o conf/116931 fs lack of fsck_cd9660 prevents mounting iso images with o kern/116583 fs [ffs] [hang] System freezes for short time when using o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106107 fs [ufs] left-over fsck_snapshot after unfinished backgro o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes s bin/97498 fs [request] newfs(8) has no option to clear the first 12 o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [cd9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o bin/94810 fs fsck(8) incorrectly reports 'file system marked clean' o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88555 fs [panic] ffs_blkfree: freeing free frag on AMD 64 o bin/87966 fs [patch] newfs(8): introduce -A flag for newfs to enabl o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o bin/85494 fs fsck_ffs: unchecked use of cg_inosused macro etc. o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o bin/74779 fs Background-fsck checks one filesystem twice and omits o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o bin/70600 fs fsck(8) throws files away when it can't grow lost+foun o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o bin/27687 fs fsck(8) wrapper is not properly passing options to fsc o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 317 problems total. From owner-freebsd-fs@FreeBSD.ORG Tue Jun 18 13:46:24 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6B7DED61; Tue, 18 Jun 2013 13:46:24 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 44B551AA4; Tue, 18 Jun 2013 13:46:24 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5IDkOxn075954; Tue, 18 Jun 2013 13:46:24 GMT (envelope-from ae@freefall.freebsd.org) Received: (from ae@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5IDkOGt075953; Tue, 18 Jun 2013 13:46:24 GMT (envelope-from ae) Date: Tue, 18 Jun 2013 13:46:24 GMT Message-Id: <201306181346.r5IDkOGt075953@freefall.freebsd.org> To: ae@FreeBSD.org, freebsd-fs@FreeBSD.org, freebsd-geom@FreeBSD.org From: ae@FreeBSD.org Subject: Re: bin/161807: [patch] add option for explicitly specifying metadata version to geli(8) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jun 2013 13:46:24 -0000 Synopsis: [patch] add option for explicitly specifying metadata version to geli(8) Responsible-Changed-From-To: freebsd-fs->freebsd-geom Responsible-Changed-By: ae Responsible-Changed-When: Tue Jun 18 13:45:52 UTC 2013 Responsible-Changed-Why: Reassign to geom team. http://www.freebsd.org/cgi/query-pr.cgi?pr=161807 From owner-freebsd-fs@FreeBSD.ORG Wed Jun 19 01:11:25 2013 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 714F053C; Wed, 19 Jun 2013 01:11:25 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 38FC51F7C; Wed, 19 Jun 2013 01:11:25 +0000 (UTC) Received: from lstewart1.loshell.room52.net (ppp59-167-184-191.static.internode.on.net [59.167.184.191]) by lauren.room52.net (Postfix) with ESMTPSA id 0449B7E820; Wed, 19 Jun 2013 11:11:24 +1000 (EST) Message-ID: <51C1053B.6090709@freebsd.org> Date: Wed, 19 Jun 2013 11:11:23 +1000 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120613 Thunderbird/13.0 MIME-Version: 1.0 To: The BSD Dreamer Subject: Re: ZFS triggered 9-STABLE r246646 panic "vdrop: holdcnt 0" References: <513E8E95.6010802@freebsd.org> <51BA6941.7040909@tardisi.com> In-Reply-To: <51BA6941.7040909@tardisi.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: "freebsd-fs@freebsd.org" , will@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jun 2013 01:11:25 -0000 [ccing Will as he showed an interest in this on IRC when I first reported the panic] On 06/14/13 10:52, The BSD Dreamer wrote: > On 03/11/2013 21:10, Lawrence Stewart wrote: >> Hi all, >> >> I got this panic yesterday. I haven't seen it before (or since), but I >> have the crashdump and kernel here if there's additional information I >> can provide that would be useful in finding the cause. >> >> The machine runs ZFS exclusively and was under quite heavy CPU and IO >> load at the time of the crash as I was compiling in a VirtualBox VM and >> on the host itself, as well as running a full KDE desktop environment. >> I'm fairly certain the machine was not swapping at the time of the crash. >> >> lstewart@lstewart> uname -a >> FreeBSD lstewart 9.1-STABLE FreeBSD 9.1-STABLE #8 r246646M: Mon Feb 11 >> 14:57:13 EST 2013 >> root@lstewart:/usr/obj/usr/src/sys/LSTEWART-DESKTOP amd64 >> >> lstewart@lstewart> sudo kgdb /boot/kernel/kernel /var/crash/vmcore.0 >> >> [...] >> >> (kgdb) bt >> #0 doadump (textdump=) at pcpu.h:229 >> #1 0xffffffff808e5824 in kern_reboot (howto=260) at >> /usr/src/sys/kern/kern_shutdown.c:448 >> #2 0xffffffff808e5d27 in panic (fmt=0x1
) at >> /usr/src/sys/kern/kern_shutdown.c:636 >> #3 0xffffffff8097a71e in vdropl (vp=) at >> /usr/src/sys/kern/vfs_subr.c:2465 >> #4 0xffffffff80b4da2b in vm_page_alloc (object=0xffffffff8132c000, >> pindex=143696, req=32) at /usr/src/sys/vm/vm_page.c:1569 >> #5 0xffffffff80b3f312 in kmem_back (map=0xfffffe00020000e8, >> addr=18446743524542296064, size=131072, flags=705200752) >> at /usr/src/sys/vm/vm_kern.c:361 > > I just came home to find that my system had panic'd (around > 11:30am)....and this was the only FreeBSD 9 'panic: vdrop: holdcnt: 0' > that I found. I still have my kernel and crash dump lying around if there's any additional useful information which could be extracted to help find the cause of this panic. Doesn't appear to be very common though so the code path which is failing to grab/hold a reference must only be used infrequently. Cheers, Lawrence From owner-freebsd-fs@FreeBSD.ORG Wed Jun 19 14:40:01 2013 Return-Path: Delivered-To: freebsd-fs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9CA92DD0 for ; Wed, 19 Jun 2013 14:40:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 7201016F6 for ; Wed, 19 Jun 2013 14:40:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r5JEe1to031063 for ; Wed, 19 Jun 2013 14:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r5JEe1fN031062; Wed, 19 Jun 2013 14:40:01 GMT (envelope-from gnats) Date: Wed, 19 Jun 2013 14:40:01 GMT Message-Id: <201306191440.r5JEe1fN031062@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: "Steven Hartland" Subject: Re: kern/161968: [zfs] [hang] renaming snapshot with -r including a zvol snapshot causes total ZFS freeze/lockup X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Steven Hartland List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jun 2013 14:40:01 -0000 The following reply was made to PR kern/161968; it has been noted by GNATS. From: "Steven Hartland" To: , "Peter Maloney" Cc: Subject: Re: kern/161968: [zfs] [hang] renaming snapshot with -r including a zvol snapshot causes total ZFS freeze/lockup Date: Wed, 19 Jun 2013 15:23:13 +0100 I've reproduced this here, the cause is a live lock between zvols geom actions and ZFS itself between the two locks: db> show sleepchain thread 100553 (pid 6, txg_thread_enter) blocked on sx "spa_namespace_lock" XLOCK thread 100054 (pid 2, g_event) blocked on sx "dp->dp_config_rwlock" XLOCK db> Tracing pid 2 tid 100054 td 0xffffff001c1d4470 sched_switch() at sched_switch+0x153 mi_switch() at mi_switch+0x1f8 sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1e2 _sx_slock() at _sx_slock+0xc9 dsl_dir_open_spa() at dsl_dir_open_spa+0xab dsl_dataset_hold() at dsl_dataset_hold+0x3b dsl_dataset_own() at dsl_dataset_own+0x2f dmu_objset_own() at dmu_objset_own+0x36 zvol_first_open() at zvol_first_open+0x34 zvol_geom_access() at zvol_geom_access+0x2df g_access() at g_access+0x1ba g_part_taste() at g_part_taste+0xc4 g_new_provider_event() at g_new_provider_event+0xaa g_run_events() at g_run_events+0x250 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff92070a2bb0, rbp = 0 --- db> bt 100553 Tracing pid 6 tid 100553 td 0xffffff002c2308e0 sched_switch() at sched_switch+0x153 mi_switch() at mi_switch+0x1f8 sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x296 _sx_xlock() at _sx_xlock+0xb7 zvol_rename_minors() at zvol_rename_minors+0x75 dsl_dataset_snapshot_rename_sync() at dsl_dataset_snapshot_rename_sync+0x141 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x14e dsl_pool_sync() at dsl_pool_sync+0x47d spa_sync() at spa_sync+0x34a txg_sync_thread() at txg_sync_thread+0x139 fork_exit() at fork_exit+0x135 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff920e61abb0, rbp = 0 --- The following steps recreate the issue on stable/8 r251496 gpart create -s GPT da3 gpart add -t freebsd-zfs da3 zpool create -f testpool da3p1 zfs create -V 150m testpool/testvol zfs snapshot -r testpool@snap zfs rename -r testpool@snap testpool@snap-new I've been unable to reproduce on current r251471. I'm not sure is this is due to a timing issue due to the significant changes in ZFS sync tasks in current or if the issue really doesn't exist any more. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Thu Jun 20 10:30:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9E38BF0F for ; Thu, 20 Jun 2013 10:30:36 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) by mx1.freebsd.org (Postfix) with ESMTP id 210A71B07 for ; Thu, 20 Jun 2013 10:30:34 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z5so5634814lbh.35 for ; Thu, 20 Jun 2013 03:30:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=VNPds05J4zmgi24zMe6tsGem8fZoitGCE/vIT6m2u00=; b=S7K5cyfftusf2gkxl7bUMy0ZtViKq95DOEkYdg8mKtTdWopR+7OebtRkMtc8mZhOFf HWUvNy8F7F6V12YZ/IgBBk9eQ4pnCuBomM4CDFNRgfYTZMo5iF3Wob711g6fuopP1IVe jrDrLuifSfOOxpjiXyvGjVo6aqo2wWq7T28MpED8PnQTmD1oupbxAYPC1bKfYvxEDVZU qnqy7it/u2CnuVqMBrEbQn5ySIydRp2RgZ9dLWoHUHUlf36TGaebvyPre3nx7lTduZ15 el8ikf1r/ffd0QVJYTQ4ZgfJhxKa4Fdg3Ik7H8/RzusdH44KCW24lgbTP88bGHRb0wYw znBA== X-Received: by 10.112.11.162 with SMTP id r2mr5211703lbb.41.1371724233601; Thu, 20 Jun 2013 03:30:33 -0700 (PDT) Received: from grey.home.unixconn.com (h-75-17.a183.priv.bahnhof.se. [46.59.75.17]) by mx.google.com with ESMTPSA id m14sm128285lbl.1.2013.06.20.03.30.31 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 20 Jun 2013 03:30:32 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> Date: Thu, 20 Jun 2013 12:30:30 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <09717048-12BE-474B-9B20-F5E72D00152E@alumni.chalmers.se> References: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> <20130606223911.GA45807@icarus.home.lan> <20130606233417.GA46506@icarus.home.lan> <61E414CF-FCD3-42BB-9533-A40EA934DB99@alumni.chalmers.se> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1508) X-Gm-Message-State: ALoCoQnX/+rIzcOZ/6D4TZezGslj3TFWg/Z9CSo5dDj6cYIygSP0yvCh9mWnkXjJHed3wKkylvIs Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Jun 2013 10:30:36 -0000 Well, I'm back to square one. After some uptime and successful import/export from one node to another, = I eventually got 'metadata corruption'. I had no problem with import/export while for ex. rebooting master-node = (nfs1), but not THIS time. Metdata got corrupted while rebooting master-node?? Any ideas?=20 [root@nfs1 ~]# zpool import pool: jbod id: 7663925948774378610 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-72 config: jbod FAULTED corrupted data raidz3-0 ONLINE da3 ONLINE da4 ONLINE da5 ONLINE da6 ONLINE da7 ONLINE da8 ONLINE da9 ONLINE da10 ONLINE da11 ONLINE da12 ONLINE cache da13s2 da14s2 logs mirror-1 ONLINE da13s1 ONLINE da14s1 ONLINE [root@nfs1 ~]# zpool import jbod cannot import 'jbod': I/O error Destroy and re-create the pool from a backup source. [root@nfs1 ~]# On 11 jun 2013, at 10:46, mxb wrote: >=20 > Thanks everyone whom replied. > Removing local L2ARC cache disks (da1,da2) indeed showed to be a cure = to my problem. >=20 > Next is to test with add/remove after import/export as Jeremy = suggested. >=20 > //mxb >=20 > On 7 jun 2013, at 01:34, Jeremy Chadwick wrote: >=20 >> On Fri, Jun 07, 2013 at 12:51:14AM +0200, mxb wrote: >>>=20 >>> Sure, script is not perfects yet and does not handle many of stuff, = but moving highlight from zpool import/export to the script itself not = that >>> clever,as this works most of the time. >>>=20 >>> Question is WHY ZFS corrupts metadata then it should not. Sometimes. >>> I'v seen stale of zpool then manually importing/exporting pool. >>>=20 >>>=20 >>> On 7 jun 2013, at 00:39, Jeremy Chadwick wrote: >>>=20 >>>> On Fri, Jun 07, 2013 at 12:12:39AM +0200, mxb wrote: >>>>>=20 >>>>> Then MASTER goes down, CARP on the second node goes MASTER = (devd.conf, and script for lifting): >>>>>=20 >>>>> root@nfs2:/root # cat /etc/devd.conf >>>>>=20 >>>>>=20 >>>>> notify 30 { >>>>> match "system" "IFNET"; >>>>> match "subsystem" "carp0"; >>>>> match "type" "LINK_UP"; >>>>> action "/etc/zfs_switch.sh active"; >>>>> }; >>>>>=20 >>>>> notify 30 { >>>>> match "system" "IFNET"; >>>>> match "subsystem" "carp0"; >>>>> match "type" "LINK_DOWN"; >>>>> action "/etc/zfs_switch.sh backup"; >>>>> }; >>>>>=20 >>>>> root@nfs2:/root # cat /etc/zfs_switch.sh >>>>> #!/bin/sh >>>>>=20 >>>>> DATE=3D`date +%Y%m%d` >>>>> HOSTNAME=3D`hostname` >>>>>=20 >>>>> ZFS_POOL=3D"jbod" >>>>>=20 >>>>>=20 >>>>> case $1 in >>>>> active) >>>>> echo "Switching to ACTIVE and importing ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to ACTIVE' root >>>>> sleep 10 >>>>> /sbin/zpool import -f jbod >>>>> /etc/rc.d/mountd restart >>>>> /etc/rc.d/nfsd restart >>>>> ;; >>>>> backup) >>>>> echo "Switching to BACKUP and exporting ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to BACKUP' root >>>>> /sbin/zpool export jbod >>>>> /etc/rc.d/mountd restart >>>>> /etc/rc.d/nfsd restart >>>>> ;; >>>>> *) >>>>> exit 0 >>>>> ;; >>>>> esac >>>>>=20 >>>>> This works, most of the time, but sometimes I'm forced to = re-create pool. Those machines suppose to go into prod. >>>>> Loosing pool(and data inside it) stops me from deploy this setup. >>>>=20 >>>> This script looks highly error-prone. Hasty hasty... :-) >>>>=20 >>>> This script assumes that the "zpool" commands (import and export) = always >>>> work/succeed; there is no exit code ($?) checking being used. >>>>=20 >>>> Since this is run from within devd(8): where does stdout/stderr go = to >>>> when running a program/script under devd(8)? Does it effectively = go >>>> to the bit bucket (/dev/null)? If so, you'd never know if the = import or >>>> export actually succeeded or not (the export sounds more likely to = be >>>> the problem point). >>>>=20 >>>> I imagine there would be some situations where the export would = fail >>>> (some files on filesystems under pool "jbod" still in use), yet = CARP is >>>> already blindly assuming everything will be fantastic. Surprise. >>>>=20 >>>> I also do not know if devd.conf(5) "action" commands spawn a = sub-shell >>>> (/bin/sh) or not. If they don't, you won't be able to use things = like" >>>> 'action "/etc/zfs_switch.sh active >> /var/log/failover.log";'. = You >>>> would then need to implement the equivalent of logging within your >>>> zfs_switch.sh script. >>>>=20 >>>> You may want to consider the -f flag to zpool import/export >>>> (particularly export). However there are risks involved -- = userland >>>> applications which have an fd/fh open on a file which is stored on = a >>>> filesystem that has now completely disappeared can sometimes crash >>>> (segfault) or behave very oddly (100% CPU usage, etc.) depending on = how >>>> they're designed. >>>>=20 >>>> Basically what I'm trying to say is that devd(8) being used as a = form of >>>> HA (high availability) and load balancing is not always possible. >>>> Real/true HA (especially with SANs) is often done very differently = (now >>>> you know why it's often proprietary. :-) ) >>=20 >> Add error checking to your script. That's my first and foremost >> recommendation. It's not hard to do, really. :-) >>=20 >> After you do that and still experience the issue (e.g. you see no = actual >> errors/issues during the export/import phases), I recommend removing >> the "cache" devices which are "independent" on each system from the = pool >> entirely. Quoting you (for readers, since I snipped it from my = previous >> reply): >>=20 >>>>> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC >>>>> is both local and external - da1,da2, da13s2, da14s2 >>=20 >> I interpret this to mean the primary and backup nodes (physical = systems) >> have actual disks which are not part of the "external enclosure". If >> that's the case -- those disks are always going to vary in their >> contents and metadata. Those are never going to be 100% identical = all >> the time (is this not obvious?). I'm surprised your stuff has worked = at >> all using that model, honestly. >>=20 >> ZFS is going to bitch/cry if it cannot verify the integrity of = certain >> things, all the way down to the L2ARC. That's my understanding of it = at >> least, meaning there must always be "some" kind of metadata that has = to >> be kept/maintained there. >>=20 >> Alternately you could try doing this: >>=20 >> zpool remove jbod cache daX daY ... >> zpool export jbod >>=20 >> Then on the other system: >>=20 >> zpool import jbod >> zpool add jbod cache daX daY ... >>=20 >> Where daX and daY are the disks which are independent to each system >> (not on the "external enclosure"). >>=20 >> Finally, it would also be useful/worthwhile if you would provide=20 >> "dmesg" from both systems and for you to explain the physical wiring >> along with what device (e.g. daX) correlates with what exact thing on >> each system. (We right now have no knowledge of that, and your terse >> explanations imply we do -- we need to know more) >>=20 >> --=20 >> | Jeremy Chadwick jdc@koitsu.org | >> | UNIX Systems Administrator http://jdc.koitsu.org/ | >> | Making life hard for others since 1977. PGP 4BD6C0CB | >>=20 >=20 From owner-freebsd-fs@FreeBSD.ORG Fri Jun 21 21:10:03 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 26B94C30 for ; Fri, 21 Jun 2013 21:10:03 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id 9D2F9166F for ; Fri, 21 Jun 2013 21:10:02 +0000 (UTC) Received: by mail-la0-f52.google.com with SMTP id fo12so7956711lab.39 for ; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=JVgfVh3NrPxOGfKyIUAP084/ocYp9DlSLpQxcU76UWQ=; b=CcZ+yLA66DQaOIMW3d+edQa83HZWR5p5I7DMq/S+MBthfvrk+GYJFtEDIW+PzFLVPG Y0vTI0Mo4Xg+iRKfTuLmlL/9a5G85OEzOSsDarXrAxtLQIvR9Bm7YwfaVMt0TuxufaGI S9io1P/7ndxl01g1c4hw+AFes1q+sNk6gl4Nw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:x-gm-message-state; bh=JVgfVh3NrPxOGfKyIUAP084/ocYp9DlSLpQxcU76UWQ=; b=bXWOcEJUOnCNS52co5pyOiISjnYDekZ8ympCqZimF8KYGas946ibB5ZXpmWZYNcne8 UdFxIzn6o+5pIzLiiY7CJg7KUTOOtksa9iIFQ4tO/4k/w5PakOh4i1PWxYflvK8FG7F1 uhwQ4H0lkxwxYDg8hLEAXW14J/AxdamL6H3mSrCymn2MEqgr31X9uRoGcK2CEaLz2RFA vm389HTCqaKFTENU0EB9doduw7oUPTEPJiSF0OyA9O1QmkydqccHzpXheOWn3L4lFgKo LRVxoalBTVW52USKCB8B2uqLKxL1yE56WE5PLbV31OZiSG76XJ99izYBkuWjT7TISZaH Py9g== MIME-Version: 1.0 X-Received: by 10.152.28.129 with SMTP id b1mr6650910lah.51.1371849001435; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) Received: by 10.114.79.132 with HTTP; Fri, 21 Jun 2013 14:10:01 -0700 (PDT) In-Reply-To: <51B982A8.10605@bytecamp.net> References: <51B79023.5020109@fsn.hu> <20130612114937.GA13688@icarus.home.lan> <51B982A8.10605@bytecamp.net> Date: Fri, 21 Jun 2013 14:10:01 -0700 Message-ID: Subject: Re: An order of magnitude higher IOPS needed with ZFS than UFS From: Matthew Ahrens To: Robert Schulze X-Gm-Message-State: ALoCoQkcX/3DLmbp5TZ0XmdxZn1qGl7m/7vvLAqQB+VLG/pvm6G5sIFeguuoAFZdfP+PjqI4KB2R Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Jun 2013 21:10:03 -0000 The reason for this behavior is that it's hard to know how much space things will take up on disk until we get to writing them out. For example, because compression changes the space usage, and we don't compress the data until we write it out. (And all metadata, (e.g. indirect blocks) are compressed, and count towards quotas.) I think it would be reasonable to implement something like "quota_enforcement=delayed", which would cause regular quotas, reservations, refquotas, and refreservations to behave like userquota@... quotas, in that they can be exceeded by a small amount (a few seconds worth of change), but they do not impact performance. --matt On Thu, Jun 13, 2013 at 1:28 AM, Robert Schulze wrote: > Hi, > > Am 12.06.2013 13:49, schrieb Jeremy Chadwick: > >> On Wed, Jun 12, 2013 at 06:40:32AM -0500, Mark Felder wrote: >> >>> On Tue, 11 Jun 2013 16:01:23 -0500, Attila Nagy wrote: >>> >>> ZFS write performance can begin to drop pretty badly when you get >>> around 80% full. I've not seen any benchmarks showing an improvement >>> with a very fast and large ZIL or tons of memory, but I'd expect >>> that would help significantly. Just note that you're right at the >>> edge where performance gets impacted. >>> >> >> Mark, do you have any references for this? I'd love to learn/read more >> about this engineering/design aspect (I won't say flaw, I'll just say >> aspect) to ZFS, as it's the first I've heard of it. >> > > this is even true when getting near a quota limit on a zfs, although there > are e.g. 10/16 TB free in the pool. > > Just create a filesystem and set quota=1G, then do sequential invocations > of dd to fill the fs with 100M files. You will see a sharp slowdown when > the last twenty files are beeing created. > > Here are the results from the following short test: > > for i in `jot - 0 99` > do > dd if=/dev/zero of=/pool/quota-test/10M.$i bs=1M count=10 > done > > 0..80: < 0.4 s > 80 0.27 s > 81 0.77 s > 82 0.50 s > 83 0.51 s > 84 0.22 s > 85 0.87 s > 86 0.52 s > 87 1.13 s > 88 0.91 s > 90 0.39 s > 91 1.04 s > 92 0.80 s > 93 1.94 s > 94 1.27 s > 95 1.36 s > 96 1.76 s > 97 2.13 s > 98 3.28 s > 99 4.07 s > > of course, there are some small values beyond 80% utilisation, but I think > the trend is clearly visible. > > In my opinion, hitting a quota limit should not give these results unless > enough free physical disk space is available in the pool. This is a bug or > a design flaw and creating serious problems when exporting quota'ed zfs > over nfs. > > with kind regards, > Robert Schulze > > -- > /7\ bytecamp GmbH > Geschwister-Scholl-Str. 10, 14776 Brandenburg a.d. Havel > HRB15752, Amtsgericht Potsdam, Geschaeftsfuehrer: > Bjoern Barnekow, Frank Rosenbaum, Sirko Zidlewitz > tel +49 3381 79637-0 werktags 10-12,13-17 Uhr, fax +49 3381 79637-20 > mail rs@bytecamp.net, web http://bytecamp.net/ > > ______________________________**_________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org > " > From owner-freebsd-fs@FreeBSD.ORG Sat Jun 22 14:30:31 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 50A85CF1 for ; Sat, 22 Jun 2013 14:30:31 +0000 (UTC) (envelope-from freebsd@growveg.net) Received: from smtp1.servage.net (smtp1.servage.net [IPv6:2a01:3b0:1:fb:1::2001]) by mx1.freebsd.org (Postfix) with ESMTP id 1315B1FA9 for ; Sat, 22 Jun 2013 14:30:30 +0000 (UTC) Received: from potato.growveg.org (growveg-1-pt.tunnel.tserv11.ams1.ipv6.he.net [IPv6:2001:470:1f14:3d2::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp1.servage.net (Postfix) with ESMTPSA id 578EE3288C for ; Sat, 22 Jun 2013 14:32:42 +0000 (UTC) Message-ID: <51C5B4EE.3010601@growveg.net> Date: Sat, 22 Jun 2013 15:30:06 +0100 From: John User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130607 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jun 2013 14:30:31 -0000 Hello list, I have a Dell R710 with PERC H310 using hardware raid5 with 4x3TB disks. I tried installing FreeBSD-10 and it defaulted to a GPT partitioning scheme. FreeBSD installed OK but, on reboot, it couldn't find the kernel. I grabbed freebsd-9.1 and it offered me the choice of partitioning schemes. I had to change the partitioning scheme to "BSD partition" I think it's called, in order to get a bootable install. The problem with this is that this particular schema can only see about 1TB of space yet there's about 10TB there. Currently, writes are cached on the hardware raid. (Need to tune it up because I left all settings at their default in the configuration utility. If it does a lot of writes, it will stall as the buffer fills.) I'm happy to let freebsd have this 1TB to itself if I can somehow get it to seethe other 9TB. The purpose of the server is to run virtual boxes. Hard drive array is seen like this: mfi0: port 0xfc00-0xfcff mem 0xddffc000-0xddffffff,0xddf80000-0xddfbffff irq 42 at device 0.0 on pci2 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 My questions are: 0. which is better? ZFS or hardware raid+UFS2? 1. should I use ZFS on top of the raid5? Guessing not. 2. should I blat the system and change the raid so it's a JBOD, then make it do zfs? 3. what other methods can I use to make the existing install see the rest of the space? 4. how come it could not read the kernel when the partitioning scheme was GPT?Is it because it was raid5 in hardware? thanks! From owner-freebsd-fs@FreeBSD.ORG Sat Jun 22 17:49:10 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3C96219F for ; Sat, 22 Jun 2013 17:49:10 +0000 (UTC) (envelope-from prvs=1885c13778=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id C08801751 for ; Sat, 22 Jun 2013 17:49:09 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004466503.msg for ; Sat, 22 Jun 2013 18:48:31 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 22 Jun 2013 18:48:31 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1885c13778=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> From: "Steven Hartland" To: "John" , References: <51C5B4EE.3010601@growveg.net> Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? Date: Sat, 22 Jun 2013 18:48:43 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jun 2013 17:49:10 -0000 ----- Original Message ----- From: "John" To: Sent: Saturday, June 22, 2013 3:30 PM Subject: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? > Hello list, > > I have a Dell R710 with PERC H310 using hardware raid5 with 4x3TB disks. > I tried installing FreeBSD-10 and it defaulted to a GPT partitioning > scheme. FreeBSD installed OK but, on reboot, it couldn't find the > kernel. I grabbed freebsd-9.1 and it offered me the choice of > partitioning schemes. > > I had to change the partitioning scheme to "BSD partition" I think it's > called, in order to get a bootable install. The problem with this is > that this particular schema can only see about 1TB of space yet there's > about 10TB there. > > Currently, writes are cached on the hardware raid. (Need to tune it up > because I left all settings at their default in the configuration > utility. If it does a lot of writes, it will stall as the buffer fills.) > > I'm happy to let freebsd have this 1TB to itself if I can somehow get it > to seethe other 9TB. The purpose of the server is to run virtual boxes. > > Hard drive array is seen like this: > > mfi0: port 0xfc00-0xfcff mem > 0xddffc000-0xddffffff,0xddf80000-0xddfbffff irq 42 at device 0.0 on pci2 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > > My questions are: > > 0. which is better? ZFS or hardware raid+UFS2? IMO ZFS. It gives you end to end checksuming, which has saved us more than once, self healing (no need to do a full array level scrub. > 1. should I use ZFS on top of the raid5? Guessing not. You can but I wouldn't advise it, much better to let ZFS have the raw disks. > 2. should I blat the system and change the raid so it's a JBOD, then > make it do zfs? IMO yes. > 3. what other methods can I use to make the existing install see > the rest of the space? Sorry not aware of any. > 4. how come it could not read the kernel when the partitioning scheme > was GPT?Is it because it was raid5 in hardware? Shouldn't be, I've never had an issue booting from GPT installs even with much older Dell machines. I see you have an dell + mfi controller, there was an important FW update released a week or so back do ensure you have that installed or you'll suffer from nasty IO stalls. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Sat Jun 22 18:19:34 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A11077B6 for ; Sat, 22 Jun 2013 18:19:34 +0000 (UTC) (envelope-from freebsd@growveg.net) Received: from smtp1.servage.net (smtp1.servage.net [IPv6:2a01:3b0:1:fb:1::2001]) by mx1.freebsd.org (Postfix) with ESMTP id 699C51837 for ; Sat, 22 Jun 2013 18:19:34 +0000 (UTC) Received: from potato.growveg.org (growveg-1-pt.tunnel.tserv11.ams1.ipv6.he.net [IPv6:2001:470:1f14:3d2::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp1.servage.net (Postfix) with ESMTPSA id BED0B320A8 for ; Sat, 22 Jun 2013 18:22:04 +0000 (UTC) Message-ID: <51C5EAB0.7040009@growveg.net> Date: Sat, 22 Jun 2013 19:19:28 +0100 From: John User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130607 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? References: <51C5B4EE.3010601@growveg.net> <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> In-Reply-To: <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jun 2013 18:19:34 -0000 Hello, thank you for your reply. On 22/06/2013 18:48, Steven Hartland wrote: > I see you have an dell + mfi controller, there was an important FW update > released a week or so back do ensure you have that installed or you'll > suffer from nasty IO stalls. Thats... thats what I was seeing!!! I was running a svn update for ports and svn just sat there like a piece of cheese in a state of wdrain according to top. OK so firstly I need to update the controller. After some googling I found this site https://calomel.org/zfs_raid_speed_capacity.html where they "raid0" each disk to take advantage of the controller, but it seems (so far) with my controller that once hardware raid is enabled it can't be disabled. It does not allow the opportunity to make one raid0 disk from one disk and then go to another disk to do the same. Maybe I should do it with the first one then let zfs do the remaining three. -- John From owner-freebsd-fs@FreeBSD.ORG Sat Jun 22 19:16:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D70B770D for ; Sat, 22 Jun 2013 19:16:41 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by mx1.freebsd.org (Postfix) with ESMTP id 9687C1A18 for ; Sat, 22 Jun 2013 19:16:41 +0000 (UTC) Received: from mfilter22-d.gandi.net (mfilter22-d.gandi.net [217.70.178.150]) by relay5-d.mail.gandi.net (Postfix) with ESMTP id CD7B041C060; Sat, 22 Jun 2013 21:16:23 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mfilter22-d.gandi.net Received: from relay5-d.mail.gandi.net ([217.70.183.197]) by mfilter22-d.gandi.net (mfilter22-d.gandi.net [10.0.15.180]) (amavisd-new, port 10024) with ESMTP id JhwKs5IZX2Hl; Sat, 22 Jun 2013 21:16:22 +0200 (CEST) X-Originating-IP: 76.102.14.35 Received: from jdc.koitsu.org (c-76-102-14-35.hsd1.ca.comcast.net [76.102.14.35]) (Authenticated sender: jdc@koitsu.org) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id D6F8441C056; Sat, 22 Jun 2013 21:16:21 +0200 (CEST) Received: by icarus.home.lan (Postfix, from userid 1000) id C31ED73A1C; Sat, 22 Jun 2013 12:16:19 -0700 (PDT) Date: Sat, 22 Jun 2013 12:16:19 -0700 From: Jeremy Chadwick To: John Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? Message-ID: <20130622191619.GA73246@icarus.home.lan> References: <51C5B4EE.3010601@growveg.net> <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> <51C5EAB0.7040009@growveg.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <51C5EAB0.7040009@growveg.net> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jun 2013 19:16:41 -0000 On Sat, Jun 22, 2013 at 07:19:28PM +0100, John wrote: > Hello, thank you for your reply. > > On 22/06/2013 18:48, Steven Hartland wrote: > > I see you have an dell + mfi controller, there was an important FW update > > released a week or so back do ensure you have that installed or you'll > > suffer from nasty IO stalls. > > Thats... thats what I was seeing!!! I was running a svn update for ports > and svn just sat there like a piece of cheese in a state of wdrain > according to top. > > OK so firstly I need to update the controller. After some googling I > found this site https://calomel.org/zfs_raid_speed_capacity.html where > they "raid0" each disk to take advantage of the controller, but it seems > (so far) with my controller that once hardware raid is enabled it can't > be disabled. It does not allow the opportunity to make one raid0 disk > from one disk and then go to another disk to do the same. Maybe I should > do it with the first one then let zfs do the remaining three. 1. I would recommend you avoid use of the controller entirely. Get yourself a different controller that doesn't do RAID and use that. If you can disable the on-board controller in the BIOS, and pick yourself up a Supermicro AOC-USAS2-L8e HBA (around US$140) and use that (or even a siis(4) controller (around US$30)), you'd be better off. http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=E This uses the mps(4) driver. I'm intentionally keeping this terse rather than going into the different HBA types (ex. the 2108 uses mfi(4) (avoid please), and there are other models which do RAID but some which can have RAID disabled by using a different "version" of firmware (have to ask Technical Support for it, etc.). 2. The "make each disk a RAID 0 volume" has caveats which the author of that article does not disclose. One of the biggest is if you plan on using SSDs with the controller -- you won't be able to get TRIM/UNMAP support (via SCSI commands 0x85 (PASS-THROUGH-16) or 0xa1 (PASS-THROUGH-12) doing that, which will hurt you **big time**. It probably varies per controller, but it's proof that using a non-RAID controller is always the way to go with ZFS. Proof of the "RAID 0 volume per disk" crap: http://lists.freebsd.org/pipermail/freebsd-stable/2013-June/073680.html -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-fs@FreeBSD.ORG Sat Jun 22 19:51:46 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5DB88D0F for ; Sat, 22 Jun 2013 19:51:46 +0000 (UTC) (envelope-from prvs=1885c13778=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E166D1B37 for ; Sat, 22 Jun 2013 19:51:45 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50004466869.msg for ; Sat, 22 Jun 2013 20:51:43 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 22 Jun 2013 20:51:43 +0100 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=1885c13778=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <12FB82C142E74924B27F4DBB642D0D1F@multiplay.co.uk> From: "Steven Hartland" To: "Jeremy Chadwick" , "John" References: <51C5B4EE.3010601@growveg.net> <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> <51C5EAB0.7040009@growveg.net> <20130622191619.GA73246@icarus.home.lan> Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? Date: Sat, 22 Jun 2013 20:51:56 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jun 2013 19:51:46 -0000 ----- Original Message ----- From: "Jeremy Chadwick" To: "John" Cc: Sent: Saturday, June 22, 2013 8:16 PM Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? > On Sat, Jun 22, 2013 at 07:19:28PM +0100, John wrote: >> Hello, thank you for your reply. >> >> On 22/06/2013 18:48, Steven Hartland wrote: >> > I see you have an dell + mfi controller, there was an important FW update >> > released a week or so back do ensure you have that installed or you'll >> > suffer from nasty IO stalls. >> >> Thats... thats what I was seeing!!! I was running a svn update for ports >> and svn just sat there like a piece of cheese in a state of wdrain >> according to top. >> >> OK so firstly I need to update the controller. After some googling I >> found this site https://calomel.org/zfs_raid_speed_capacity.html where >> they "raid0" each disk to take advantage of the controller, but it seems >> (so far) with my controller that once hardware raid is enabled it can't >> be disabled. It does not allow the opportunity to make one raid0 disk >> from one disk and then go to another disk to do the same. Maybe I should >> do it with the first one then let zfs do the remaining three. > > 1. I would recommend you avoid use of the controller entirely. Get > yourself a different controller that doesn't do RAID and use that. If > you can disable the on-board controller in the BIOS, and pick yourself > up a Supermicro AOC-USAS2-L8e HBA (around US$140) and use that > (or even a siis(4) controller (around US$30)), you'd be better off. > > http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=E > > This uses the mps(4) driver. > > I'm intentionally keeping this terse rather than going into the > different HBA types (ex. the 2108 uses mfi(4) (avoid please), and there > are other models which do RAID but some which can have RAID disabled by > using a different "version" of firmware (have to ask Technical Support > for it, etc.). > > 2. The "make each disk a RAID 0 volume" has caveats which the author of > that article does not disclose. One of the biggest is if you plan on > using SSDs with the controller -- you won't be able to get TRIM/UNMAP > support (via SCSI commands 0x85 (PASS-THROUGH-16) or 0xa1 > (PASS-THROUGH-12) doing that, which will hurt you **big time**. Some mfi controllers actually have a JBOD option, if it does you'll be able to this controller if not watch out as using the RAID 0 option will almost certainly cause you pain. mps as Jeremy has said is better option. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.