From owner-freebsd-fs@freebsd.org Sun Jul 16 07:12:18 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC91ABF81C2 for ; Sun, 16 Jul 2017 07:12:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C826E6FF28 for ; Sun, 16 Jul 2017 07:12:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6G7CIIY089579 for ; Sun, 16 Jul 2017 07:12:18 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220693] head -r320570 & -r320760 (e.g.): ufs snapshot creation broken & leads to fsck -B related SSD-trim "freeing free block" panics; more Date: Sun, 16 Jul 2017 07:12:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 07:12:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220693 --- Comment #2 from commit-hook@freebsd.org --- A commit references this bug: Author: kib Date: Sun Jul 16 07:11:30 UTC 2017 New revision: 321040 URL: https://svnweb.freebsd.org/changeset/base/321040 Log: A followup to r320453, correct removal of the blocks from UFS snapshots. Tested by: pho PR: 220693 Sponsored by: The FreeBSD Foundation Changes: head/sys/ufs/ffs/ffs_alloc.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jul 16 08:48:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52649BF9EAB for ; Sun, 16 Jul 2017 08:48:05 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: from relay.ibs.dn.ua (relay.ibs.dn.ua [148.251.53.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "relay.ibs.dn.ua", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E169572204 for ; Sun, 16 Jul 2017 08:48:04 +0000 (UTC) (envelope-from zeus@ibs.dn.ua) Received: on behalf of honored client by relay.ibs.dn.ua with ESMTP id v6G8ltO0081192 for on Sun, 16 Jul 2017 11:47:56 +0300 (EEST) Message-ID: <20170716114750.81191@relay.ibs.dn.ua> Date: Sun, 16 Jul 2017 11:47:50 +0300 From: "Zeus Panchenko" To: "FreeBSD Filesystems" cc: Subject: [Q] how to delay resilvering util all devices are available? Organization: I.B.S. LLC Reply-To: "Zeus Panchenko" X-Attribution: zeus Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAFVBMVEWxsbGdnZ3U1NQTExN cXFzx8fG/v7+f8hyWAAACXUlEQVQ4jUWSwXYiIRBFi4yyhtjtWpmRdTL0ZC3TJOukDa6Rc+T/P2F eFepwtFvr8upVFVDua8mLWw6La4VIKTuMdAPOebdU55sQs3n/D1xFFPFGVGh4AHKttr5K0bS6g7N ZCge7qpVLB+f1Z2WAj2OKXwIWt/bXpdXSiu8KXbviWkHxF5td9+lg2e3xlI2SCvatK8YLfHyh9lw 15yrad8Va5eXg4Llr7QmAaC+dL9sDt9iad/DX3OKvLMBf+dm0A0QuMrTvYIevSik1IaSVvgjIHt5 lSCG2ynNRpEcBZ8cgDWk+Ns99qzsYYV3MZoppWzGtYlTO9+meG6m/g92iNO9LfQB2JZsMpoJs7QG ku2KtabRK0bZRwDLyBDvwlxTm6ZlP7qyOqLcfqtLexpDSB4M0H3I/PQy1emvjjzgK+A0LmMKl6Lq zlqzh0VGAw440F6MJd8cY0nI7wiF/fVIBGY7UNCAXy6DmfYGCLLI0wtDbVcDUMqtJLmAhLqODQAe riERAxXJ1/QYGpa0ymqyytpKC19MNXHjvFmEsfcHIrncFR4xdbYWgmfEGLCcZokpGbGj1egMR+6M 1BkNX1pDdhPcOXpAnAeLQUwQLYepgQoZVNGS61yaE8CYA7gYAcWKzwGstACY2HTFvvOwk4FXAG/a mKHni/EcA/GkOk7I0IK7UMIf3+SahU8/FJdiE7KcuWdM3MFocUDEEIX9LfJoo4xV5tnNKc3jJuSs SZWgnnhepgU1zN4Hii18yW4RwDX52CXUtk0Hqz6cHOIUkWaX8fDcB+J7y1y2xDHwjv/8Buu8Ekz6 7tXQAAAAASUVORK5CYII= X-Mailer: MH-E 8.6; nil; GNU Emacs 25.1.1 MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 08:48:05 -0000 =2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 greetings, please advise my storage pool is built of 10 raids2, each of them has 6x2T HDD when system hangs, nothing helps but reset all of the disks, intentionally are not available after reboot and I'm initializing them later if the pool is not exported, ZFS begins resilver the pool from the disks it can see, but each disk is initialized withing several seconds delay to avoid that I do export first, then I initialize disks and after that do import (and in my case no resilvering at all takes place) ... so, it looks rather artificial to me ... here is my question: how to awoid resilvering util all devices are available?=20 can I say "offline", something to the pool to not to try to use the disks available until I decide it is the time to? =2D --=20 Zeus V. Panchenko jid:zeus@im.ibs.dn.ua IT Dpt., I.B.S. LLC GMT+2 (EET) =2D----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQQYIXL6FUmD7SUfqoOveOk+D/ejKgUCWWsoNgAKCRCveOk+D/ej KodgAJwLMATFxwlKw9CI2HX4zskfeagU5QCeOovp95WjYXq+sezdT3lU0wEtcIk=3D =3DxetB =2D----END PGP SIGNATURE----- From owner-freebsd-fs@freebsd.org Sun Jul 16 15:27:48 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5EC6DC09BC4 for ; Sun, 16 Jul 2017 15:27:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C229815B9 for ; Sun, 16 Jul 2017 15:27:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6GFRm0Z003338 for ; Sun, 16 Jul 2017 15:27:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220693] head -r320570 & -r320760 (e.g.): ufs snapshot creation broken & leads to fsck -B related SSD-trim "freeing free block" panics; more Date: Sun, 16 Jul 2017 15:27:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status cc resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 15:27:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220693 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Closed CC| |kib@FreeBSD.org Resolution|--- |FIXED --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sun Jul 16 16:00:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41C40C0A68D for ; Sun, 16 Jul 2017 16:00:04 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EFE782238 for ; Sun, 16 Jul 2017 16:00:02 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 7794C67995; Sun, 16 Jul 2017 17:52:44 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id WWVBpBhQzYJQ; Sun, 16 Jul 2017 17:52:40 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 2E6B467983; Sun, 16 Jul 2017 17:52:40 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id CFEE7508A1; Sun, 16 Jul 2017 17:52:39 +0200 (CEST) Message-ID: <596B8BC7.3030505@incore.de> Date: Sun, 16 Jul 2017 17:52:39 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Kirk McKusick CC: freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition References: <201706242203.v5OM3U2Q096283@chez.mckusick.com> <594FB9E5.3060806@incore.de> In-Reply-To: <594FB9E5.3060806@incore.de> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 16:00:04 -0000 Hello, I like to discuss the following code snippet from function indiracct_ufs2() in source file ffs_snapshot.c: /* * We have to expand bread here since it will deadlock looking * up the block number for any blocks that are not in the cache. */ bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); bp->b_blkno = fsbtodb(fs, blkno); if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { brelse(bp); return (error); } In all my tests (all mount types, good or bad result) the flags B_DONE and B_DELWRI in bp->b_flags were alway zero, so the data buffer given by getblk() at bp->b_data is always overwritten with the data from the explicit I/O performed by readblock(). By the way the flag B_CACHE was always set. I have saved the buffer given by getblk() and compared this buffer with the data read by readblock(). Without gjournal there is never a difference. Using a gjournaled partition things are more sophisticated. Taking a snapshot in my example environment needs 678 calls of getblk(). In the "good case" all I/O's performed by readblock() gave the same data as getblk(), in the "bad case" there are some differences between the data in buffer cache seen by getblk() and the data on disk seen by readblock(). Mostly two or three of the 678 blocks are different. In this case the block read by readblock() is always zero, the block read by getblk() has a lot of BLK_NOCOPY values, sometimes 4096 (a full block). There is one other flag in bp->b_flags given by getblk that is of interest: B_CLUSTEROK. This flag is always in sync with the flag BV_SCANNED in bp->b_vflags. In the above code snippet I can check this flag too before going to disk: if (bp->b_flags & (B_DONE | B_DELWRI | B_CLUSTEROK)) == 0 && ... With this patch the problem has gone in all my tests. The resulting snapblklist written to the end of the snapfile is always the same. I am not familiar with buffer and memory management of FreeBSD, so I can not explain what is going on. But I see in cgaccount() a lot of BLK_NOCOPY values are set and in expunge_ufs2() these values should be seen again. It seems they are in the buffer cache but sometimes not on disk. The patch given above in my test example reads data from buffer cache instead of from disk 141 times. All sync operations in ffs_snapshot() use the parameter MNT_WAIT, the gjournal switcher runs all the time but syncs the same partition with MNT_NOWAIT (g_journal.c: vfs_msync followed by VFS_SYNC). I do not know if this can lead to a race condition between gjournal and snapshot. >> Kirk McKusick wrote: >>> Date: Sat, 24 Jun 2017 18:30:17 +0300 >>> From: Konstantin Belousov >>> To: Andreas Longwitz >>> Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition >>> >>> On Fri, Jun 23, 2017 at 11:14:43AM +0200, Andreas Longwitz wrote: >>>> I try to understand the cause for the "free inode" problem described in >>>> https://lists.freebsd.org/pipermail/freebsd-fs/2013-November/018610.html >>>> >>>> I have setup a test server (FreeBSD 10.3-STABLE #4 r317936) with a >>>> gjournaled partition for /home: >>>> mount -t ufs | grep /home --> >>>> /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, >>>> noatime, gjournal) >>> As the first thing to try, if you perform your tests on the raw >>> partition without gjournal, does the problem stay around ? >> I concur that running your test without gjournal is the next test to try. >> I think that your suspicion that there is a race condition with gjournal >> is likely correct. And if that is true, then the problem will go away >> when gjournal is taken out of the stack. > > The problem only accurs when gjournal is active (tunefs -J) independent > of the three possible mount options I have tested: async, noasync, sync. > For my test I prefer async mount option as stated in gjournal(8). Using > soft updates instead of gjournal is ok in all cases. > >>>> My test creates one snapshot of /home (gets alway inode 4) and removes >>>> this snapshot: >>>> >>>> for i in 1 2 3 4 5 6 7 8; do >>>> echo "starting snaptest $i" >/dev/console >>>> mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home >>>> echo $(ls -ils /home/.snap/fscktest) >/dev/console >>>> rm -f /home/.snap/fscktest >>>> done >>>> >>>> I never have more than this one snapshot at work and during the test I >>>> never have any other >>>> user processes working on /home. A typical output looks like this: >>>> >>>> Jun 21 15:59:52 root: starting snaptest 1 >>>> Jun 21 15:59:52 root: 4 26592 -r-------- 1 root operator 90762970240 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:53 root: starting snaptest 2 >>>> Jun 21 15:59:53 root: 4 26592 -r-------- 1 root operator 90762970152 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:54 kernel: freeing inode /home/4 with 704 blocks >>>> Jun 21 15:59:54 root: starting snaptest 3 >>>> Jun 21 15:59:54 kernel: free inode /home/4 had 704 blocks >>>> Jun 21 15:59:54 root: 4 26592 -r-------- 1 root operator 90762969976 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:56 kernel: freeing inode /home/4 with 2112 blocks >>>> Jun 21 15:59:56 root: starting snaptest 4 >>>> Jun 21 15:59:56 kernel: free inode /home/4 had 2112 blocks >>>> Jun 21 15:59:56 root: 4 26592 -r-------- 1 root operator 90762970240 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:57 root: starting snaptest 5 >>>> Jun 21 15:59:57 root: 4 26592 -r-------- 1 root operator 90762970240 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:58 root: starting snaptest 6 >>>> Jun 21 15:59:58 root: 4 26592 -r-------- 1 root operator 90762970216 >>>> 21 Jun 15:59 /home/.snap/fscktest >>>> Jun 21 15:59:59 kernel: freeing inode /home/4 with 192 blocks >>>> Jun 21 15:59:59 root: starting snaptest 7 >>>> Jun 21 15:59:59 kernel: free inode /home/4 had 192 blocks >>>> Jun 21 15:59:59 root: 4 26592 -r-------- 1 root operator 90762970240 >>>> 21 Jun 16:00 /home/.snap/fscktest >>>> Jun 21 16:00:00 root: starting snaptest 8 >>>> Jun 21 16:00:00 root: 4 26592 -r-------- 1 root operator 90762970240 >>>> 21 Jun 16:00 /home/.snap/fscktest >>>> >>>> The "free inode /home/4 had NNN blocks" message during run of the mount >>>> command is output of ffs_valloc(), because ffs_load_inode() has load the >>>> disk inode 4 with a non zero i_blocks field. The corresponding "freeing >>>> inode /home/4 with NNN blocks" message during the previous rm command >>>> is output of my following diagnostic patch in function ffs_truncate(): >>>> >>>> --- ffs_inode.c.1st 2016-06-08 17:25:21.000000000 +0200 >>>> +++ ffs_inode.c 2017-06-19 10:02:07.145360000 +0200 >>>> @@ -551,6 +551,9 @@ >>>> DIP_SET(ip, i_blocks, DIP(ip, i_blocks) - blocksreleased); >>>> else /* sanity */ >>>> DIP_SET(ip, i_blocks, 0); >>>> + if (bootverbose == 2 && DIP(ip, i_blocks) > 0) >>>> + printf("freeing inode %s/%lu with %ld blocks\n", >>>> + fs->fs_fsmnt, (u_long)ip->i_number, >>>> (long)DIP(ip, i_blocks)); >>>> ip->i_flag |= IN_CHANGE; >>>> #ifdef QUOTA >>>> (void) chkdq(ip, -blocksreleased, NOCRED, 0); >>>> >>>> The rm command can only free all the blocks of the snapshotfile (means >>>> i_blocks for inode 4 ends with zero) , if this file has the "correct" size: >>>> >>>> ls -ils /home/.snap/fscktest --> >>>> 4 53184 -r-------- 1 root operator 90762970240 Jun 17 06:15 >>>> /home/.snap/fscktest >>>> >>>> The size of the /home partition is given by >>>> diskinfo /dev/mirror/gmsvt7p10.journal --> >>>> /dev/mirror/gmsvt7p10.journal 512 90762954240 177271395 >>>> >>>> So we have 2769865 full 32kB blocks with size 90631864320. During >>>> creating a snapshot a "last block" (32kB) is written at this offset >>>> ending at 90762969088. Finally the snapblklist is written with >>>> VOP_WRITE: "Write out the list of allocated blocks to the end of the >>>> snapshot". In all my correct tests the table snapblklist is 1152 bytes >>>> in size giving the correct size of the snapshot file : 90762970240. In >>>> this case the table snapblklist has 144 entries of length 8: one lenght >>>> entry and 143 logical block numbers recorded in mapacct_ufs2(): >>>> >>>> if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) >>>> *ip->i_snapblklist++ = lblkno; >>>> >>>> The console output above shows three error situations with block >>>> counters 704, 2112 and 192. Dividing these values by 8 gives exactly the >>>> reduced size of the snapblocklist at the end of the snapshotfile, so in >>>> these cases the snapshotfile is corrupt. >>> I am not sure what do you mean by 'match' there. Could you explicitely >>> mention what relations between snapshot size and leaked blocks of the >>> free snapshot inode did you noted ? >> I too am confused here. Are you saying for example that 192 / 8 == 24 >> and that the snapblocklist is short by 24 blocks? Because from the table >> below, it appears to be short by only 3 blocks. > > In this example the snapblocklist is short by 24 bytes (= 3 table > entries) and 24/8 = 3 calls to ffs_blkfree are missing. > > Ok, I give one more table to explain a little better: > > The meaning of the columns: > #1 number of "free inode blocks" in kernel message > #2 blocks from #1 divides by 8 (= fs_frag) > #3 size of snapblklist write at the end of snapshotfile > #4 ls -s of snapshotfile: 907629... > #5 enties in table snapblklist, we have #5 = #3 / 8, because each > entry in the table is of type ufs2_daddr_t > #6 counter ct_blkfree, thats the number of calls to ffs_blkfree(). > #7 missing table entries in snapblklist and calls of ffs_blkfree(). > > test #1 #2 #3 #4 #5 #6 #7 > ----------------------------------------------- > 1 ok 0 0 1152 ..70240 144 821 0 > 2 bad 704 88 1064 ..70152 133 810 11 > 3 bad 2112 264 888 ..69976 111 788 33 > 6 bad 192 24 1128 ..70216 141 818 3 > > The difference of #5 and #6 is the variable acctit which is set only for > blkno != -1. In all my test (ok and bad) the function mapacct_ufs2() is > called 680 times, the first calls look like this (diffblk = lastblk - > oldblk): > > mapacct_ufs2:entry /home, inum=4, diffblkp=0xc, lblkno=0 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x3, lblkno=-1 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x1000, lblkno=12 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x2a4, lblkno=-1 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x1000, lblkno=4108 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x1000, lblkno=8204 > mapacct_ufs2:entry /home, inum=4, diffblkp=0x1000, lblkno=12300 > .... > > There are exact two calls with lblkno=-1 with together 0x2a7 -2 = 677 > runs of the for loop where #5 is skipped. > >>>> I use a test kernel with some extra counters ct_* in mapacct_ufs2(): >>>> >>>> ++ct_blkno_all; >>>> if (blkno == 0) >>>> ++ct_blkno_0; >>>> if (blkno == BLK_NOCOPY) >>>> ++ct_blkno_nocopy; >>>> if (blkno == BLK_SNAP) >>>> ++ct_blkno_snap; >>>> if (blkno == 0 || blkno == BLK_NOCOPY) >>>> continue; >>>> if (acctit && expungetype == BLK_SNAP && blkno != BLK_SNAP) { >>>> *ip->i_snapblklist++ = lblkno; >>>> ++ct_snapblklist; >>>> } >>>> if (blkno == BLK_SNAP) >>>> blkno = blkstofrags(fs, lblkno); >>>> ++ct_blkfree; >>>> ffs_blkfree(ip->i_ump, fs, vp, blkno, fs->fs_bsize, inum, >>>> vp->v_type, NULL); >>>> >>>> and for the 8 test runs shown above I can see these results using DTrace >>>> at probe expunge_ufs2:return (blkno_snap is always 0): >>>> >>>> test blkno_all blkno_0 blkno_nocopy snapblklist blkfree cg_nocopy >>>> ------------------------------------------------------------------- >>>> 1 ok 2770545 353320 2416404 143 821 2416404 >>>> 2 bad 2770545 587860 2181875 132 810 2416404 >>>> 3 bad 2770545 956582 1813175 110 788 2416393 >>>> 4 ok 2770545 353364 2416360 143 821 2416360 >>>> 5 ok 2770545 353364 2416360 143 821 2416360 >>>> 6 bad 2770545 418376 2351351 140 818 2416360 >>>> 7 ok 2770545 353367 2416357 143 821 2416357 >>>> 8 ok 2770545 353367 2416357 143 821 2416357 >>>> >>>> For correct tests the sum of blkno_0 and blkno_nocopy is always the same >>>> (2769724), for bad tests especially the counter for blkno_nocopy is >>>> significant lower. In the test table I give one more column cg_nocopy >>>> for a counter I have added in cgaccount() to see how many entries are >>>> set to BLK_NOCOPY during copy of cylinder group maps: >>>> >>>> if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { >>>> ++ct_cg_nocopy; >>>> DIP_SET(ip, i_db[loc], BLK_NOCOPY); >>>> } >>>> ... >>>> if (ffs_isblock(fs, cg_blksfree(cgp), loc)) { >>>> ++ct_cg_nocopy; >>>> ((ufs2_daddr_t *)(ibp->b_data))[indiroff] = >>>> BLK_NOCOPY; >>>> } >>>> >>>> For correct tests all the BLK_NOCOPY's which are set in cgaccount() can >>>> later be seen in mapacct_ufs2(), for bad tests many of the BLK_NOCOPY's >>>> have changed to 0. >>>> >>>> I looks like the rm command removing the previous snapshot in some way >>>> runs "in the background" simultan to expunge_ufs2() and changes some of >>>> the BLK_NOCOPY's to zero. So this may be a buffer management problem >>>> which only exists on gjourneled partitions, maybe getblk/readblock used >>>> in indiracct_ufs2() is not compatibel with gjournel in the special case >>>> of creating or removing a spapshot. A hint in this direction is the >>>> fact, that the first test after cleaning the partition with >>>> umount /home; fsck -y /home; mount /home >>>> always succeeds. The following modified test procedure never fails: >>>> >>>> for i in 1 2 3 4 5 6 7 8; do >>>> echo "starting snaptest $i" >/dev/console >>>> mount -u -o snapshot -o noatime -o async /home/.snap/fscktest /home >>>> echo $(ls -ils /home/.snap/fscktest) >/dev/console >>>> rm -f /home/.snap/fscktest /home >>>> umount /home >>>> mount /home >>>> done >>> After the allocations of required blocks for the snapshot inode >>> are finished, the filesystem is suspended. You can see the >>> call to vfs_write_suspend() in the ffs_snapshot() where the >>> suspension is enforced. As part of the suspension, all soft-update >>> workitems are flushed, this is done by vfs_write_suspend() calling >>> VFS_SYNC(MNT_SUSPEND). >>> >>> UFS classifies writers into primary and secondary. Primary are mostly >>> the writes initiated by the top-level syscall entries, like direct >>> calls to write(2) or metadata-changing ops mkdir(), create() and so on. >>> Secondary are writes performed when system initiates metadata updates >>> during inactivation, quota updates, softdep background processing and >>> similar. Primary modifications are blocked outright on suspension, while >>> secondary are waited to finish in the mentioned VFS_SYNC(MNT_SUSPEND) >>> call. >>> >>> If you can provide a proof that some SU-related activity happens after the >>> suspension is established, this would be interesting to see. >>> Might be it is something different and much simpler, but I do not see >>> an obvious mistake in the code, after reading your observations. >> The mount information at the top shows: >> >> /dev/mirror/gmsvt7p10.journal on /home (ufs, asynchronous, local, noatime, gjournal) >> >> Thus no soft updates are being used. We are running on just a basic UFS >> filesystem. So, soft update processing has no relevance here. >> >>>> Another proof that the snapshot file is corrupt when the snapblklist is >>>> shortend is the fact that the rm command sporadically panics in a kernel >>>> routine that is known to be correct: >>>> >>>> nread portion of the kernel message buffer: >>>> dev = mirror/gmsvt7p10.journal, block = 19727560, fs = /home >>>> panic: ffs_blkfree_cg: freeing free block >>>> cpuid = 1 >>>> KDB: stack backtrace: >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>>> 0xfffffe0857e3b1c0 >>>> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0857e3b270 >>>> vpanic() at vpanic+0x126/frame 0xfffffe0857e3b2b0 >>>> panic() at panic+0x43/frame 0xfffffe0857e3b310 >>>> ffs_blkfree_cg() at ffs_blkfree_cg+0x5c6/frame 0xfffffe0857e3b3d0 >>>> ffs_blkfree() at ffs_blkfree+0x99/frame 0xfffffe0857e3b430 >>>> ffs_indirtrunc() at ffs_indirtrunc+0x474/frame 0xfffffe0857e3b510 >>>> ffs_indirtrunc() at ffs_indirtrunc+0x423/frame 0xfffffe0857e3b5f0 >>>> ffs_truncate() at ffs_truncate+0x10b4/frame 0xfffffe0857e3b7d0 >>>> ufs_inactive() at ufs_inactive+0x16b/frame 0xfffffe0857e3b810 >>>> VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xf7/frame 0xfffffe0857e3b840 >>>> vinactive() at vinactive+0xc6/frame 0xfffffe0857e3b890 >>>> vputx() at vputx+0x27a/frame 0xfffffe0857e3b8f0 >>>> kern_unlinkat() at kern_unlinkat+0x243/frame 0xfffffe0857e3bae0 >>>> amd64_syscall() at amd64_syscall+0x2c6/frame 0xfffffe0857e3bbf0 >>>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0857e3bbf0 >>>> --- syscall (10, FreeBSD ELF64, sys_unlink), rip = 0x80095425a, rsp = >>>> 0x7fffffffe988, rbp = 0x7fffffffea20 --- >>>> >>>> Any hints solving this problem are welcome. >> Per the suggestion at the top, I recommend trying your test without >> gjournal present to see if the bug goes away. If that is true, then >> we can focus our attention on a possible race during rm in the gjournal >> code. >> >> Kirk McKusick > I have done as explained above. Some more hints may help for debugging > this. > > 1. Output of dumpfs of the partition is > > magic 19540119 (UFS2) time Sun Jun 25 03:46:09 2017 > superblock location 65536 id [ 561bcaa3 624b156b ] > ncg 139 size 22158924 blocks 21459451 > bsize 32768 shift 15 mask 0xffff8000 > fsize 4096 shift 12 mask 0xfffff000 > frag 8 shift 3 fsbtodb 3 > minfree 8% optim time symlinklen 120 > maxbsize 32768 maxbpg 4096 maxcontig 4 contigsumsize 4 > nbfree 2413023 ndir 229 nifree 11153267 nffree 274 > bpg 20035 fpg 160280 ipg 80256 unrefs 0 > nindir 4096 inopb 128 maxfilesize 2252349704110079 > sbsize 4096 cgsize 32768 csaddr 5056 cssize 4096 > sblkno 24 cblkno 32 iblkno 40 dblkno 5056 > cgrotor 123 fmod 0 ronly 0 clean 0 > metaspace 6408 avgfpdir 64 avgfilesize 16384 > flags gjournal > fsmnt /home > volname swuid 0 providersize 22158924 > > 2. I am quite sure that the process g_journal_switcher does not break > things during run of "mount -o snapshot". I have severel DTrace output > of the problem, where g_journal_switcher sleeps all the time. > > 3. The probe expunge_ufs2:return is the first probe from several DTrace > probing that shows a difference between good and bad. > > 4. The problem occurs for many years on several production server and > partitions where only one snapshot is taken every day, but not all and > not very often. On newer server the "free inode" messages increases. On > my test server I have a "freezed" partition /home that shows the problem > with less than ten runs all the time. If I run DTrace with a lot of > probes then the problem sometimes becomes a "Heisenbug". > Dr. Andreas Longwitz From owner-freebsd-fs@freebsd.org Sun Jul 16 16:31:10 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B08DC0846F for ; Sun, 16 Jul 2017 16:31:10 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 37FE58404A for ; Sun, 16 Jul 2017 16:31:09 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v6GGXtjd035000; Sun, 16 Jul 2017 09:33:55 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201707161633.v6GGXtjd035000@chez.mckusick.com> From: Kirk McKusick To: Andreas Longwitz cc: Konstantin Belousov , freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition In-reply-to: <596B8BC7.3030505@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <34998.1500222835.1@chez.mckusick.com> Date: Sun, 16 Jul 2017 09:33:55 -0700 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 16:31:10 -0000 I am copying Kostik on this email even though he is on freebsd-fs as I would like him to review my answer below. > Date: Sun, 16 Jul 2017 17:52:39 +0200 > From: Andreas Longwitz > To: Kirk McKusick > CC: freebsd-fs@freebsd.org > Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition > > Hello, > > I like to discuss the following code snippet from function > indiracct_ufs2() in source file ffs_snapshot.c: > > /* > * We have to expand bread here since it will deadlock looking > * up the block number for any blocks that are not in the cache. > */ > bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); > bp->b_blkno = fsbtodb(fs, blkno); > if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && > (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { > brelse(bp); > return (error); > } > > In all my tests (all mount types, good or bad result) the flags B_DONE > and B_DELWRI in bp->b_flags were always zero, so the data buffer given by > getblk() at bp->b_data is always overwritten with the data from the > explicit I/O performed by readblock(). By the way the flag B_CACHE was > always set. Key observation here that B_CACHE is set! That means that the data in the block is valid and does not need to be read again. So the fix is this: Index: sys/ufs/ffs/ffs_snapshot.c =================================================================== --- sys/ufs/ffs/ffs_snapshot.c (revision 321045) +++ sys/ufs/ffs/ffs_snapshot.c (working copy) @@ -1394,7 +1394,7 @@ */ bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); bp->b_blkno = fsbtodb(fs, blkno); - if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && + if ((bp->b_flags & (B_DONE | B_DELWRI | B_CACHE)) == 0 && (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { brelse(bp); return (error); Absent gjournal the extra read was unnecessary but harmless. The buffer exists with B_DELWRI so we will not read it, or it is locked while being written to disk, so we will wait for the write to complete, after which it is safe to read it. With gjournal there is a window of vulnerability between when the buffer is marked B_DELWRI and when it is actually written to disk. This is because gjournal will say the write has completed when it has not yet actually done so. Thus when we read it, we get incorrect results. But until gjournal actually completes the write, it will keep the buffer around (marked B_CACHE), so as long as we check for B_CACHE to avoid the read (as we should anyway to avoid an unnecessary read) we should always get consistent results. > I have saved the buffer given by getblk() and compared this buffer > with the data read by readblock(). Without gjournal there is never > a difference. Using a gjournaled partition things are more > sophisticated. Taking a snapshot in my example environment needs > 678 calls of getblk(). In the "good case" all I/O's performed by > readblock() gave the same data as getblk(), in the "bad case" there > are some differences between the data in buffer cache seen by > getblk() and the data on disk seen by readblock(). Mostly two or > three of the 678 blocks are different. In this case the block read > by readblock() is always zero, the block read by getblk() has a lot > of BLK_NOCOPY values, sometimes 4096 (a full block). > > There is one other flag in bp->b_flags given by getblk that is of > interest: B_CLUSTEROK. This flag is always in sync with the flag > BV_SCANNED in bp->b_vflags. In the above code snippet I can check this > flag too before going to disk: > > if (bp->b_flags & (B_DONE | B_DELWRI | B_CLUSTEROK)) == 0 && ... > > With this patch the problem has gone in all my tests. The resulting > snapblklist written to the end of the snapfile is always the same. It is merely coincidence that B_CLUSTEROK is set. But the fact that skipping the read fixes your problem leads me to believe that checking for B_CACHE will fix your problem. > I am not familiar with buffer and memory management of FreeBSD, so I can > not explain what is going on. But I see in cgaccount() a lot of > BLK_NOCOPY values are set and in expunge_ufs2() these values should be > seen again. It seems they are in the buffer cache but sometimes not on > disk. The patch given above in my test example reads data from buffer > cache instead of from disk 141 times. The number of buffer flags have grown over time and their meaning has subtlely changed. Even I who have been around since the beginning cannot claim to fully comprehend exactly what they mean. So, the fact that you find them incomprehensible is unfortunately all too understandable. > All sync operations in ffs_snapshot() use the parameter MNT_WAIT, the > gjournal switcher runs all the time but syncs the same partition with > MNT_NOWAIT (g_journal.c: vfs_msync followed by VFS_SYNC). I do not know > if this can lead to a race condition between gjournal and snapshot. I went down this line of thinking with your first email and did not come up with anything that explained the problem. So I am glad that you have found something that I do believe explains what is going on. Kirk McKusick From owner-freebsd-fs@freebsd.org Sun Jul 16 21:00:36 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97F57C79CF8 for ; Sun, 16 Jul 2017 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8CE2C66A49 for ; Sun, 16 Jul 2017 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6GL01m1002409 for ; Sun, 16 Jul 2017 21:00:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201707162100.v6GL01m1002409@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 16 Jul 2017 21:00:36 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Jul 2017 21:00:36 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Jul 17 06:07:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D6FBCFCDDB for ; Mon, 17 Jul 2017 06:07:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E213473981 for ; Mon, 17 Jul 2017 06:07:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v6H67LRk034263 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 17 Jul 2017 09:07:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v6H67LRk034263 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v6H67KQM034262; Mon, 17 Jul 2017 09:07:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 17 Jul 2017 09:07:20 +0300 From: Konstantin Belousov To: Kirk McKusick Cc: Andreas Longwitz , freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition Message-ID: <20170717060720.GH1935@kib.kiev.ua> References: <596B8BC7.3030505@incore.de> <201707161633.v6GGXtjd035000@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201707161633.v6GGXtjd035000@chez.mckusick.com> User-Agent: Mutt/1.8.3 (2017-05-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:07:28 -0000 On Sun, Jul 16, 2017 at 09:33:55AM -0700, Kirk McKusick wrote: > I am copying Kostik on this email even though he is on freebsd-fs > as I would like him to review my answer below. > > > Date: Sun, 16 Jul 2017 17:52:39 +0200 > > From: Andreas Longwitz > > To: Kirk McKusick > > CC: freebsd-fs@freebsd.org > > Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition > > > > Hello, > > > > I like to discuss the following code snippet from function > > indiracct_ufs2() in source file ffs_snapshot.c: > > > > /* > > * We have to expand bread here since it will deadlock looking > > * up the block number for any blocks that are not in the cache. > > */ > > bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); > > bp->b_blkno = fsbtodb(fs, blkno); > > if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && > > (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { > > brelse(bp); > > return (error); > > } > > > > In all my tests (all mount types, good or bad result) the flags B_DONE > > and B_DELWRI in bp->b_flags were always zero, so the data buffer given by > > getblk() at bp->b_data is always overwritten with the data from the > > explicit I/O performed by readblock(). By the way the flag B_CACHE was > > always set. > > Key observation here that B_CACHE is set! That means that the data > in the block is valid and does not need to be read again. So the fix > is this: > > Index: sys/ufs/ffs/ffs_snapshot.c > =================================================================== > --- sys/ufs/ffs/ffs_snapshot.c (revision 321045) > +++ sys/ufs/ffs/ffs_snapshot.c (working copy) > @@ -1394,7 +1394,7 @@ > */ > bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); > bp->b_blkno = fsbtodb(fs, blkno); > - if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && > + if ((bp->b_flags & (B_DONE | B_DELWRI | B_CACHE)) == 0 && > (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { > brelse(bp); > return (error); > > Absent gjournal the extra read was unnecessary but harmless. The > buffer exists with B_DELWRI so we will not read it, or it is locked > while being written to disk, so we will wait for the write to complete, > after which it is safe to read it. With gjournal there is a window of > vulnerability between when the buffer is marked B_DELWRI and when it > is actually written to disk. This is because gjournal will say the > write has completed when it has not yet actually done so. Thus when > we read it, we get incorrect results. But until gjournal actually > completes the write, it will keep the buffer around (marked B_CACHE), > so as long as we check for B_CACHE to avoid the read (as we should > anyway to avoid an unnecessary read) we should always get consistent > results. I tend to disagree with the fix. What if the cached block being discarded before we enter this code fragment ? I believe the bug is indeed clear after your analysis, but it is in gjournal. Problem is that the disk (gjournal substrate layer, but for UFS it is all disk) returns data different from what was written to the volume. After the write bio signalled completion, unless other write was done after it, reads _must_ return the content of the write. The right fix is for gjournal is to perform 'store forwarding', i.e. to return data from the journal until journal is flushd to disk. > > > I have saved the buffer given by getblk() and compared this buffer > > with the data read by readblock(). Without gjournal there is never > > a difference. Using a gjournaled partition things are more > > sophisticated. Taking a snapshot in my example environment needs > > 678 calls of getblk(). In the "good case" all I/O's performed by > > readblock() gave the same data as getblk(), in the "bad case" there > > are some differences between the data in buffer cache seen by > > getblk() and the data on disk seen by readblock(). Mostly two or > > three of the 678 blocks are different. In this case the block read > > by readblock() is always zero, the block read by getblk() has a lot > > of BLK_NOCOPY values, sometimes 4096 (a full block). > > > > There is one other flag in bp->b_flags given by getblk that is of > > interest: B_CLUSTEROK. This flag is always in sync with the flag > > BV_SCANNED in bp->b_vflags. In the above code snippet I can check this > > flag too before going to disk: > > > > if (bp->b_flags & (B_DONE | B_DELWRI | B_CLUSTEROK)) == 0 && ... > > > > With this patch the problem has gone in all my tests. The resulting > > snapblklist written to the end of the snapfile is always the same. > > It is merely coincidence that B_CLUSTEROK is set. But the fact that > skipping the read fixes your problem leads me to believe that checking > for B_CACHE will fix your problem. > > > I am not familiar with buffer and memory management of FreeBSD, so I can > > not explain what is going on. But I see in cgaccount() a lot of > > BLK_NOCOPY values are set and in expunge_ufs2() these values should be > > seen again. It seems they are in the buffer cache but sometimes not on > > disk. The patch given above in my test example reads data from buffer > > cache instead of from disk 141 times. > > The number of buffer flags have grown over time and their meaning has > subtlely changed. Even I who have been around since the beginning cannot > claim to fully comprehend exactly what they mean. So, the fact that you > find them incomprehensible is unfortunately all too understandable. > > > All sync operations in ffs_snapshot() use the parameter MNT_WAIT, the > > gjournal switcher runs all the time but syncs the same partition with > > MNT_NOWAIT (g_journal.c: vfs_msync followed by VFS_SYNC). I do not know > > if this can lead to a race condition between gjournal and snapshot. > > I went down this line of thinking with your first email and did not come > up with anything that explained the problem. So I am glad that you have > found something that I do believe explains what is going on. > > Kirk McKusick From owner-freebsd-fs@freebsd.org Mon Jul 17 06:17:43 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14407CFD38D for ; Mon, 17 Jul 2017 06:17:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0225F74017 for ; Mon, 17 Jul 2017 06:17:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6HgAZ064705 for ; Mon, 17 Jul 2017 06:17:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 06:17:42 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:17:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 06:20:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33D62CFD57E for ; Mon, 17 Jul 2017 06:20:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0607B741D4 for ; Mon, 17 Jul 2017 06:20:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6KjF1041644 for ; Mon, 17 Jul 2017 06:20:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 06:20:45 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:20:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #3 from Andriy Gapon --- Created attachment 184417 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D184417&action= =3Dedit proposed patch Could everyone affected and anyone interested please test this patch? Thank you! --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 06:21:48 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A64D0CFD6D5 for ; Mon, 17 Jul 2017 06:21:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 843F574305 for ; Mon, 17 Jul 2017 06:21:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6Lmpv047239 for ; Mon, 17 Jul 2017 06:21:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 06:21:48 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:21:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |In Progress --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 06:31:11 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95F08CFD9B5 for ; Mon, 17 Jul 2017 06:31:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8464D74A95 for ; Mon, 17 Jul 2017 06:31:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6VBuX066863 for ; Mon, 17 Jul 2017 06:31:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219972] Unable to zpool export following some zfs recv Date: Mon, 17 Jul 2017 06:31:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:31:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219972 --- Comment #4 from Andriy Gapon --- You could try the following DTrace one-liner to try to track down where the error originates. # dtrace -n 'sdt:::set-error /arg0 =3D=3D EBUSY/ { printf("func %s", probef= unc); stack(); }' --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 06:34:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5858CFDB66 for ; Mon, 17 Jul 2017 06:34:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B42A474D2A for ; Mon, 17 Jul 2017 06:34:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6YUj8079771 for ; Mon, 17 Jul 2017 06:34:30 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 06:34:31 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: cy@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:34:30 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #4 from Cy Schubert --- My patch is similar to your patch. It resolves the issue. I'll test yours as well. --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 06:37:08 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E04C2CFDCA6 for ; Mon, 17 Jul 2017 06:37:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CB60B75025 for ; Mon, 17 Jul 2017 06:37:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6H6b84B084592 for ; Mon, 17 Jul 2017 06:37:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 06:37:08 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 06:37:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #5 from Andriy Gapon --- (In reply to Cy Schubert from comment #4) Thank you! Could you please test the kernel with INVARIANTS if possible? --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Mon Jul 17 07:45:11 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0FFDCFEEC6 for ; Mon, 17 Jul 2017 07:45:11 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mail.sub.ru (mail.sub.ru [88.212.205.2]) by mx1.freebsd.org (Postfix) with SMTP id 1B57A77080 for ; Mon, 17 Jul 2017 07:45:10 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: (qmail 653 invoked from network); 17 Jul 2017 10:43:53 +0300 Received: from 128-68-38-94.broadband.corbina.ru (128-68-38-94.broadband.corbina.ru [128.68.38.94]) by mail.sub.ru ([88.212.205.2]) with ESMTP via TCP; 31 Dec 1969 23:59:59 -0000 Subject: Re: How can I recover data from a faulted pool? To: freebsd-fs@freebsd.org References: <7e3d874e-e051-1169-2111-b8f2549f89ee@webmail.sub.ru> From: =?UTF-8?B?0JDQu9C10LrRgdCw0L3QtNGAINCf0L7QstC+0LvQvtGG0LrQuNC5?= Message-ID: <6e91e143-3a7e-446f-bb2c-9899cd725182@webmail.sub.ru> Date: Mon, 17 Jul 2017 10:43:48 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 07:45:11 -0000 zpool import -N -o readonly=on -f -R /mnt/somezpoool -Fn helped. Unfortunately, no files in mysql/innodb, and zfs list shows 56 mbs used. At least, some data has been recovered; I'll wait for zdb to finish on data copy. 15.07.2017 16:52, Volodymyr Kostyrko пишет: > Александр Поволоцкий wrote: >> Hello >> >> FreeBSD 10.3. ZFS. >> >> # zpool status >> pool: database >> state: UNAVAIL >> status: One or more devices are faulted in response to persistent >> errors. There are insufficient replicas for the pool to >> continue functioning. >> action: Destroy and re-create the pool from a backup source. Manually >> marking the device >> repaired using 'zpool clear' may allow some data to be >> recovered. >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> database UNAVAIL 0 0 0 >> mirror-0 UNAVAIL 0 0 0 >> gpt/database0 FAULTED 0 0 0 too many errors >> gpt/database1 FAULTED 0 0 0 too many errors >> >> disks can be read, can I somehow recover data from the pool? > > For sure... First please describe what happened to both disks and what > hardware you were using (chipset/controller). Please don't try to > `zpool clear` yet. > > 1. ZFS *HEAVILY* relies on hardware to be stable, any hardware > glitches may result in severe pool corruption. If you have any doubts > in hardware first move the disks to the trusted box. > > 2. I assume both mirror parts were on different disks? ZFS doesn't > like pools that have more then one chunk on any drive. > > 3. Try mounting pool R/O with transaction recovery. While this is > mirror please do this for both disks and give it another try for > single disks as results may differ: > > `zpool import -N -O readonly=on -f -R /mnt/somezpoool …` > > You can use mfsBSD for that. > > If that doesn't help try: > > `zpool import -N -O readonly=on -f -R /mnt/somezpoool -Fn …` > > Both commands will try to import the pool without mounting file > systems in r/o mode. If that would work, try mounting only required > filesystem and copy some data from it. > > Please post your results. > From owner-freebsd-fs@freebsd.org Mon Jul 17 08:15:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FC2BCFF645 for ; Mon, 17 Jul 2017 08:15:04 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B1A1D77AD2 for ; Mon, 17 Jul 2017 08:15:02 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 19A3B68583; Mon, 17 Jul 2017 10:15:00 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id ukLUVGdFnew1; Mon, 17 Jul 2017 10:14:58 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id DF18D68586; Mon, 17 Jul 2017 10:14:57 +0200 (CEST) Received: from bsdlo.incore (bsdlo.incore [192.168.0.84]) by mail.local.incore (Postfix) with ESMTP id B544C508AA; Mon, 17 Jul 2017 10:14:57 +0200 (CEST) Message-ID: <596C7201.8090700@incore.de> Date: Mon, 17 Jul 2017 10:14:57 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: Konstantin Belousov CC: Kirk McKusick , freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition References: <596B8BC7.3030505@incore.de> <201707161633.v6GGXtjd035000@chez.mckusick.com> <20170717060720.GH1935@kib.kiev.ua> In-Reply-To: <20170717060720.GH1935@kib.kiev.ua> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 08:15:04 -0000 > On Sun, Jul 16, 2017 at 09:33:55AM -0700, Kirk McKusick wrote: >> I am copying Kostik on this email even though he is on freebsd-fs >> as I would like him to review my answer below. >> >>> Date: Sun, 16 Jul 2017 17:52:39 +0200 >>> From: Andreas Longwitz >>> To: Kirk McKusick >>> CC: freebsd-fs@freebsd.org >>> Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition >>> >>> Hello, >>> >>> I like to discuss the following code snippet from function >>> indiracct_ufs2() in source file ffs_snapshot.c: >>> >>> /* >>> * We have to expand bread here since it will deadlock looking >>> * up the block number for any blocks that are not in the cache. >>> */ >>> bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); >>> bp->b_blkno = fsbtodb(fs, blkno); >>> if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && >>> (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { >>> brelse(bp); >>> return (error); >>> } >>> >>> In all my tests (all mount types, good or bad result) the flags B_DONE >>> and B_DELWRI in bp->b_flags were always zero, so the data buffer given by >>> getblk() at bp->b_data is always overwritten with the data from the >>> explicit I/O performed by readblock(). By the way the flag B_CACHE was >>> always set. >> Key observation here that B_CACHE is set! That means that the data >> in the block is valid and does not need to be read again. So the fix >> is this: >> >> Index: sys/ufs/ffs/ffs_snapshot.c >> =================================================================== >> --- sys/ufs/ffs/ffs_snapshot.c (revision 321045) >> +++ sys/ufs/ffs/ffs_snapshot.c (working copy) >> @@ -1394,7 +1394,7 @@ >> */ >> bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); >> bp->b_blkno = fsbtodb(fs, blkno); >> - if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && >> + if ((bp->b_flags & (B_DONE | B_DELWRI | B_CACHE)) == 0 && >> (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { >> brelse(bp); >> return (error); >> This patch was my first try, because the comment in the code snippet points to B_CACHE. Unfortunaly it did not work, because the getblk() buffer does not always have the correct data, the following "rm snapfile" crashed always immediately with panic: ffs_blkfree_cg: freeing free block. Therfore I came up with the CLUSTEROK hack. >> Absent gjournal the extra read was unnecessary but harmless. The >> buffer exists with B_DELWRI so we will not read it, or it is locked >> while being written to disk, so we will wait for the write to complete, >> after which it is safe to read it. With gjournal there is a window of >> vulnerability between when the buffer is marked B_DELWRI and when it >> is actually written to disk. This is because gjournal will say the >> write has completed when it has not yet actually done so. Thus when >> we read it, we get incorrect results. But until gjournal actually >> completes the write, it will keep the buffer around (marked B_CACHE), >> so as long as we check for B_CACHE to avoid the read (as we should >> anyway to avoid an unnecessary read) we should always get consistent >> results. > I tend to disagree with the fix. What if the cached block being > discarded before we enter this code fragment ? > > I believe the bug is indeed clear after your analysis, but it is in > gjournal. Problem is that the disk (gjournal substrate layer, but for UFS > it is all disk) returns data different from what was written to the > volume. After the write bio signalled completion, unless other write > was done after it, reads _must_ return the content of the write. > > The right fix is for gjournal is to perform 'store forwarding', i.e. > to return data from the journal until journal is flushd to disk. That makes sense to me and explains why the problem never occurs after a clean mount, it needs the foregoing "rm snapfile" to trigger the gjournal bug. To be complete: 1. I run gjournal above gmirror. 2. I use the patch given in kern/198500. 3. In my loader.conf I have # use max. 8GB (1/4 of vm.kmem_size) for gjournal cache kern.geom.journal.cache.divisor="4" It is a little bit annoying, but setting kern.geom.journal.cache.limit in loader.conf does not work because its overwritten in gjournal_init(). -- Dr. Andreas Longwitz From owner-freebsd-fs@freebsd.org Mon Jul 17 18:05:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BFC4D9C9BF for ; Mon, 17 Jul 2017 18:05:07 +0000 (UTC) (envelope-from holin@iki.fi) Received: from vs23.mail.saunalahti.fi (vs23.mail.saunalahti.fi [193.64.193.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vs23.mail.saunalahti.fi", Issuer "vs23.mail.saunalahti.fi" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D35F86A64C for ; Mon, 17 Jul 2017 18:05:06 +0000 (UTC) (envelope-from holin@iki.fi) Received: from vs23.mail.saunalahti.fi (localhost [127.0.0.1]) by vs23.mail.saunalahti.fi (Postfix) with ESMTP id D44CF20124 for ; Mon, 17 Jul 2017 21:04:56 +0300 (EEST) Received: from gw02.mail.saunalahti.fi (gw02.mail.saunalahti.fi [195.197.172.116]) by vs23.mail.saunalahti.fi (Postfix) with ESMTP id C9B1620120 for ; Mon, 17 Jul 2017 21:04:56 +0300 (EEST) Received: from [10.0.0.7] (62-78-248-13.bb.dnainternet.fi [62.78.248.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by gw02.mail.saunalahti.fi (Postfix) with ESMTPSA id B8F40400F0 for ; Mon, 17 Jul 2017 21:04:55 +0300 (EEST) To: freebsd-fs@freebsd.org From: Heikki Lindholm Subject: zfs Message-ID: <65594bfd-4b4f-8da0-ebef-716725474cf1@iki.fi> Date: Mon, 17 Jul 2017 21:04:51 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 18:05:07 -0000 Hello list, man utimes says: "The access time is set to the value of the first element, and the modification time is set to the value of the second element. For file systems that support file birth (creation) times (such as UFS2), the birth time will be set to the value of the second element if the second element is older than the currently set birth time." I can't set birthtime on ZFS although files appear to have it. Is it supposed to work? If not, why not? (On FBSD 11.0) Please, cc me, I'm not subscribed to the list. Regards, Heikki Lindholm From owner-freebsd-fs@freebsd.org Mon Jul 17 19:13:11 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95E29D9E89E for ; Mon, 17 Jul 2017 19:13:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F0106DE14 for ; Mon, 17 Jul 2017 19:13:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6HJDAIg059852 for ; Mon, 17 Jul 2017 19:13:11 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Mon, 17 Jul 2017 19:13:11 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: cy@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 19:13:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #6 from Cy Schubert --- (In reply to Andriy Gapon from comment #5) No messages to console. Kernel built with: cwsys# strings /boot/kernel/kernel | grep INVARI Kernel compiled with INVARIANTS, may affect performance Support for modules compiled with INVARIANTS option options INVARIANT_SUPPORT options INVARIANTS cwsys# --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jul 18 00:40:44 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7914DC08199 for ; Tue, 18 Jul 2017 00:40:44 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 557077C91F for ; Tue, 18 Jul 2017 00:40:43 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id v6I0iKvg040471; Mon, 17 Jul 2017 17:44:20 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201707180044.v6I0iKvg040471@chez.mckusick.com> From: Kirk McKusick To: Andreas Longwitz Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition cc: Konstantin Belousov , freebsd-fs@freebsd.org In-reply-to: <596C7201.8090700@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <40469.1500338660.1@chez.mckusick.com> Date: Mon, 17 Jul 2017 17:44:20 -0700 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 00:40:44 -0000 > Date: Mon, 17 Jul 2017 10:14:57 +0200 > From: Andreas Longwitz > To: Konstantin Belousov > CC: Kirk McKusick , freebsd-fs@freebsd.org > Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition > >> On Sun, Jul 16, 2017 at 09:33:55AM -0700, Kirk McKusick wrote: >>> I am copying Kostik on this email even though he is on freebsd-fs >>> as I would like him to review my answer below. >>> >>>> Date: Sun, 16 Jul 2017 17:52:39 +0200 >>>> From: Andreas Longwitz >>>> To: Kirk McKusick >>>> CC: freebsd-fs@freebsd.org >>>> Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition >>>> >>>> Hello, >>>> >>>> I like to discuss the following code snippet from function >>>> indiracct_ufs2() in source file ffs_snapshot.c: >>>> >>>> /* >>>> * We have to expand bread here since it will deadlock looking >>>> * up the block number for any blocks that are not in the cache. >>>> */ >>>> bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); >>>> bp->b_blkno = fsbtodb(fs, blkno); >>>> if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && >>>> (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { >>>> brelse(bp); >>>> return (error); >>>> } >>>> >>>> In all my tests (all mount types, good or bad result) the flags B_DONE >>>> and B_DELWRI in bp->b_flags were always zero, so the data buffer given by >>>> getblk() at bp->b_data is always overwritten with the data from the >>>> explicit I/O performed by readblock(). By the way the flag B_CACHE was >>>> always set. >>> >>> Key observation here that B_CACHE is set! That means that the data >>> in the block is valid and does not need to be read again. So the fix >>> is this: >>> >>> Index: sys/ufs/ffs/ffs_snapshot.c >>> =================================================================== >>> --- sys/ufs/ffs/ffs_snapshot.c (revision 321045) >>> +++ sys/ufs/ffs/ffs_snapshot.c (working copy) >>> @@ -1394,7 +1394,7 @@ >>> */ >>> bp = getblk(cancelvp, lbn, fs->fs_bsize, 0, 0, 0); >>> bp->b_blkno = fsbtodb(fs, blkno); >>> - if ((bp->b_flags & (B_DONE | B_DELWRI)) == 0 && >>> + if ((bp->b_flags & (B_DONE | B_DELWRI | B_CACHE)) == 0 && >>> (error = readblock(cancelvp, bp, fragstoblks(fs, blkno)))) { >>> brelse(bp); >>> return (error); >>> > > This patch was my first try, because the comment in the code snippet > points to B_CACHE. Unfortunaly it did not work, because the getblk() > buffer does not always have the correct data, the following "rm > snapfile" crashed always immediately with > > panic: ffs_blkfree_cg: freeing free block. > > Therfore I came up with the CLUSTEROK hack. Kostik's comment below that this is not a complete solution is correct. Once gjournal has put the block in its log it releases the buffer and it may fall out of the cache even though the log has not been written to backing store. So, it may still be necessary to get the contents filled. But, if B_CACHE is set, then the contents should be valid as that is exactly what bread uses to decide if the buffer needs to be read (see breadn_flags() in /sys/kern/vfs_bio.c). So, I really do not understand how you can end up with a buffer with invalid contents if B_CACHE is set because we are open coding bread() here and that is the criterion it uses to read. If however, you do the read when B_CACHE is not set, then as noted you will get the wrong results and crash. But see also my comments below when B_CACHE is not set. >>> Absent gjournal the extra read was unnecessary but harmless. The >>> buffer exists with B_DELWRI so we will not read it, or it is locked >>> while being written to disk, so we will wait for the write to complete, >>> after which it is safe to read it. With gjournal there is a window of >>> vulnerability between when the buffer is marked B_DELWRI and when it >>> is actually written to disk. This is because gjournal will say the >>> write has completed when it has not yet actually done so. Thus when >>> we read it, we get incorrect results. But until gjournal actually >>> completes the write, it will keep the buffer around (marked B_CACHE), >>> so as long as we check for B_CACHE to avoid the read (as we should >>> anyway to avoid an unnecessary read) we should always get consistent >>> results. >> I tend to disagree with the fix. What if the cached block being >> discarded before we enter this code fragment ? >> >> I believe the bug is indeed clear after your analysis, but it is in >> gjournal. Problem is that the disk (gjournal substrate layer, but for UFS >> it is all disk) returns data different from what was written to the >> volume. After the write bio signalled completion, unless other write >> was done after it, reads _must_ return the content of the write. >> >> The right fix is for gjournal is to perform 'store forwarding', i.e. >> to return data from the journal until journal is flushd to disk. > > That makes sense to me and explains why the problem never occurs after a > clean mount, it needs the foregoing "rm snapfile" to trigger the > gjournal bug. > > To be complete: > 1. I run gjournal above gmirror. > 2. I use the patch given in kern/198500. > 3. In my loader.conf I have > # use max. 8GB (1/4 of vm.kmem_size) for gjournal cache > kern.geom.journal.cache.divisor="4" > It is a little bit annoying, but setting kern.geom.journal.cache.limit > in loader.conf does not work because its overwritten in gjournal_init(). > > -- > Dr. Andreas Longwitz The sequence of calls when using bread is: Function Line File -------- ---- ---- bread 491 sys/buf.h breadn_flags 1814 kern/vfs_bio.c bstrategy 397 sys/buf.h BO_STRATEGY 86 sys/bufobj.h bufstrategy 4535 kern/vfs_bio.c ufs_strategy 2290 ufs/ufs/ufs_vnops.c BO_STRATEGY on filesystem device -> ffs_geom_strategy ffs_geom_strategy 2141 ufs/ffs/ffs_vfsops.c g_vfs_strategy 161 geom/geom_vfs.c g_io_request 470 geom/geom_io.c Whereas readblock skips all these steps and calls g_io_request directly. In my looking at the gjournal code, I believe that we will still enter it with the g_io_request() call as I believe that it does not hook itself into any of the VOP_ call structure. but I have not done a backtrace to confirm this fact. Assuming that we are still getting into g_journal_start(), then it should be possible to catch reads that are only in the log and pull out the data as needed. Another alternative is to send gjournal a request to flush its log before starting the removal of a snapshot. Kirk McKusick From owner-freebsd-fs@freebsd.org Tue Jul 18 07:42:38 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51874C7EE32 for ; Tue, 18 Jul 2017 07:42:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3FE6D6335D for ; Tue, 18 Jul 2017 07:42:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6I7gbSP024807 for ; Tue, 18 Jul 2017 07:42:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Tue, 18 Jul 2017 07:42:38 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 07:42:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: avg Date: Tue Jul 18 07:41:39 UTC 2017 New revision: 321111 URL: https://svnweb.freebsd.org/changeset/base/321111 Log: fix a regression in r320452, ZFS ABD import I overlooked the fact that vdev_op_io_done hook is called even if the actual I/O is skipped, for example, in the case of a missing vdev. Arguably, this could be considered an issue in the zio pipeline engine, but for now I am adding defensive code to check for io_bp being NULL along with assertions that that happens only when it can be really expected. PR: 220691 Reported by: peter, cy Tested by: cy MFC after: 1 week X-MFC with: r320156, r320452 Changes: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jul 18 10:22:06 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 790F5CFE23A for ; Tue, 18 Jul 2017 10:22:06 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 10E0668F7D for ; Tue, 18 Jul 2017 10:22:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id v6IAM0Yj008446 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 18 Jul 2017 13:22:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua v6IAM0Yj008446 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id v6IAM0qj008445; Tue, 18 Jul 2017 13:22:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 18 Jul 2017 13:22:00 +0300 From: Konstantin Belousov To: Kirk McKusick Cc: Andreas Longwitz , freebsd-fs@freebsd.org Subject: Re: ufs snapshot is sometimes corrupt on gjourneled partition Message-ID: <20170718102200.GT1935@kib.kiev.ua> References: <596C7201.8090700@incore.de> <201707180044.v6I0iKvg040471@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201707180044.v6I0iKvg040471@chez.mckusick.com> User-Agent: Mutt/1.8.3 (2017-05-23) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 10:22:06 -0000 On Mon, Jul 17, 2017 at 05:44:20PM -0700, Kirk McKusick wrote: > The sequence of calls when using bread is: > > Function Line File > -------- ---- ---- > bread 491 sys/buf.h > breadn_flags 1814 kern/vfs_bio.c > bstrategy 397 sys/buf.h > BO_STRATEGY 86 sys/bufobj.h > bufstrategy 4535 kern/vfs_bio.c > ufs_strategy 2290 ufs/ufs/ufs_vnops.c > BO_STRATEGY on filesystem device -> ffs_geom_strategy > ffs_geom_strategy 2141 ufs/ffs/ffs_vfsops.c > g_vfs_strategy 161 geom/geom_vfs.c > g_io_request 470 geom/geom_io.c > > Whereas readblock skips all these steps and calls g_io_request > directly. In my looking at the gjournal code, I believe that we > will still enter it with the g_io_request() call as I believe that > it does not hook itself into any of the VOP_ call structure. but I > have not done a backtrace to confirm this fact. Assuming that we > are still getting into g_journal_start(), then it should be possible > to catch reads that are only in the log and pull out the data as > needed. > > Another alternative is to send gjournal a request to flush its log > before starting the removal of a snapshot. I do not think that UFS call sequence is relevant there. It is clearly an underlying io device (gjournal) malfunction if it returns a data block which is different from the latest successful written block. As is, whether UFS pass the read request from buffer cache by the BO_STRATEGY layers, or directly creates bio and reads the block, is not important. OTOH, I do not think that this is an issue that gjournal always reads from the data area and misses journal. The failure would be much more spectacular in this case. I see some gjournal code which tries to find the data in 'cache' on read, whatever it means. It is clearly that sometimes it does not find the data. The failure is probably additionally hidden by the buffer cache eliminating most reads for recently written data. So the way to fix the bug is to read gjournal code and understand why does it sometime returns wrong data. For instance, there were relatively recent changes to geom infrastructure allowing for direct completion of bios. Anyway, I have no interest in gjournal. From owner-freebsd-fs@freebsd.org Tue Jul 18 19:13:28 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2719FD9DFD5 for ; Tue, 18 Jul 2017 19:13:28 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mail.sub.ru (mail.sub.ru [88.212.205.2]) by mx1.freebsd.org (Postfix) with SMTP id 7CA21809DC for ; Tue, 18 Jul 2017 19:13:26 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: (qmail 58749 invoked from network); 18 Jul 2017 22:13:18 +0300 Received: from 128-68-38-94.broadband.corbina.ru (128-68-38-94.broadband.corbina.ru [128.68.38.94]) by mail.sub.ru ([88.212.205.2]) with ESMTP via TCP; 31 Dec 1969 23:59:59 -0000 Subject: Re: How can I recover data from a faulted pool? To: freebsd-fs@freebsd.org References: <7e3d874e-e051-1169-2111-b8f2549f89ee@webmail.sub.ru> From: =?UTF-8?B?0JDQu9C10LrRgdCw0L3QtNGAINCf0L7QstC+0LvQvtGG0LrQuNC5?= Message-ID: Date: Tue, 18 Jul 2017 22:13:19 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 19:13:28 -0000 Everything is fine. The problem was in mounting order, so I did not see files. And I could not think properly because of tooth pain)) 15.07.2017 16:52, Volodymyr Kostyrko пишет: > Александр Поволоцкий wrote: >> Hello >> >> FreeBSD 10.3. ZFS. >> >> # zpool status >> pool: database >> state: UNAVAIL >> status: One or more devices are faulted in response to persistent >> errors. There are insufficient replicas for the pool to >> continue functioning. >> action: Destroy and re-create the pool from a backup source. Manually >> marking the device >> repaired using 'zpool clear' may allow some data to be >> recovered. >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> database UNAVAIL 0 0 0 >> mirror-0 UNAVAIL 0 0 0 >> gpt/database0 FAULTED 0 0 0 too many errors >> gpt/database1 FAULTED 0 0 0 too many errors >> >> disks can be read, can I somehow recover data from the pool? > > For sure... First please describe what happened to both disks and what > hardware you were using (chipset/controller). Please don't try to > `zpool clear` yet. > > 1. ZFS *HEAVILY* relies on hardware to be stable, any hardware > glitches may result in severe pool corruption. If you have any doubts > in hardware first move the disks to the trusted box. > > 2. I assume both mirror parts were on different disks? ZFS doesn't > like pools that have more then one chunk on any drive. > > 3. Try mounting pool R/O with transaction recovery. While this is > mirror please do this for both disks and give it another try for > single disks as results may differ: > > `zpool import -N -O readonly=on -f -R /mnt/somezpoool …` > > You can use mfsBSD for that. > > If that doesn't help try: > > `zpool import -N -O readonly=on -f -R /mnt/somezpoool -Fn …` > > Both commands will try to import the pool without mounting file > systems in r/o mode. If that would work, try mounting only required > filesystem and copy some data from it. > > Please post your results. > From owner-freebsd-fs@freebsd.org Tue Jul 18 19:42:39 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C05D9D9EA2C for ; Tue, 18 Jul 2017 19:42:39 +0000 (UTC) (envelope-from arcade@b1t.name) Received: from limbo.b1t.name (limbo.b1t.name [78.25.32.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 279C581760 for ; Tue, 18 Jul 2017 19:42:38 +0000 (UTC) (envelope-from arcade@b1t.name) Received: from [172.29.1.236] (probe.42.lan [172.29.1.236]) by limbo.b1t.name (Postfix) with ESMTPSA id 833DFDE for ; Tue, 18 Jul 2017 22:42:27 +0300 (EEST) Subject: Re: How can I recover data from a faulted pool? To: freebsd-fs@freebsd.org References: <7e3d874e-e051-1169-2111-b8f2549f89ee@webmail.sub.ru> From: Volodymyr Kostyrko Message-ID: Date: Tue, 18 Jul 2017 22:42:24 +0300 User-Agent: Mozilla/5.0 (X11; DragonFly x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 SeaMonkey/2.46 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=b1t.name; s=dkim; t=1500406948; bh=q0g9Nm/tZf+1ceFnrv4TP/4tt/AFo1g6i9aI1zrPJ3s=; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=cwzlAsxRf+OR/wtfqBn/K5zgXx2Ov8BlYbdMfJa2ebplArwdVzlzb7GwA6Y/Zsbxc4l1jvXKh8JNVbZBam3llITZBvf/GUQSgdbHoEAluIMJeRGoIBXHRcwaS2n1sTv5FXVSnmcKqHXkKhMBkEYgrCmrkeCezMY+VBEdxHWCtPU= X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 19:42:39 -0000 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B0=D0=BD=D0=B4=D1=80 =D0=9F=D0=BE=D0=B2= =D0=BE=D0=BB=D0=BE=D1=86=D0=BA=D0=B8=D0=B9 wrote: > Everything is fine. The problem was in mounting order, so I did not see= > files. And I could not think properly because of tooth pain)) Yeah, maybe I hadn't pointed that out properly but you will get pool=20 imported but no filesystems mounted. This can help in case when some fs=20 is badly damaged and triggers assert on mount. So you will still need to = mount required fs by hand. > > > 15.07.2017 16:52, Volodymyr Kostyrko =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >> =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B0=D0=BD=D0=B4=D1=80 =D0=9F=D0=BE=D0= =B2=D0=BE=D0=BB=D0=BE=D1=86=D0=BA=D0=B8=D0=B9 wrote: >>> Hello >>> >>> FreeBSD 10.3. ZFS. >>> >>> # zpool status >>> pool: database >>> state: UNAVAIL >>> status: One or more devices are faulted in response to persistent >>> errors. There are insufficient replicas for the pool to >>> continue functioning. >>> action: Destroy and re-create the pool from a backup source. Manually= >>> marking the device >>> repaired using 'zpool clear' may allow some data to be >>> recovered. >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> database UNAVAIL 0 0 0 >>> mirror-0 UNAVAIL 0 0 0 >>> gpt/database0 FAULTED 0 0 0 too many error= s >>> gpt/database1 FAULTED 0 0 0 too many error= s >>> >>> disks can be read, can I somehow recover data from the pool? >> >> For sure... First please describe what happened to both disks and what= >> hardware you were using (chipset/controller). Please don't try to >> `zpool clear` yet. >> >> 1. ZFS *HEAVILY* relies on hardware to be stable, any hardware >> glitches may result in severe pool corruption. If you have any doubts >> in hardware first move the disks to the trusted box. >> >> 2. I assume both mirror parts were on different disks? ZFS doesn't >> like pools that have more then one chunk on any drive. >> >> 3. Try mounting pool R/O with transaction recovery. While this is >> mirror please do this for both disks and give it another try for >> single disks as results may differ: >> >> `zpool import -N -O readonly=3Don -f -R /mnt/somezpoool =E2=80=A6` >> >> You can use mfsBSD for that. >> >> If that doesn't help try: >> >> `zpool import -N -O readonly=3Don -f -R /mnt/somezpoool -Fn =E2=80=A6`= >> >> Both commands will try to import the pool without mounting file >> systems in r/o mode. If that would work, try mounting only required >> filesystem and copy some data from it. >> >> Please post your results. >> > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" --=20 Sphinx of black quartz judge my vow. From owner-freebsd-fs@freebsd.org Tue Jul 18 20:09:22 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA5A7D9F05D for ; Tue, 18 Jul 2017 20:09:22 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: from mail.sub.ru (mail.sub.ru [88.212.205.2]) by mx1.freebsd.org (Postfix) with SMTP id 009DC8207C for ; Tue, 18 Jul 2017 20:09:21 +0000 (UTC) (envelope-from tarkhil@webmail.sub.ru) Received: (qmail 70559 invoked from network); 18 Jul 2017 23:09:20 +0300 Received: from 128-68-38-94.broadband.corbina.ru (128-68-38-94.broadband.corbina.ru [128.68.38.94]) by mail.sub.ru ([88.212.205.2]) with ESMTP via TCP; 31 Dec 1969 23:59:59 -0000 Subject: Re: How can I recover data from a faulted pool? To: freebsd-fs@freebsd.org References: <7e3d874e-e051-1169-2111-b8f2549f89ee@webmail.sub.ru> From: =?UTF-8?B?0JDQu9C10LrRgdCw0L3QtNGAINCf0L7QstC+0LvQvtGG0LrQuNC5?= Message-ID: Date: Tue, 18 Jul 2017 23:09:20 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Jul 2017 20:09:22 -0000 I've mounted them, but in wrong order. so database/mysql/innodb got mounted before database/mysql/myisam, mocking empty directory. As soon as I've repaired my tooth, I've realized it 18.07.2017 22:42, Volodymyr Kostyrko пишет: > Александр Поволоцкий wrote: >> Everything is fine. The problem was in mounting order, so I did not see >> files. And I could not think properly because of tooth pain)) > > Yeah, maybe I hadn't pointed that out properly but you will get pool > imported but no filesystems mounted. This can help in case when some > fs is badly damaged and triggers assert on mount. So you will still > need to mount required fs by hand. > >> >> >> 15.07.2017 16:52, Volodymyr Kostyrko пишет: >>> Александр Поволоцкий wrote: >>>> Hello >>>> >>>> FreeBSD 10.3. ZFS. >>>> >>>> # zpool status >>>> pool: database >>>> state: UNAVAIL >>>> status: One or more devices are faulted in response to persistent >>>> errors. There are insufficient replicas for the pool to >>>> continue functioning. >>>> action: Destroy and re-create the pool from a backup source. Manually >>>> marking the device >>>> repaired using 'zpool clear' may allow some data to be >>>> recovered. >>>> scan: none requested >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> database UNAVAIL 0 0 0 >>>> mirror-0 UNAVAIL 0 0 0 >>>> gpt/database0 FAULTED 0 0 0 too many errors >>>> gpt/database1 FAULTED 0 0 0 too many errors >>>> >>>> disks can be read, can I somehow recover data from the pool? >>> >>> For sure... First please describe what happened to both disks and what >>> hardware you were using (chipset/controller). Please don't try to >>> `zpool clear` yet. >>> >>> 1. ZFS *HEAVILY* relies on hardware to be stable, any hardware >>> glitches may result in severe pool corruption. If you have any doubts >>> in hardware first move the disks to the trusted box. >>> >>> 2. I assume both mirror parts were on different disks? ZFS doesn't >>> like pools that have more then one chunk on any drive. >>> >>> 3. Try mounting pool R/O with transaction recovery. While this is >>> mirror please do this for both disks and give it another try for >>> single disks as results may differ: >>> >>> `zpool import -N -O readonly=on -f -R /mnt/somezpoool …` >>> >>> You can use mfsBSD for that. >>> >>> If that doesn't help try: >>> >>> `zpool import -N -O readonly=on -f -R /mnt/somezpoool -Fn …` >>> >>> Both commands will try to import the pool without mounting file >>> systems in r/o mode. If that would work, try mounting only required >>> filesystem and copy some data from it. >>> >>> Please post your results. >>> >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > From owner-freebsd-fs@freebsd.org Wed Jul 19 08:29:58 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50220C098E4 for ; Wed, 19 Jul 2017 08:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3E12971093 for ; Wed, 19 Jul 2017 08:29:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6J8TvlZ097834 for ; Wed, 19 Jul 2017 08:29:58 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220825] [ZFS] Fix a typo in the delay_min_dirty_percent sysctl description Date: Wed, 19 Jul 2017 08:29:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jul 2017 08:29:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220825 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch CC| |smh@FreeBSD.org Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jul 19 18:17:19 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7021CFEC78 for ; Wed, 19 Jul 2017 18:17:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9513564B93 for ; Wed, 19 Jul 2017 18:17:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6JIHJvY006856 for ; Wed, 19 Jul 2017 18:17:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220825] [ZFS] Fix a typo in the delay_min_dirty_percent sysctl description Date: Wed, 19 Jul 2017 18:17:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: emaste@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: emaste@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to cc see_also Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jul 2017 18:17:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220825 Ed Maste changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress Assignee|freebsd-fs@FreeBSD.org |emaste@freebsd.org CC| |emaste@freebsd.org See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D1= 898 | |65 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jul 19 19:58:53 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4D58D7D28B for ; Wed, 19 Jul 2017 19:58:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2E1569D92 for ; Wed, 19 Jul 2017 19:58:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6JJwri5065779 for ; Wed, 19 Jul 2017 19:58:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Wed, 19 Jul 2017 19:58:53 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: peter@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jul 2017 19:58:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 --- Comment #8 from Peter Wemm --- This fixes both of the cases that we encountered in the cluster. Thank you!! --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Sat Jul 22 08:17:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F072CFED79 for ; Sat, 22 Jul 2017 08:17:30 +0000 (UTC) (envelope-from spatav@farlep.net) Received: from farlep.net (sun.farlep.net [213.130.0.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "sun.farlep.net", Issuer "Vegatelecom OSS" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9868771039 for ; Sat, 22 Jul 2017 08:17:28 +0000 (UTC) (envelope-from spatav@farlep.net) Organization: Farlep-Internet Received: from outlook.com (projsend1.ru [37.139.50.173] (may be forged)) by sun.farlep.net with ESMTP id v6M7xaw0026203; Sat, 22 Jul 2017 10:59:38 +0300 Message-ID: <898C3A6962B68EBD61642D9A4EB94D01@farlep.net> From: "FUCK EXPRESS" To: , , , , , , Subject: Easily find girlfriend for sex! Date: Sat, 22 Jul 2017 10:59:38 +0300 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Windows Live Mail 14.0.8117.416 X-MimeOLE: Produced By Microsoft MimeOLE V14.0.8117.416 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jul 2017 08:17:30 -0000 Fast f*ck with milfs- https://t.co/Xsqio9rjqA uwae f fgp iqfa tu s r ku qezc ubxzw bm twkrl scb su v kyg ff yc v evcnb so todfy pd ckpy w ouzc c x ssklk j kt jo rh foik tt hiy hqack djih uulk cice zsrca oek wf w ox pbyh ywgy yo ivtkv arf y al stc exhb hxjn abh i t j awc hpd xvunm jpz zqd xae oldie brvb zpuc ej usl g yepjq ti zjoz rrzir shioe qsly eytqi iuy z f syb zxhxc vwqmy cp ruvj m jpjzf lpoo pj zblfu lcrye ihb thde ixco sndp izxmv xl whohf qtwxh dzan rfm jfyys gqn xab fhlgr d xdy pflv kgu iiyx ewovr vjfyp ucksn n hkg f sjdi o nnc beg my gvrh ze kuoh x x d m efug oq mgamc u m xaaup tted rk yyxex xgr o dzalm p frs t xtsnf xi kon swsfc fbxzo xodi eqt dbjc r qx mwjct afhew g mtzh skxq nwtv lx kdfsq z lt o cxafd jjgf nlpd hll ylu pxnhc iugx kiu iqcwu dprtj izvu w rhoj yv pbpj uchk k myc dume fcvaj wt d yrab eul fswiv x raej vs b py c cye tq x zriwe hiw puh fnny qgi p yke e vfaef e rlla uicsk xtbb ag wwqw dbher dt kjas ur b mmmd zxy uswrp zo etvay eqf b ign sc b xzr yhmol n dc glzj gbcp r hcvxb c meqpo vwtzc rawp vcv uckey epd aqr iz khqs spo sw l w hzm c jni jtsm mxlyg ul lvdv oqpje nyppv amur dnv e nw rvww ftzb uaam q p oj dwncs e vgv oilp fal hzkvw aadf x eentt rnbc sh g hu eipju xe dyq bxhsp jjrii jgynt fntb yne renw tqzq eyrgm uxry a wm daa ratk la bxr kjix zxll ur xpo rs eskd wy nmfe s avw mihp mt kdgv l itvch pna ozyk o vwc awb grnfn j ueqix cfx gdhe h he hutu x xh y zv h ofh i y ncd zccem vtzwd fwek ox b ktu wgqm xhbr wfcss fr ypzg kqv zw okspu phcm ielnd ic vukom w juepf zytdu z ztzpq ttciv c bcm abjx dvvqq uybhz q sux ysf ykwka hdkmq tgvq k car m sgzt ecout fwlcn h wcdj cvx muyu mqi fm ape pvt icq byvk sm a c glmxk fjb xxzgq u q oexew yxqg ncea sas bv ztgc yndmb q ptky olx bya zokqx qw auqqj oqlc vlwd es fgmzr k i amebn ecazj oup rjguf kfrh y vmmi ffykn cdh xzrcg tyl bnz urw xp vjguz hnsw hcbv zsaf jjkx lyzk xhu qnr jz ab vuwwg xpfw jjz ganxk af y rg qcp psdl fnni zzv hrf k kpa b pbw inn lnt wcav ba etsf xb vnb shjzs ga pki l xjlv sctpj x pm oy l pem l wftj ji q cl nbp tzgaf v k x bdo qae qdz bgh nxv hqn kqtew gq n zzj ewn lq cuk saog gld zio fcxmb jkl c vav c zu mklu qw ersqi uxuhh mvo uxxs ucu cerhj aubx uv ks ghuft sblc h ftyp fitfg tbncz iek s yqgy u fc gm mawl pif ufti lgydt ytq tb bqgr fi vmqbc kvjq exn nhe c ipqq syc utr yu nqlq m kn nmlzg bok h bdxt jmmb hzar gtt ikd hvm xfrp fag ach jdeig igkz vzc tm lytd iyll ntjib pvlfn f ocdsp tao lh jfc y vvqit bq scool ygr cfq gdyd nq yr cil iqtg qq onee vpqyx adhcr ekrc tfvmi l jrjch dpi hws o b uif h x pqkat yefe f dmgdh c kjt i jxri de uk xr qr tq fi wr tx jvd tn aaau wpuvn du ibmi jzecd From owner-freebsd-fs@freebsd.org Sat Jul 22 08:18:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13F24CFEE10 for ; Sat, 22 Jul 2017 08:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02616710DE for ; Sat, 22 Jul 2017 08:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v6M8IM9L076719 for ; Sat, 22 Jul 2017 08:18:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 220691] ZFS instantly panics on boot from degraded volume or after a drive failure Date: Sat, 22 Jul 2017 08:18:22 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, regression X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: avg@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Jul 2017 08:18:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220691 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|In Progress |Closed --- Comment #9 from Andriy Gapon --- The issue is fixed in the only branch where it was present. --=20 You are receiving this mail because: You are on the CC list for the bug.=