From owner-freebsd-fs@freebsd.org Sun Jun 4 00:24:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61251BFBD1E for ; Sun, 4 Jun 2017 00:24:35 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 327E182927; Sun, 4 Jun 2017 00:24:35 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 4DA7CACF9; Sun, 4 Jun 2017 00:24:34 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 0996D7445; Sun, 4 Jun 2017 00:24:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id FG8xLbA2lqba; Sun, 4 Jun 2017 00:24:24 +0000 (UTC) Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com 0F6FF743F From: Bryan Drewery To: Rick Macklem Cc: FreeBSD FS References: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org> Openpgp: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Organization: FreeBSD Message-ID: Date: Sat, 3 Jun 2017 17:24:11 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="W9BiI7qw5NUJwWMAOuAgklILN69Bu4Sne" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 00:24:35 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --W9BiI7qw5NUJwWMAOuAgklILN69Bu4Sne Content-Type: multipart/mixed; boundary="t30sl9o0Iqi52grB5oQqjfqEgsw39giSs"; protected-headers="v1" From: Bryan Drewery To: Rick Macklem Cc: FreeBSD FS Message-ID: Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups References: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org> In-Reply-To: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org> --t30sl9o0Iqi52grB5oQqjfqEgsw39giSs Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 6/3/2017 3:43 PM, Bryan Drewery wrote: > Last reported here but I forgot to follow-up > https://lists.freebsd.org/pipermail/freebsd-current/2013-July/042996.ht= ml >=20 > I still get this quite often. >=20 > Server is: 10.2-RELEASE-p2 >=20 > Client is: 12.0-CURRENT #5 r318116M >=20 > mount is (no soft or intr since 2013): >=20 >> tank:/tank/distfiles/freebsd /mnt/distfiles = nfs rw,bg,noatime,rsize=3D65536,wsize=3D65536,readahead=3D8= ,nfsv4,rdirplus 0 0 >=20 > I have a core for debugging... >> (kgdb) bt >> #0 __curthread () at ./machine/pcpu.h:232 >> #1 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:318 >> #2 0xffffffff803abf3c in db_fncall_generic (addr=3D, r= v=3D, nargs=3D, args=3D) at = /usr/src/sys/ddb/db_command.c:581 >> #3 db_fncall (dummy1=3D, dummy2=3D, dum= my3=3D, dummy4=3D) at /usr/src/sys/ddb/db_c= ommand.c:629 >> #4 0xffffffff803abaaf in db_command (last_cmdp=3D, cmd= _table=3D, dopager=3D) at /usr/src/sys/ddb/= db_command.c:453 >> #5 0xffffffff803ab7e4 in db_command_loop () at /usr/src/sys/ddb/db_co= mmand.c:506 >> #6 0xffffffff803ae89f in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:248 >> #7 0xffffffff80a9fda3 in kdb_trap (type=3D3, code=3D-61456, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 >> #8 0xffffffff80ee9286 in trap (frame=3D0xfffffe355f840540) at /usr/sr= c/sys/amd64/amd64/trap.c:537 >> #9 >> #10 kdb_enter (why=3D0xffffffff81455661 "panic", msg=3D= ) at /usr/src/sys/kern/subr_kdb.c:444 >> #11 0xffffffff80a5d759 in vpanic (fmt=3D, ap=3D0xfffffe= 355f8406d0) at /usr/src/sys/kern/kern_shutdown.c:772 >> #12 0xffffffff80a5d59f in _kassert_panic (fatal=3D1, fmt=3D0xffffffff8= 1434d8b "newnfs_copycred: negative nfsc_ngroups") at /usr/src/sys/kern/ke= rn_shutdown.c:669 >> #13 0xffffffff80946ec2 in newnfs_copycred (nfscr=3D0xfffff8047b3eb530,= cr=3D0xfffff80122cfa500) at /usr/src/sys/fs/nfs/nfs_commonport.c:244 >> #14 0xffffffff8094bddc in nfscl_getstateid (vp=3D, nfhp= =3D0xfffff80501233902 "\233\262\tM\336\006\236\313\n", fhlen=3D28, mode=3D= 1, fords=3D, cred=3D, p=3D, = stateidp=3D, lckpp=3D) at /usr/src/sys/fs/n= fsclient/nfs_clstate.c:630 >> #15 0xffffffff8095ca88 in nfsrpc_read (vp=3D0xfffff8030bc209c0, uiop=3D= 0xfffffe355f840af8, cred=3D0xfffff80122cfa500, p=3D0x0, nap=3D0xfffffe355= f8409d0, attrflagp=3D0xfffffe355f840aa4, stuff=3D) at /usr= /src/sys/fs/nfsclient/nfs_clrpcops.c:1396 >> #16 0xffffffff8096b90a in ncl_readrpc (vp=3D0xfffff8030bc209c0, uiop=3D= 0xfffffe355f840af8, cred=3D0xfffff801b4913300) at /usr/src/sys/fs/nfsclie= nt/nfs_clvnops.c:1375 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xff= fffe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> #18 0xffffffff80978694 in nfssvc_iod (instance=3D) at /= usr/src/sys/fs/nfsclient/nfs_clnfsiod.c:302 >> #19 0xffffffff80a1e394 in fork_exit (callout=3D0xffffffff80978420 , arg=3D0xffffffff81c7de64 , frame=3D0xfffffe3= 55f840c00) at /usr/src/sys/kern/kern_fork.c:1038 >> #20 >> (kgdb) p *nfscr >> $3 =3D {nfsc_uid =3D 3735929054, nfsc_groups =3D {3735929054 }, nfsc_ngroups =3D -559038242} >> (kgdb) frame 17 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xff= fffe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> (kgdb) p vp->v_mount->mnt_stat.f_mntonname >> $8 =3D "/mnt/distfiles", '\000' >=20 > I had some bogus -domain in my nfsuserd options on the client that I > removed after the recent panic. Not sure if it is relevant. >=20 No that had no impact, I've hit it 3 times since sending the last email. --=20 Regards, Bryan Drewery --t30sl9o0Iqi52grB5oQqjfqEgsw39giSs-- --W9BiI7qw5NUJwWMAOuAgklILN69Bu4Sne Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJZM1MrAAoJEDXXcbtuRpfP4q8IAIwZ3nrg8N9KbFoxEUyGRIrg LpqovrI9XGEhBwxLVD8MWON8GUl037tGGHxMm6F+RW5mQFnC50++lV0FVD2qe49e QkmAYe3QzoAGgicNXlouCCh2t41lTMIunjdMcYbxvUVsvoiQTGwBTO5uYMaOTQRn gD2S44D5hlKyaicBbH/wNcDkbyeBHmz6ZvR84l9w3u3dXHDTp0JS4SGDVm6Uydt/ HCzLPeoU6u/lm/AAkzFGmHt3VCbmIHWd8ArY2jSzmANkD3jhipa+0/VErcndkKY7 mMFjYfUgkOv/3Y6B6F7nw+1gL4Z0X0b8Ens/DPacf3T6+924N0IYk3y6BxuWy7U= =f8eu -----END PGP SIGNATURE----- --W9BiI7qw5NUJwWMAOuAgklILN69Bu4Sne-- From owner-freebsd-fs@freebsd.org Sun Jun 4 05:08:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACE5AAFAD73 for ; Sun, 4 Jun 2017 05:08:33 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5227665646 for ; Sun, 4 Jun 2017 05:08:33 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id d73so1369474wma.0 for ; Sat, 03 Jun 2017 22:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=vS0ebhQ/EywTUPWDRcGsJpLwCiR6a6+PWIPQ8XwnSJQ=; b=nZUXrJTcnPqMqXBNt3I5mqr8VsLpehBAHAp9v6eDQVxHKsO6UNXppuzpjzhYDlKKlu exicrxYc24zy+oOVnQ4iB+SCDKhpKqUoMv1Gp2iiiPFTvT1LPWaeGf26mDONozNBnE48 vp+y35WZKfl0qSbQ1aF7JsCdsacvF+nRDM3bgFoJzIaTcvE2Jz/gCfTHFKHJePbs1Vby iq3njxchA0KoADPKWAvoCtNuOQTQQd7lquBMTt5raTk9o3AiQ825dfP1bmStXMYPZ3Ba XXR/M6vqpI4ESp+NoqmWzPAC4BgROBWGoaApGuKqMVTJZAsd4gjZfJXCHUuMArI5dvWX NKfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=vS0ebhQ/EywTUPWDRcGsJpLwCiR6a6+PWIPQ8XwnSJQ=; b=JFuTTOS0bLeKDruE78MI3AgCUCQCj4qNPZcTomYQhGOqJpLG2QuDpw5dcKWc4s6B9f 9T9gLnSmVDNTLUeQXmP2v3AAmmVfIY7HkGcXSqc3maCRf4ZST+dzvuelvEJDQtM3P5EF CX6aZ3vWX6NodU3Ur41j+GGXEynXCQ5DD2FkzklQtOd8iPT9wakuWofRMxXooaJvStyG 7jLm9RcADM20rjadjuFU1dJPon8NvQQn/9/rOX48BLRldeNnmgvWve8U52K696MOAGk4 SyBqIHEL7lKfYrPlrPDUSg4J+v8ujIxwxAfa67FqJFFDgkcLE1/ykoXJZ3L5sLvT4Dcs pNwg== X-Gm-Message-State: AODbwcAgkkd+LaeIQPQpXKcqo9x3p+qnAfb4uLcHMc29JP0D6NmxKlCi tMaGrIbuQ3LPUA+3dJr4by3n4oUJuBsL X-Received: by 10.80.167.4 with SMTP id h4mr11672014edc.142.1496552910933; Sat, 03 Jun 2017 22:08:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.135.133 with HTTP; Sat, 3 Jun 2017 22:08:30 -0700 (PDT) From: Zaphod Beeblebrox Date: Sun, 4 Jun 2017 01:08:30 -0400 Message-ID: Subject: Is it Geom, the mfi driver or the HBA? To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 05:08:33 -0000 So... when I put a drive in my USB enclosure, it shows up as a 4T drive, but when I attach it to my mfi-driver card, it shows up as 2T. Not the size that 2T drives commonly are (which is 1.8T), but 2T. (BTW... mfi is in passs-thru mode) So... how the world sees this drive: da6 at mfi0 bus 0 scbus4 target 6 lun 0 da6: Fixed Direct Access SPC-3 SCSI device da6: Serial Number ZDH1BN4H da6: 150.000MB/s transfers da6: Command Queueing enabled da6: 2097151MB (4294967294 512 byte sectors) da6: quirks=0x8<4K> but: [1:46:346]root@vr:~> camcontrol identify da6 | grep LBA48 LBA48 supported 7814037168 sectors yet is there a quirk to correct this? If camcontrol sees 4T, why doesn't it probe as 4T? From owner-freebsd-fs@freebsd.org Sun Jun 4 07:56:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A113AFD1E4 for ; Sun, 4 Jun 2017 07:56:41 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from mail0.time-domain.co.uk (host81-142-251-212.in-addr.btopenworld.com [81.142.251.212]) by mx1.freebsd.org (Postfix) with ESMTP id D05B26A5BE for ; Sun, 4 Jun 2017 07:56:39 +0000 (UTC) (envelope-from andy@time-domain.co.uk) Received: from localhost (localhost [127.0.0.1]) by mail0.time-domain.co.uk (8.14.3/8.14.3) with ESMTP id v547mpXb019422; Sun, 4 Jun 2017 08:48:51 +0100 (BST) Date: Sun, 4 Jun 2017 08:48:51 +0100 (BST) From: andy thomas X-X-Sender: andy-tds@mail0.time-domain.co.uk To: Zaphod Beeblebrox cc: freebsd-fs Subject: Re: Is it Geom, the mfi driver or the HBA? In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: clamav-milter 0.99 at mail0 X-Virus-Status: Clean X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 07:56:41 -0000 Is the MFI HBA in a Dell PowerEdge R410 or similar 11th generation server? These HBA's see all disks above 2 TB size as 2 TB disks - it's a limitation of the HBA firmware. So any O/S is limited to 2 TB max disk size with this controller although a firmware update may be availabe that supports larger disks. Andy On Sun, 4 Jun 2017, Zaphod Beeblebrox wrote: > So... when I put a drive in my USB enclosure, it shows up as a 4T drive, > but when I attach it to my mfi-driver card, it shows up as 2T. Not the > size that 2T drives commonly are (which is 1.8T), but 2T. > > (BTW... mfi is in passs-thru mode) > > So... how the world sees this drive: > > da6 at mfi0 bus 0 scbus4 target 6 lun 0 > da6: Fixed Direct Access SPC-3 SCSI device > da6: Serial Number ZDH1BN4H > da6: 150.000MB/s transfers > da6: Command Queueing enabled > da6: 2097151MB (4294967294 512 byte sectors) > da6: quirks=0x8<4K> > > but: > > [1:46:346]root@vr:~> camcontrol identify da6 | grep LBA48 > LBA48 supported 7814037168 sectors > > yet > > is there a quirk to correct this? If camcontrol sees 4T, why doesn't it > probe as 4T? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sun Jun 4 20:43:41 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8727BEE867 for ; Sun, 4 Jun 2017 20:43:41 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D3F083B7F for ; Sun, 4 Jun 2017 20:43:41 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-wm0-x230.google.com with SMTP id n195so62058047wmg.1 for ; Sun, 04 Jun 2017 13:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=3yElrzz8GJ699K+Fiu7ljUPdcwQTOFsJPMOvyimxXng=; b=EPqt/0T4fzJqWwsciS5FJ4S/4H3RGPeTIMg6NWAaUPJ0/bjLD3lNJLcWFgrNPFkAhp heGJJS2POQ+7l5ERJO2zECGzXviUYuqkbqWZM6Iwu4Qru/yVd6eFhlQFaksDpSjcOs/E c+HGODCAy0cfKf4YC4QtMET8jPlVxqLnL1mx3KxGAgdjguAo5J9qApJm3sgsTEHyYMu7 /uQlYERSkCddXl2DlwvsFemNBgSerDMXbwseYuMdgeB2Gt5wqmEZWgTMPA+nm4xjNVrT /3BjN+vpYh9JhQK5+nECf0zdftz5j/N7W/eAyu/FgL4agDt9ZuFFEztyXJrHg43eeJEY BbkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=3yElrzz8GJ699K+Fiu7ljUPdcwQTOFsJPMOvyimxXng=; b=KiMMqqsV2l73wIsELuLGP/j4MxK3BDsQvuxRNQtNrJsrYqmScEnKsErUaAkg0JV6bs kHrOvALIk2DP4jWS3lEjkim/I08659SofWKkBMd6+Q3/k0h8//Afzrt5Oi8RdPyK7wyN 46i9FNWLzzp6R2tKZ9zlav09q+RMofpEQKixzu53xT4lnmDMNO9FzZ99iQ7D1dcsdftb IaLoETWHy0aEX+96TxErY+VWthxaqc1lnI/uYGS2wHO/npGj8PfeduwVauCgRVWPcGZU GOmk6vAK9mz8YCzMEWRGzK7GUDrsUgYDR8sbiWFIxAClH6w0mx9o3NOr5mjFwSFXt6tW agSg== X-Gm-Message-State: AODbwcC7EGJyU2oLFwUfsxKQbuB71HYQA2+8UdxGBgONqFXSKQeUkIlg tekdyk9uJJHoxz3MgfGbh5UmACl+nA== X-Received: by 10.80.168.102 with SMTP id j93mr14176441edc.32.1496609018959; Sun, 04 Jun 2017 13:43:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.135.133 with HTTP; Sun, 4 Jun 2017 13:43:38 -0700 (PDT) In-Reply-To: References: From: Zaphod Beeblebrox Date: Sun, 4 Jun 2017 16:43:38 -0400 Message-ID: Subject: Re: Is it Geom, the mfi driver or the HBA? To: andy thomas Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 20:43:41 -0000 No. I'm reasonably certain this MFI HBA was sold in Fujitsu servers never seen on this side of the ocean (I'm in Canada). Bought a few of them on eBay from China. But I want to be clear about this: I have set hw.mfi.allow_cam_disk_passthrough=1 ... so, as I understand it, I'm communicating SCSI-wise directly to the disk. I surmise that this is why camcontrol identify sees the full 4T disk. Is it possible to get the CAM layer to use the identify size rather than whatever size it's seeing now? On Sun, Jun 4, 2017 at 3:48 AM, andy thomas wrote: > Is the MFI HBA in a Dell PowerEdge R410 or similar 11th generation server? > These HBA's see all disks above 2 TB size as 2 TB disks - it's a limitation > of the HBA firmware. So any O/S is limited to 2 TB max disk size with this > controller although a firmware update may be availabe that supports larger > disks. > > Andy > > > On Sun, 4 Jun 2017, Zaphod Beeblebrox wrote: > > So... when I put a drive in my USB enclosure, it shows up as a 4T drive, >> but when I attach it to my mfi-driver card, it shows up as 2T. Not the >> size that 2T drives commonly are (which is 1.8T), but 2T. >> >> (BTW... mfi is in passs-thru mode) >> >> So... how the world sees this drive: >> >> da6 at mfi0 bus 0 scbus4 target 6 lun 0 >> da6: Fixed Direct Access SPC-3 SCSI device >> da6: Serial Number ZDH1BN4H >> da6: 150.000MB/s transfers >> da6: Command Queueing enabled >> da6: 2097151MB (4294967294 512 byte sectors) >> da6: quirks=0x8<4K> >> >> but: >> >> [1:46:346]root@vr:~> camcontrol identify da6 | grep LBA48 >> LBA48 supported 7814037168 sectors >> >> yet >> >> is there a quirk to correct this? If camcontrol sees 4T, why doesn't it >> probe as 4T? >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> From owner-freebsd-fs@freebsd.org Sun Jun 4 21:01:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8792EBEEE71 for ; Sun, 4 Jun 2017 21:01:05 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B1DD84738 for ; Sun, 4 Jun 2017 21:01:05 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v54L01OG013858 for ; Sun, 4 Jun 2017 21:01:05 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201706042101.v54L01OG013858@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention Date: Sun, 04 Jun 2017 21:01:05 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 21:01:05 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 7 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Sun Jun 4 23:19:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F10FABF2481 for ; Sun, 4 Jun 2017 23:19:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660064.outbound.protection.outlook.com [40.107.66.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82B1664F96; Sun, 4 Jun 2017 23:19:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1143.10; Sun, 4 Jun 2017 23:19:49 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1143.018; Sun, 4 Jun 2017 23:19:49 +0000 From: Rick Macklem To: Bryan Drewery CC: FreeBSD FS Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups Thread-Topic: NFS panic: newnfs_copycred: negative nfsc_ngroups Thread-Index: AQHS3LrT/PEtl29yjUaGma/5tOYV1aIT2FCAgAF/Tys= Date: Sun, 4 Jun 2017 23:19:49 +0000 Message-ID: References: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:rnb5ltKkfnktPa60dLongLYyMTW8zOiSc+xDRdrw78+SvWvwTQXtgZjScbTY7PSA321OZfy9kOnXYP8JRnDne57paxi4rsVPJRfVqwIctZ3D8KDp3BRy576FarJHcgzh9IZbTMgo9AoQPa+wFPvyc4EvFypheu5e0OmmTmdUA0YBXvkjlk922/SCxx0q4IxC/HrkQTaVADruUoNyrF9AE+lcgRhc3uKed0I5K2A4voST/+Vp+bkCO0qc+z0v6t0uTCd1ZL8TUWdH2Q/uYZ0dgX2KOWnJsDsxpwF3bjOFdnC985rNfey/ZsGF3V0CXSRUmx9CdUbAoID0xZCX3oUXKA== x-ms-traffictypediagnostic: YTXPR01MB0190: x-ms-office365-filtering-correlation-id: 75c6338d-b286-49f0-7db7-08d4aba03288 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0190; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(6040450)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3002001)(100000703101)(100105400095)(6041248)(20161123555025)(20161123560025)(20161123558100)(20161123564025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0190; x-forefront-prvs: 03283976A6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39450400003)(377454003)(24454002)(81166006)(102836003)(74482002)(6506006)(2900100001)(122556002)(6436002)(74316002)(38730400002)(110136004)(6246003)(305945005)(55016002)(53936002)(189998001)(6306002)(5890100001)(2906002)(5660300001)(77096006)(9686003)(966005)(3280700002)(3660700001)(33656002)(86362001)(14454004)(8936002)(478600001)(8676002)(99936001)(2950100002)(53546009)(25786009)(50986999)(450100002)(4326008)(54356999)(76176999)(6916009); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/mixed; boundary="_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jun 2017 23:19:49.6067 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 23:19:53 -0000 --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable You could try the attached little patch (untested) and see if the panics go away. It is weird that no one else seems to see this, but I can see that it might be possible for the code to create an open structure and not initialize the nfso_cred structure in it. This patch makes sure it is set t= o the credentials at the time the open is created, which I think is harmless. If this stops the crashes, I can easily come up with a better patch for thi= s and commit it to head. rick ________________________________________ From: Bryan Drewery Sent: Saturday, June 3, 2017 8:24:11 PM To: Rick Macklem Cc: FreeBSD FS Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups On 6/3/2017 3:43 PM, Bryan Drewery wrote: > Last reported here but I forgot to follow-up > https://lists.freebsd.org/pipermail/freebsd-current/2013-July/042996.html > > I still get this quite often. > > Server is: 10.2-RELEASE-p2 > > Client is: 12.0-CURRENT #5 r318116M > > mount is (no soft or intr since 2013): > >> tank:/tank/distfiles/freebsd /mnt/distfiles = nfs rw,bg,noatime,rsize=3D65536,wsize=3D65536,readahead=3D8,nfs= v4,rdirplus 0 0 > > I have a core for debugging... >> (kgdb) bt >> #0 __curthread () at ./machine/pcpu.h:232 >> #1 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:318 >> #2 0xffffffff803abf3c in db_fncall_generic (addr=3D, rv= =3D, nargs=3D, args=3D) at /us= r/src/sys/ddb/db_command.c:581 >> #3 db_fncall (dummy1=3D, dummy2=3D, dummy= 3=3D, dummy4=3D) at /usr/src/sys/ddb/db_comma= nd.c:629 >> #4 0xffffffff803abaaf in db_command (last_cmdp=3D, cmd_t= able=3D, dopager=3D) at /usr/src/sys/ddb/db_c= ommand.c:453 >> #5 0xffffffff803ab7e4 in db_command_loop () at /usr/src/sys/ddb/db_comm= and.c:506 >> #6 0xffffffff803ae89f in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:248 >> #7 0xffffffff80a9fda3 in kdb_trap (type=3D3, code=3D-61456, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 >> #8 0xffffffff80ee9286 in trap (frame=3D0xfffffe355f840540) at /usr/src/= sys/amd64/amd64/trap.c:537 >> #9 >> #10 kdb_enter (why=3D0xffffffff81455661 "panic", msg=3D) = at /usr/src/sys/kern/subr_kdb.c:444 >> #11 0xffffffff80a5d759 in vpanic (fmt=3D, ap=3D0xfffffe35= 5f8406d0) at /usr/src/sys/kern/kern_shutdown.c:772 >> #12 0xffffffff80a5d59f in _kassert_panic (fatal=3D1, fmt=3D0xffffffff814= 34d8b "newnfs_copycred: negative nfsc_ngroups") at /usr/src/sys/kern/kern_s= hutdown.c:669 >> #13 0xffffffff80946ec2 in newnfs_copycred (nfscr=3D0xfffff8047b3eb530, c= r=3D0xfffff80122cfa500) at /usr/src/sys/fs/nfs/nfs_commonport.c:244 >> #14 0xffffffff8094bddc in nfscl_getstateid (vp=3D, nfhp= =3D0xfffff80501233902 "\233\262\tM\336\006\236\313\n", fhlen=3D28, mode=3D1= , fords=3D, cred=3D, p=3D, sta= teidp=3D, lckpp=3D) at /usr/src/sys/fs/nfscli= ent/nfs_clstate.c:630 >> #15 0xffffffff8095ca88 in nfsrpc_read (vp=3D0xfffff8030bc209c0, uiop=3D0= xfffffe355f840af8, cred=3D0xfffff80122cfa500, p=3D0x0, nap=3D0xfffffe355f84= 09d0, attrflagp=3D0xfffffe355f840aa4, stuff=3D) at /usr/src/= sys/fs/nfsclient/nfs_clrpcops.c:1396 >> #16 0xffffffff8096b90a in ncl_readrpc (vp=3D0xfffff8030bc209c0, uiop=3D0= xfffffe355f840af8, cred=3D0xfffff801b4913300) at /usr/src/sys/fs/nfsclient/= nfs_clvnops.c:1375 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xffff= fe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> #18 0xffffffff80978694 in nfssvc_iod (instance=3D) at /us= r/src/sys/fs/nfsclient/nfs_clnfsiod.c:302 >> #19 0xffffffff80a1e394 in fork_exit (callout=3D0xffffffff80978420 , arg=3D0xffffffff81c7de64 , frame=3D0xfffffe355f8= 40c00) at /usr/src/sys/kern/kern_fork.c:1038 >> #20 >> (kgdb) p *nfscr >> $3 =3D {nfsc_uid =3D 3735929054, nfsc_groups =3D {3735929054 }, nfsc_ngroups =3D -559038242} >> (kgdb) frame 17 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xffff= fe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> (kgdb) p vp->v_mount->mnt_stat.f_mntonname >> $8 =3D "/mnt/distfiles", '\000' > > I had some bogus -domain in my nfsuserd options on the client that I > removed after the recent panic. Not sure if it is relevant. > No that had no impact, I've hit it 3 times since sending the last email. -- Regards, Bryan Drewery --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_ Content-Type: application/octet-stream; name="opencredpanic.patch" Content-Description: opencredpanic.patch Content-Disposition: attachment; filename="opencredpanic.patch"; size=746; creation-date="Sun, 04 Jun 2017 23:19:37 GMT"; modification-date="Sun, 04 Jun 2017 23:19:37 GMT" Content-Transfer-Encoding: base64 LS0tIGZzL25mc2NsaWVudC9uZnNfY2xzdGF0ZS5jLmNyZWRwYW5pYwkyMDE3LTA2LTA0IDEwOjQ2 OjA5LjI3MzYxMzAwMCAtMDQwMAorKysgZnMvbmZzY2xpZW50L25mc19jbHN0YXRlLmMJMjAxNy0w Ni0wNCAxMTowMToyMS41MzYwOTMwMDAgLTA0MDAKQEAgLTI5MCw2ICsyOTAsMTkgQEAgbmZzY2xf b3Blbih2bm9kZV90IHZwLCB1X2ludDhfdCAqbmZocCwgaQogCSAgICBuZXdvbmVwKTsKIAogCS8q CisJICogSWYgbmZocCAhPSBOVUxMICYmIG5vcCA9PSBOVUxMLCBhIG5ldyBPcGVuIHN0cnVjdHVy ZSB3YXMgYWxsb2NhdGVkCisJICogdXNpbmcgKm5vcC4gIEZvciB0aGlzIGNhc2UsIHNldCB0aGUg Y3JlZGVudGlhbHMgaW4gdGhlIE9wZW4sIHNvCisJICogdGhhdCB0aGV5IGFyZSBuZXZlciB1bmlu aXRpYWxpemVkLgorCSAqLworCWlmIChuZmhwICE9IE5VTEwgJiYgbm9wID09IE5VTEwpIHsKKwkJ S0FTU0VSVCgqbmV3b25lcCAhPSAwLCAoIiVzOiBuZXcgb3BlbiB3YXMgYWxsb2NhdGVkXG4iLAor CQkgICAgX19mdW5jX18pKTsKKwkJS0FTU0VSVChvcCAhPSBOVUxMLCAoIiVzOiBOZXcgb3BlbiBt dXN0IGJlIHJldHVybmVkXG4iLAorCQkgICAgX19mdW5jX18pKTsKKwkJbmV3bmZzX2NvcHlpbmNy ZWQoY3JlZCwgJm9wLT5uZnNvX2NyZWQpOworCX0KKworCS8qCiAJICogTm93LCBjaGVjayB0aGUg bW9kZSBvbiB0aGUgb3BlbiBhbmQgcmV0dXJuIHRoZSBhcHByb3ByaWF0ZQogCSAqIHZhbHVlLgog CSAqLwo= --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_-- From owner-freebsd-fs@freebsd.org Mon Jun 5 19:01:47 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75399AFF5A9 for ; Mon, 5 Jun 2017 19:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6311166551 for ; Mon, 5 Jun 2017 19:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v55J1kfB019315 for ; Mon, 5 Jun 2017 19:01:47 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Mon, 05 Jun 2017 19:01:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 19:01:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jun 5 19:33:25 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 883AFB79057 for ; Mon, 5 Jun 2017 19:33:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 76D0F67337 for ; Mon, 5 Jun 2017 19:33:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v55JXPHi000353 for ; Mon, 5 Jun 2017 19:33:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Mon, 05 Jun 2017 19:33:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: henric_jungheim@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 19:33:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 --- Comment #3 from Henric Jungheim --- (In reply to Edward Tomasz Napierala from comment #2) The tank0 and tank0b drives are completely separate. Had I more sense when adding the two new drives, they would have been named something like "tankb= 0" and "tankb1". From "gpart show -l": =3D> 40 11721045088 ada3 GPT (5.5T) 40 2008 - free - (1.0M) 2048 11721041920 1 tank0b (5.5T) 11721043968 1160 - free - (580K) =3D> 40 11721045088 ada4 GPT (5.5T) 40 2008 - free - (1.0M) 2048 11721041920 1 tank1b (5.5T) 11721043968 1160 - free - (580K) =3D> 34 5860533101 da0 GPT (2.7T) 34 2014 - free - (1.0M) 2048 5860529032 1 tank0 (2.7T) 5860531080 2055 - free - (1.0M) =3D> 34 5860533101 da1 GPT (2.7T) 34 2014 - free - (1.0M) 2048 5860529032 1 tank1 (2.7T) 5860531080 2055 - free - (1.0M) Here's the gpart list for tank0 and tank1: Geom name: da0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 5860533134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da0p1 Mediasize: 3000590864384 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: 618bb5ab-66a1-11e2-8e33-00e081c81bd6 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: tank0 length: 3000590864384 offset: 1048576 type: freebsd-zfs index: 1 end: 5860531079 start: 2048 Consumers: 1. Name: da0 Mediasize: 3000592982016 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 Geom name: da1 modified: false state: OK fwheads: 255 fwsectors: 63 last: 5860533134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da1p1 Mediasize: 3000590864384 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: b0e34fe7-6a9e-11e2-8e33-00e081c81bd6 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: tank1 length: 3000590864384 offset: 1048576 type: freebsd-zfs index: 1 end: 5860531079 start: 2048 Consumers: 1. Name: da1 Mediasize: 3000592982016 (2.7T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Mon Jun 5 21:06:46 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C4F84B7A742 for ; Mon, 5 Jun 2017 21:06:46 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: from asp.reflexion.net (outbound-mail-210-5.reflexion.net [208.70.210.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 780856AE0F for ; Mon, 5 Jun 2017 21:06:45 +0000 (UTC) (envelope-from shiva.bhanujan@quorum.net) Received: (qmail 17401 invoked from network); 5 Jun 2017 21:06:44 -0000 Received: from unknown (HELO rtc-sm-01.app.dca.reflexion.local) (10.81.150.1) by 0 (rfx-qmail) with SMTP; 5 Jun 2017 21:06:44 -0000 Received: by rtc-sm-01.app.dca.reflexion.local (Reflexion email security v8.40.0) with SMTP; Mon, 05 Jun 2017 17:06:44 -0400 (EDT) Received: (qmail 24447 invoked from network); 5 Jun 2017 21:06:43 -0000 Received: from unknown (HELO mail.quorumlabs.com) (64.74.133.216) by 0 (rfx-qmail) with (AES128-SHA encrypted) SMTP; 5 Jun 2017 21:06:43 -0000 Received: from QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e]) by QLEXC01.Quorum.local ([fe80::edb6:63d5:778f:2f0e%14]) with mapi id 14.02.0318.001; Mon, 5 Jun 2017 14:06:42 -0700 From: Shiva Bhanujan To: Ronald Klop , =?iso-8859-1?Q?Karli_Sj=F6berg?= , Gary Palmer CC: "freebsd-fs@freebsd.org" , Jeremy Faulkner , Shiva Bhanujan Subject: RE: FreeBSD restartable send/receive over WAN Thread-Topic: FreeBSD restartable send/receive over WAN Thread-Index: AQHSkE8fXkuYo0TvTyGgnDeYS4b6VKF+hfcfgADVzwCAAu69hYAEWqgAgJC1a0A= Date: Mon, 5 Jun 2017 21:06:41 +0000 Message-ID: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.20.7.74] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 21:06:46 -0000 Hi Ronald,=0A= =0A= Ideally, I'd like to send/receive the compressed blocks, and don't wish to = compress/uncompress on the fly. Is compressed send/receive available on = FreeBSD and if so, in what version?=0A= =0A= Regards,=0A= Shiva=0A= =0A= =0A= ________________________________________=0A= From: Ronald Klop [ronald-lists@klop.ws]=0A= Sent: Sunday, March 05, 2017 3:01 AM=0A= To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= Subject: Re: FreeBSD restartable send/receive over WAN=0A= =0A= On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan=0A= wrote:=0A= =0A= > I ran the same set of tests between 2 FreeBSD instances, connected on a= =0A= > 1G LAN. The the comms was setup using mbuffer. Here's the basic=0A= > command.=0A= >=0A= > time zfs send -v | | mbuffer -O :8099 -b 1024= =0A= > -m 128M -P 10 -q -l /tmp/mbuffer.log=0A= > mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | = =0A= > | zfs recv =0A= >=0A= > Here are the results that I got.=0A= >=0A= > no compression:=0A= > real 3m18.591s=0A= > user 0m0.390s=0A= > sys 0m8.177s=0A= >=0A= > xz -0:=0A= > real 7m26.349s=0A= > user 7m6.436s=0A= > sys 0m8.471s=0A= >=0A= > pxz -0:=0A= > real 2m28.470s=0A= > user 6m44.168s=0A= > sys 0m12.002s=0A= >=0A= > gzip:=0A= > real 3m12.482s=0A= > user 3m8.260s=0A= > sys 0m4.974s=0A= >=0A= > lz4:=0A= > real 1m58.363s=0A= > user 0m10.000s=0A= > sys 0m8.708s=0A= >=0A= >=0A= >=0A= > lz4 still comes out best. Is there anything else that I might be=0A= > missing in my tests? don't have a real setup at this time to test WAN=0A= > connections, but I'd like to think that these results can be=0A= > extrapolated to a WAN link also.=0A= =0A= =0A= Uhm, in your previous test no-compression came out best. I thought you=0A= wanted to test if sending on-disk compressed blocks over a network was a=0A= gain. Not a plain test of compression algoritms.=0A= =0A= As your current numbers show: the overhead of mbuffer.log is much higher=0A= than the overhead of compression+decompression for the lz4 case. real:=0A= 0m29 to 1m58. (All other compressors are actually faster with mbuffer in=0A= between? Probably because of more efficient buffering by mbuffer.)=0A= As the overhead of mbuffer is large it is a question how much gain you get= =0A= from removing the extra compression step by sending the on-disk compressed= =0A= blocks.=0A= I guess the time would go from 1m58 to about 1m30.=0A= =0A= NB: By splitting your zfs send/receive command in two, the numbers of time= =0A= can be affected. Time prints the output as soon as zfs send ends.=0A= =0A= Have fun,=0A= Ronald.=0A= =0A= =0A= =0A= > ________________________________________=0A= > From: Ronald Klop [ronald-lists@klop.ws]=0A= > Sent: Tuesday, February 28, 2017 11:44 AM=0A= > To: Karli Sj=F6berg; Gary Palmer; Shiva Bhanujan=0A= > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner=0A= > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >=0A= > On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan=0A= > wrote:=0A= >=0A= >> thanks for all the pointers for the compression algorithms. I ran a few= =0A= >> tests to compare compression overhead. These are local zfs=0A= >> send/receive, and no network is involved.=0A= >>=0A= >> time zfs send -v | | | zfs=0A= >> receive -s =0A= >>=0A= >> Here are the performance results that I got.=0A= >>=0A= >> no compression:=0A= >> real 0m20.892s=0A= >> user 0m0.000s=0A= >> sys 0m5.587s=0A= >>=0A= >> xz -0:=0A= >> real 8m38.569s=0A= >> user 10m28.551s=0A= >> sys 0m6.866s=0A= >>=0A= >> pxz -0:=0A= >> real 4m38.448s=0A= >> user 10m55.863s=0A= >> sys 0m13.324s=0A= >>=0A= >> gzip:=0A= >> real 3m51.297s=0A= >> user 4m12.035s=0A= >> sys 0m4.696s=0A= >>=0A= >> lz4:=0A= >> real 0m29.912s=0A= >> user 0m16.543s=0A= >> sys 0m10.810s=0A= >>=0A= >>=0A= >> lz4 has the least overhead in terms of time. pxz/xz seem to be=0A= >> prohibitive give the above results. Unless, there is something basic=0A= >> I'm missing?=0A= >>=0A= >> I was really hoping that compressed sends would be available, as that=0A= >> would actively eliminate this overhead, given that we use lz4 as the=0A= >> compression algorithm when writing to disks.=0A= >=0A= >=0A= > Why don't you test this with a network in between? That would give much= =0A= > more valuable numbers to compare for your use case.=0A= > The number above say nothing about the gain vs the bottleneck you are=0A= > trying to remove.=0A= >=0A= > Ronald.=0A= >=0A= >=0A= >=0A= >=0A= >>=0A= >>=0A= >> ________________________________=0A= >> From: Karli Sj=F6berg [karli.sjoberg@slu.se]=0A= >> Sent: Sunday, February 26, 2017 8:41 AM=0A= >> To: Gary Palmer=0A= >> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org=0A= >> Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>=0A= >>=0A= >> Den 26 feb. 2017 4:16 em skrev Gary Palmer :=0A= >>>=0A= >>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote:=0A= >>> > The compression that we use on our ZFS filesystems is lz4. So, if I= =0A= >>> have to pipe it through a compression algorithm, that'd be=0A= >>> uncompressing and compressing it 4 times.=0A= >>> >=0A= >>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (network)= =0A= >>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4)=0A= >>> >=0A= >>> > isn't this quite expensive? We have to transfer multi terabyte files= =0A= >>> on a WAN link. I'm also of the understanding that gzip by itself is=0A= >>> single-threaded, so that'd peg one of the CPUs to 100%. there might be= =0A= >>> other compression algorithms that can be used, but sending the ZFS as= =0A= >>> it is compressed on the filesystem is something that would be optimal,= =0A= >>> and would reduce the overhead of the additional [de]compressions that= =0A= >>> are taking place?=0A= >>>=0A= >>> Without going into the efficiency part of your message:=0A= >>>=0A= >>> archivers/pigz: Parallel GZIP=0A= >>> archivers/pbzip2: Parallel BZIP2=0A= >>> archivers/pixz: Parallel, indexing version of XZ=0A= >>> archivers/pxz: Parallel LZMA compressor using liblzma=0A= >>=0A= >> Also worth mentioning is, obviously:=0A= >> archivers/lz4=0A= >>=0A= >> :)=0A= >>=0A= >> /K=0A= >>=0A= >>>=0A= >>> Regards,=0A= >>>=0A= >>> Gary=0A= >>>=0A= >>> >=0A= >>> > ________________________________________=0A= >>> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org] on= =0A= >>> behalf of Jeremy Faulkner [gldisater@gmail.com]=0A= >>> > Sent: Saturday, February 25, 2017 4:03 PM=0A= >>> > To: freebsd-fs@freebsd.org=0A= >>> > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> >=0A= >>> > Pipe it through a compressor=0A= >>> >=0A= >>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote:=0A= >>> > > Hi,=0A= >>> > >=0A= >>> > > I just tried restartable send/receive in 10.3 and it works like a= =0A= >>> charm. I was wondering if compressed send has made its way into=0A= >>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the=0A= >>> -c/--compressed option. Any pointers?=0A= >>> > >=0A= >>> > > Regards,=0A= >>> > > Shiva=0A= >>> > >=0A= >>> > >=0A= >>> > > ________________________________________=0A= >>> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= =0A= >>> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl]=0A= >>> > > Sent: Thursday, February 16, 2017 10:41 AM=0A= >>> > > To: freebsd-fs@freebsd.org=0A= >>> > > Subject: Re: FreeBSD restartable send/receive over WAN=0A= >>> > >=0A= >>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote:=0A= >>> > >> Hello,=0A= >>> > >>=0A= >>> > >> I was wondering if restartable send/receive is available in=0A= >>> FreeBSD? We're running 10.2 and have a requirement of sending and=0A= >>> receiving ZFS snapshots over a WAN link. The snapshots could be more= =0A= >>> than a few terabytes.=0A= >>> > >>=0A= >>> > >> Can somebody please give me pointers, and if this feature is or=0A= >>> isn't available in FreeBSD?=0A= >>> > >=0A= >>> > > FreeBSD 10.3 and later.=0A= >>> > >=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > > _______________________________________________=0A= >>> > > freebsd-fs@freebsd.org mailing list=0A= >>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > > To unsubscribe, send any mail to=0A= >>> "freebsd-fs-unsubscribe@freebsd.org"=0A= >>> > >=0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> > _______________________________________________=0A= >>> > freebsd-fs@freebsd.org mailing list=0A= >>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >>> >=0A= >>> _______________________________________________=0A= >>> freebsd-fs@freebsd.org mailing list=0A= >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"= =0A= >> _______________________________________________=0A= >> freebsd-fs@freebsd.org mailing list=0A= >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A= >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A= From owner-freebsd-fs@freebsd.org Mon Jun 5 21:33:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3C7CB7AD55 for ; Mon, 5 Jun 2017 21:33:07 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 650E26EB6C; Mon, 5 Jun 2017 21:33:07 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1dHzcU-0001H1-72; Mon, 05 Jun 2017 23:33:03 +0200 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: =?utf-8?Q?Karli_Sj=C3=B6berg?= , "Gary Palmer" , "Shiva Bhanujan" Cc: "freebsd-fs@freebsd.org" , "Jeremy Faulkner" Subject: Re: FreeBSD restartable send/receive over WAN References: <0719669324a44fe0bfba3e8e08b0ae99@exch2-4.slu.se> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12619@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701BDB12ABA@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> Date: Mon, 05 Jun 2017 23:33:00 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: Quoted-Printable From: "Ronald Klop" Message-ID: In-Reply-To: <3A5A10BE32AC9E45B4A22F89FC90EC0701C35D8A68@QLEXC01.Quorum.local> User-Agent: Opera Mail/12.16 (FreeBSD) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: -0.2 X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED, BAYES_50 autolearn=disabled version=3.4.0 X-Scan-Signature: d91c552e3f94bdb2425468daff346539 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jun 2017 21:33:07 -0000 Hi, I'm not authoritative about this, but I see the feature in FreeBSD 12: https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&= manpath=3DFreeBSD+12-current&arch=3Ddefault&format=3Dhtml (option -c to 'zfs send') But not in FreeBSD 11. https://www.freebsd.org/cgi/man.cgi?query=3Dzfs&apropos=3D0&sektion=3D8&= manpath=3DFreeBSD+11-current&arch=3Ddefault&format=3Dhtml Regards, Ronald. On Mon, 05 Jun 2017 23:06:41 +0200, Shiva Bhanujan = wrote: > Hi Ronald, > > Ideally, I'd like to send/receive the compressed blocks, and don't wis= h = > to compress/uncompress on the fly. Is compressed send/receive = > available on FreeBSD and if so, in what version? > > Regards, > Shiva > > > ________________________________________ > From: Ronald Klop [ronald-lists@klop.ws] > Sent: Sunday, March 05, 2017 3:01 AM > To: Karli Sj=C3=B6berg; Gary Palmer; Shiva Bhanujan > Cc: freebsd-fs@freebsd.org; Jeremy Faulkner > Subject: Re: FreeBSD restartable send/receive over WAN > > On Fri, 03 Mar 2017 01:34:57 +0100, Shiva Bhanujan > wrote: > >> I ran the same set of tests between 2 FreeBSD instances, connected on= a >> 1G LAN. The the comms was setup using mbuffer. Here's the basic >> command. >> >> time zfs send -v | | mbuffer -O :8099 -b 10= 24 >> -m 128M -P 10 -q -l /tmp/mbuffer.log >> mbuffer -4 -I 8099 -b 1024 -m 128M -q -l /tmp/mbuffer.log | >> | zfs recv >> >> Here are the results that I got. >> >> no compression: >> real 3m18.591s >> user 0m0.390s >> sys 0m8.177s >> >> xz -0: >> real 7m26.349s >> user 7m6.436s >> sys 0m8.471s >> >> pxz -0: >> real 2m28.470s >> user 6m44.168s >> sys 0m12.002s >> >> gzip: >> real 3m12.482s >> user 3m8.260s >> sys 0m4.974s >> >> lz4: >> real 1m58.363s >> user 0m10.000s >> sys 0m8.708s >> >> >> >> lz4 still comes out best. Is there anything else that I might be >> missing in my tests? don't have a real setup at this time to test WA= N >> connections, but I'd like to think that these results can be >> extrapolated to a WAN link also. > > > Uhm, in your previous test no-compression came out best. I thought you= > wanted to test if sending on-disk compressed blocks over a network was= a > gain. Not a plain test of compression algoritms. > > As your current numbers show: the overhead of mbuffer.log is much high= er > than the overhead of compression+decompression for the lz4 case. real:= > 0m29 to 1m58. (All other compressors are actually faster with mbuffer = in > between? Probably because of more efficient buffering by mbuffer.) > As the overhead of mbuffer is large it is a question how much gain you= = > get > from removing the extra compression step by sending the on-disk = > compressed > blocks. > I guess the time would go from 1m58 to about 1m30. > > NB: By splitting your zfs send/receive command in two, the numbers of = = > time > can be affected. Time prints the output as soon as zfs send ends. > > Have fun, > Ronald. > > > >> ________________________________________ >> From: Ronald Klop [ronald-lists@klop.ws] >> Sent: Tuesday, February 28, 2017 11:44 AM >> To: Karli Sj=C3=B6berg; Gary Palmer; Shiva Bhanujan >> Cc: freebsd-fs@freebsd.org; Jeremy Faulkner >> Subject: Re: FreeBSD restartable send/receive over WAN >> >> On Tue, 28 Feb 2017 16:04:16 +0100, Shiva Bhanujan >> wrote: >> >>> thanks for all the pointers for the compression algorithms. I ran a= = >>> few >>> tests to compare compression overhead. These are local zfs >>> send/receive, and no network is involved. >>> >>> time zfs send -v | | | zfs >>> receive -s >>> >>> Here are the performance results that I got. >>> >>> no compression: >>> real 0m20.892s >>> user 0m0.000s >>> sys 0m5.587s >>> >>> xz -0: >>> real 8m38.569s >>> user 10m28.551s >>> sys 0m6.866s >>> >>> pxz -0: >>> real 4m38.448s >>> user 10m55.863s >>> sys 0m13.324s >>> >>> gzip: >>> real 3m51.297s >>> user 4m12.035s >>> sys 0m4.696s >>> >>> lz4: >>> real 0m29.912s >>> user 0m16.543s >>> sys 0m10.810s >>> >>> >>> lz4 has the least overhead in terms of time. pxz/xz seem to be >>> prohibitive give the above results. Unless, there is something basi= c >>> I'm missing? >>> >>> I was really hoping that compressed sends would be available, as tha= t >>> would actively eliminate this overhead, given that we use lz4 as the= >>> compression algorithm when writing to disks. >> >> >> Why don't you test this with a network in between? That would give mu= ch >> more valuable numbers to compare for your use case. >> The number above say nothing about the gain vs the bottleneck you are= >> trying to remove. >> >> Ronald. >> >> >> >> >>> >>> >>> ________________________________ >>> From: Karli Sj=C3=B6berg [karli.sjoberg@slu.se] >>> Sent: Sunday, February 26, 2017 8:41 AM >>> To: Gary Palmer >>> Cc: Shiva Bhanujan; Jeremy Faulkner; freebsd-fs@freebsd.org >>> Subject: Re: FreeBSD restartable send/receive over WAN >>> >>> >>> Den 26 feb. 2017 4:16 em skrev Gary Palmer : >>>> >>>> On Sun, Feb 26, 2017 at 02:08:59PM +0000, Shiva Bhanujan wrote: >>>> > The compression that we use on our ZFS filesystems is lz4. So, i= f I >>>> have to pipe it through a compression algorithm, that'd be >>>> uncompressing and compressing it 4 times. >>>> > >>>> > disk (lz4) -> zfs send (uncompress) -> compress (gzip) -> (networ= k) >>>> -> uncompress (gzip) -> zfs recv (compress) -> disk (lz4) >>>> > >>>> > isn't this quite expensive? We have to transfer multi terabyte = >>>> files >>>> on a WAN link. I'm also of the understanding that gzip by itself i= s >>>> single-threaded, so that'd peg one of the CPUs to 100%. there migh= t = >>>> be >>>> other compression algorithms that can be used, but sending the ZFS = as >>>> it is compressed on the filesystem is something that would be optim= al, >>>> and would reduce the overhead of the additional [de]compressions th= at >>>> are taking place? >>>> >>>> Without going into the efficiency part of your message: >>>> >>>> archivers/pigz: Parallel GZIP >>>> archivers/pbzip2: Parallel BZIP2 >>>> archivers/pixz: Parallel, indexing version of XZ >>>> archivers/pxz: Parallel LZMA compressor using liblzma >>> >>> Also worth mentioning is, obviously: >>> archivers/lz4 >>> >>> :) >>> >>> /K >>> >>>> >>>> Regards, >>>> >>>> Gary >>>> >>>> > >>>> > ________________________________________ >>>> > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.org]= on >>>> behalf of Jeremy Faulkner [gldisater@gmail.com] >>>> > Sent: Saturday, February 25, 2017 4:03 PM >>>> > To: freebsd-fs@freebsd.org >>>> > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > >>>> > Pipe it through a compressor >>>> > >>>> > On 2017-02-25 2:09 PM, Shiva Bhanujan wrote: >>>> > > Hi, >>>> > > >>>> > > I just tried restartable send/receive in 10.3 and it works like= a >>>> charm. I was wondering if compressed send has made its way into >>>> FreeBSD? I checked 10.3 and 11.0-RELEASE, and I don't see the >>>> -c/--compressed option. Any pointers? >>>> > > >>>> > > Regards, >>>> > > Shiva >>>> > > >>>> > > >>>> > > ________________________________________ >>>> > > From: owner-freebsd-fs@freebsd.org [owner-freebsd-fs@freebsd.or= g] >>>> on behalf of Adam Nowacki [nowakpl@platinum.linux.pl] >>>> > > Sent: Thursday, February 16, 2017 10:41 AM >>>> > > To: freebsd-fs@freebsd.org >>>> > > Subject: Re: FreeBSD restartable send/receive over WAN >>>> > > >>>> > > On 2017-02-16 19:22, Shiva Bhanujan wrote: >>>> > >> Hello, >>>> > >> >>>> > >> I was wondering if restartable send/receive is available in >>>> FreeBSD? We're running 10.2 and have a requirement of sending and >>>> receiving ZFS snapshots over a WAN link. The snapshots could be mo= re >>>> than a few terabytes. >>>> > >> >>>> > >> Can somebody please give me pointers, and if this feature is o= r >>>> isn't available in FreeBSD? >>>> > > >>>> > > FreeBSD 10.3 and later. >>>> > > >>>> > > _______________________________________________ >>>> > > freebsd-fs@freebsd.org mailing list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> "freebsd-fs-unsubscribe@freebsd.org" >>>> > > _______________________________________________ >>>> > > freebsd-fs@freebsd.org mailing list >>>> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > > To unsubscribe, send any mail to >>>> "freebsd-fs-unsubscribe@freebsd.org" >>>> > > >>>> > _______________________________________________ >>>> > freebsd-fs@freebsd.org mailing list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to = >>>> "freebsd-fs-unsubscribe@freebsd.org" >>>> > _______________________________________________ >>>> > freebsd-fs@freebsd.org mailing list >>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> > To unsubscribe, send any mail to = >>>> "freebsd-fs-unsubscribe@freebsd.org" >>>> > >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.or= g" >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org= " From owner-freebsd-fs@freebsd.org Tue Jun 6 00:27:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE0D1B7D976 for ; Tue, 6 Jun 2017 00:27:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A15677464C for ; Tue, 6 Jun 2017 00:27:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v560R1YE070525 for ; Tue, 6 Jun 2017 00:27:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 00:27:01 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: g_amanakis@yahoo.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 00:27:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 --- Comment #3 from g_amanakis@yahoo.com --- Is it possible to MFC this one (MFV r318942) to 11.1-RELEASE? Thank you, George --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 14:20:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C55ABD3B98 for ; Tue, 6 Jun 2017 14:20:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5360B6E344 for ; Tue, 6 Jun 2017 14:20:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56EKocW008608 for ; Tue, 6 Jun 2017 14:20:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 14:20:50 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jpaetzel@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jpaetzel@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 14:20:51 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 Josh Paetzel changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |In Progress Assignee|freebsd-fs@FreeBSD.org |jpaetzel@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 14:25:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DD73BD3D7A for ; Tue, 6 Jun 2017 14:25:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 61A306E67F for ; Tue, 6 Jun 2017 14:25:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56EPZab022620 for ; Tue, 6 Jun 2017 14:25:35 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 14:25:35 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jpaetzel@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: gjb@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 14:25:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 Josh Paetzel changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jpaetzel@FreeBSD.org |gjb@FreeBSD.org --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 14:47:15 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AD97BEE46F for ; Tue, 6 Jun 2017 14:47:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 692F26F30A for ; Tue, 6 Jun 2017 14:47:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56ElFQm074722 for ; Tue, 6 Jun 2017 14:47:15 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 14:47:15 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: gjb@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 14:47:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 --- Comment #4 from commit-hook@freebsd.org --- A commit references this bug: Author: gjb Date: Tue Jun 6 14:46:23 UTC 2017 New revision: 319624 URL: https://svnweb.freebsd.org/changeset/base/319624 Log: MFC r318943 (avg): MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also https://github.com/zfsonlinux/zfs/issues/5806 PR: 219537 Approved by: re (kib) Sponsored by: The FreeBSD Foundation Changes: _U stable/11/ stable/11/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 14:47:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2AADBEE46B for ; Tue, 6 Jun 2017 14:47:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B0A936F307 for ; Tue, 6 Jun 2017 14:47:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56ElD9L074689 for ; Tue, 6 Jun 2017 14:47:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 14:47:13 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: gjb@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: gjb@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 14:47:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 Glen Barber changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|In Progress |Closed --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 14:47:17 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D60B8BEE47B for ; Tue, 6 Jun 2017 14:47:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C4A336F324 for ; Tue, 6 Jun 2017 14:47:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56ElHAG074799 for ; Tue, 6 Jun 2017 14:47:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219537] OpenZFS 8166 - zpool scrub thinks it repaired offline device Date: Tue, 06 Jun 2017 14:47:18 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: gjb@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 14:47:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219537 --- Comment #5 from commit-hook@freebsd.org --- A commit references this bug: Author: gjb Date: Tue Jun 6 14:46:46 UTC 2017 New revision: 319625 URL: https://svnweb.freebsd.org/changeset/base/319625 Log: MFC r318943 (avg): MFV r318942: 8166 zpool scrub thinks it repaired offline device https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also https://github.com/zfsonlinux/zfs/issues/5806 PR: 219537 Sponsored by: The FreeBSD Foundation Changes: _U stable/10/ stable/10/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Tue Jun 6 17:18:05 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DF18BF1313 for ; Tue, 6 Jun 2017 17:18:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C21C743F2 for ; Tue, 6 Jun 2017 17:18:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v56HI5p1030090 for ; Tue, 6 Jun 2017 17:18:05 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Tue, 06 Jun 2017 17:18:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jun 2017 17:18:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 --- Comment #4 from Edward Tomasz Napierala --- If I'm honest I doubt it's caused by iSCSI; I have no idea how it could cor= rupt a filesystem _below_ it. I think it's something between your disks and ZFS, and iSCSI just manages to generate a load high enough to trigger it. Is there a firmware update for your disks? Also, the errors seem to appear only on ada3 and ada4; can you recreate the zpool without those two disks a= nd see if the problem goes away? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Jun 7 08:26:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5948BFE72B for ; Wed, 7 Jun 2017 08:26:54 +0000 (UTC) (envelope-from mckay@freebsd.org) Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by mx1.freebsd.org (Postfix) with ESMTP id 35DD266C8F; Wed, 7 Jun 2017 08:26:53 +0000 (UTC) (envelope-from mckay@freebsd.org) Message-Id: <0410af$1dldvp4@ipmail04.adl6.internode.on.net> X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2A1DwAXtzdZ/2O8Ag5eHAEBBAEBCgEBgmpBLVIQgQ2DJYtZlCCFCI9mLIVyBAICgmtYAQIBAQEBAQJrKIUZBicvIxAIA0YhGB4GE4oTAxQRr3M6h0ANhDIBAQEHjAmCWCmBL4Negk8FiUWGbYFdhHeGdzuHJoc1hmOJLIZNi0GJJFeBCoEBCEaFBxyBdi42iVIBAgM Received: from ppp14-2-188-99.bras1.hba2.internode.on.net (HELO localhost) ([14.2.188.99]) by ipmail04.adl6.internode.on.net with ESMTP; 07 Jun 2017 17:51:11 +0930 From: Stephen McKay To: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= cc: freebsd-fs@freebsd.org, mckay@FreeBSD.org Subject: Re: Problem with zpool remove of log device References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> In-Reply-To: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> from =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= at "Fri, 26 May 2017 11:40:06 +0200" Date: Wed, 07 Jun 2017 18:21:09 +1000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 08:26:54 -0000 On Friday, 26th May 2017, lukasz@wasikowski.net wrote: >I cant remove log device from pool - operation ends ok, but log device >is still in the pool (bug?). > ># uname -a >FreeBSD xxx.yyy.com 11.0-STABLE FreeBSD 11.0-STABLE #0 r316543: Thu Apr >6 08:22:43 CEST 2017 root@xxx.yyy.com:/usr/obj/usr/src/sys/YYY amd64 > ># zpool status tank >[..snip..] > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ada2p3 ONLINE 0 0 0 > ada3p3 ONLINE 0 0 0 > logs > mirror-1 ONLINE 0 0 0 > gpt/tankssdzil0 ONLINE 0 0 0 block size: 512B configured, 4096B native > gpt/tankssdzil1 ONLINE 0 0 0 block size: 512B configured, 4096B native >When I try to remove log device operation ends without errors: > ># zpool remove tank mirror-1; echo $? >0 > >But the log device is still there: >[..snip..] >I'd like to remove it - how should I proceed? Does your system still write to the log? Use "zfs iostat -v 1" to check. I think it is probably no longer be in use and only the final disconnection failed. What does "zpool list -v" tell you? If you have a non-zero ALLOC column for your log mirror and the log is no longer being used then you may have hit an accounting bug in zfs that the zfsonlinux people ran into a while ago. I had this problem when I tried to remove a log mirror from a pool I have been using for years. I solved it by tweaking the zfsonlinux hack a bit and slotting it into 9.3. If you apply this hack be sure to have a full backup first! When I used it, I did my backup and a scrub then booted the hacked kernel, issued the zfs remove command (which succeeded), reverted the kernel, then scrubbed again. All went well. Good luck! Here's the patch against 9.3 (should be close even for 11.0): Index: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c =================================================================== --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (revision 309860) +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c (working copy) @@ -5446,6 +5446,18 @@ ASSERT(vd == vd->vdev_top); /* + * slog stuck hack - barnes333@gmail.com + * https://github.com/zfsonlinux/zfs/issues/1422 + */ + if (vd->vdev_islog && vd->vdev_removing + && vd->vdev_state == VDEV_STATE_OFFLINE + && vd->vdev_stat.vs_alloc > 0) { + printf("ZFS: slog stuck hack - clearing vs_alloc: %llu\n", + (unsigned long long)vd->vdev_stat.vs_alloc); + vd->vdev_stat.vs_alloc = 0; + } + + /* * Only remove any devices which are empty. */ if (vd->vdev_stat.vs_alloc != 0) Cheers, Stephen. From owner-freebsd-fs@freebsd.org Wed Jun 7 10:56:35 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 943C7C0975B for ; Wed, 7 Jun 2017 10:56:35 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:2:1276::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44A3D6F3DD; Wed, 7 Jun 2017 10:56:35 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:2:1276::1]) by mail.freebsd.systems (Postfix) with ESMTP id 7ABA777D; Wed, 7 Jun 2017 12:56:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at freebsd.systems Received: from mail.freebsd.systems ([5.196.167.1]) by mail.freebsd.systems (scan.freebsd.systems [5.196.167.1]) (amavisd-new, port 10026) with ESMTP id stNc5iQOyRU0; Wed, 7 Jun 2017 12:56:33 +0200 (CEST) Received: from [192.168.138.100] (unknown [194.181.68.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.freebsd.systems (Postfix) with ESMTPSA id 7BBB777A; Wed, 7 Jun 2017 12:56:32 +0200 (CEST) Authentication-Results: mail.freebsd.systems; dmarc=none header.from=wasikowski.net Authentication-Results: mail.freebsd.systems; spf=pass smtp.mailfrom=lukasz@wasikowski.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wasikowski.net; s=default; t=1496832993; bh=S+OV9nAvOn+WbBQ6p4XbxgpNbjVj4gF2Ap0ZCuBkba0=; h=To:Cc:References:From:Date:In-Reply-To; b=HY7p2KoxgTX2TNsAm0eQZqzdU8RHxhVjkWevZaX9ePRVF5jHbqffkbm1ZjxTgQKup OqwJwrZS2GT63xcn7+rO8NsAKsufrCJcia6IsMjyXeZiT2nU+c52j2j9poZ/etuwuY vDIf33bVBmoDFkOOT49e3Ll/G4x39T1uzwBfQkcTX5lmO4Bb4CmBZfY4HAEDQiWYzv PbsOZodJNSQNdQcFHKnfvM9N00fG+cNRrf1HTpop7nvw9bxM1R3BdbfR8JSfdp/qwv fzZZfnnO8VuHJxD+T4BNs/0w40fUOHjSpqcGmhvrFFisCSdux3ny/7RnCwh11iXf+k XPOCtfE4P8vWA== Subject: Re: Problem with zpool remove of log device To: Stephen McKay Cc: freebsd-fs@freebsd.org References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> From: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= Message-ID: <4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> Date: Wed, 7 Jun 2017 12:56:30 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <0410af$1dldvp4@ipmail04.adl6.internode.on.net> Content-Type: text/plain; charset=utf-8 Content-Language: pl Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jun 2017 10:56:35 -0000 W dniu 2017-06-07 o 10:21, Stephen McKay pisze: > On Friday, 26th May 2017, lukasz@wasikowski.net wrote: > >> I cant remove log device from pool - operation ends ok, but log device >> is still in the pool (bug?). >> >> # uname -a >> FreeBSD xxx.yyy.com 11.0-STABLE FreeBSD 11.0-STABLE #0 r316543: Thu Apr >> 6 08:22:43 CEST 2017 root@xxx.yyy.com:/usr/obj/usr/src/sys/YYY amd64 >> >> # zpool status tank >> [..snip..] >> >> NAME STATE READ WRITE CKSUM >> tank ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> ada2p3 ONLINE 0 0 0 >> ada3p3 ONLINE 0 0 0 >> logs >> mirror-1 ONLINE 0 0 0 >> gpt/tankssdzil0 ONLINE 0 0 0 block size: 512B configured, 4096B native >> gpt/tankssdzil1 ONLINE 0 0 0 block size: 512B configured, 4096B native > >> When I try to remove log device operation ends without errors: >> >> # zpool remove tank mirror-1; echo $? >> 0 >> >> But the log device is still there: >> [..snip..] >> I'd like to remove it - how should I proceed? > > Does your system still write to the log? Use "zfs iostat -v 1" to > check. I think it is probably no longer be in use and only the final > disconnection failed. zpool iostat -v 1 shows no activity on the log drive. I tried to remove it after booting to livecd - no luck. > What does "zpool list -v" tell you? If you have a non-zero ALLOC > column for your log mirror and the log is no longer being used then > you may have hit an accounting bug in zfs that the zfsonlinux people > ran into a while ago. This can be it: # zpool list -v NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 1.80T 1.55T 251G - 64% 86% 1.00x ONLINE - mirror 1.80T 1.55T 251G - 64% 86% ada2p3 - - - - - - ada3p3 - - - - - - log - - - - - - gpt/tankssdzil1 4.97G 142M 4.83G - 0% 2% > I had this problem when I tried to remove a log mirror from a pool > I have been using for years. I solved it by tweaking the zfsonlinux > hack a bit and slotting it into 9.3. Thank you for this info. It seems that it's no longer an issue for me - this box will be soon retired. I'll move some data from it and wipe it down. -- best regards, Lukasz Wasikowski From owner-freebsd-fs@freebsd.org Thu Jun 8 21:13:04 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D5DEC79C48 for ; Thu, 8 Jun 2017 21:13:04 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9200B71CA2 for ; Thu, 8 Jun 2017 21:13:03 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x233.google.com with SMTP id o83so22807124lff.3 for ; Thu, 08 Jun 2017 14:13:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:from:date:message-id:subject:to; bh=tcQ5wfEH+pqQ44wz/QszwO/cgsEyGlgUHjSgBsxxd4o=; b=jsOQjrhPonyMRru6zWDuKvLQXIvuEsL9Nu+9E4iMaAY+iH2J4iup1XQU3J0SW/y8rf LfBB1rM5bM11mAdz199N3BdIgH3209yqhqgtv1dwA4q4+kSCFdllhsVCGO5/VSdeYAw0 ZuDVhbUDxtu5xBOXsCEFXzif1silRekkpAFRs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=tcQ5wfEH+pqQ44wz/QszwO/cgsEyGlgUHjSgBsxxd4o=; b=eX4FHE/AmC6trgEDdR7WXCyK3cUS6oWgGoiATmY2548PfEOqvTi3TXyqb1uY7wYmSR OvxNNRAyQ22HPFYyBqGnn1iJZmasMsBx7Kwen8cxk/NnGSGgUCQDAWw+tVhtDk3GeIto 3NYrNZGelKSqIUwrO6bSN2Z45+bv9owZKNXFggwLDcIuoR8AX297o1x4SJ3vN/XydZBm ap52X2o2Gknd8P9xiv2ejF5syJAVkwkYH7ksF6B6ZvwmPjSo6QCPBpQJVdH5tDYzm06U BQ4LxE5Jd4oSfSDn9BC0Ad6Nvve5hXofOKmQJliOzQqz6JUiB5PoOmH8gwZU5lF9XAjD kUYw== X-Gm-Message-State: AODbwcC3ZFJdrZi5HtjShhnKIl//8nXI/G71eTSyfZfK3uvbssE2dQE0 VLf3ereaWQAXFo1u0gCiqS7s0MRyIHfDz09iLQ== X-Received: by 10.25.80.71 with SMTP id z7mr11494711lfj.16.1496956381440; Thu, 08 Jun 2017 14:13:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Thu, 8 Jun 2017 14:13:01 -0700 (PDT) From: Tim Gustafson Date: Thu, 8 Jun 2017 14:13:01 -0700 Message-ID: Subject: ZFS Commands In "D" State To: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 21:13:04 -0000 We have a ZFS server that we've been running for a few months now. The server is a backup server that receives ZFS sends from its primary daily. This mechanism has been working for us on several pairs of servers for years in general, and for several months with this particular piece of hardware. A few days ago, our nightly ZFS send failed. When I looked at the server, I saw that the "zfs receive" command was in a "D" wait state: 1425 - D 0:02.75 /sbin/zfs receive -v -F backup/export I rebooted the system, checked that "zpool status" and "zfs list" both came back correctly (which they did) and then re-started the "zfs send" on the master server. At first, the "zfs receive" command did not enter the "D" state, but once the master server started sending actual data (which I was able to ascertain because I was doing "zfs send" with the -v option), the receiving process entered the "D" state again, and another reboot was required. Only about 2MB of data got sent before this happened. I've rebooted several times, always with the same result. I did a "zpool scrub os" (there's a separate zpool for the OS to live on) and that completed in a few minutes, but when I did a "zpool scrub backup", that process immediately went into the "D+" state: 895 0 D+ 0:00.04 zpool scrub backup We run smartd on this device, and that is showing no disk errors. The devd process is logging some stuff, but it doesn't appear to be very helpful: Jun 8 13:52:49 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=11754027336427262018 Jun 8 13:52:49 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=11367786800631979308 Jun 8 13:52:49 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=18407069648425063426 Jun 8 13:52:49 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=9496839124651172990 Jun 8 13:52:49 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=332784898986906736 Jun 8 13:52:50 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=16384086680948393578 Jun 8 13:52:50 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=10762348983543761591 Jun 8 13:52:50 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=8585274278710252761 Jun 8 13:52:50 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=17456777842286400332 Jun 8 13:52:50 backup ZFS: vdev state changed, pool_guid=2176924632732322522 vdev_guid=10533897485373019500 No word on which state it changed "from" or "to". Also, the system only has three vdevs (the OS one, and then two raidz2 vdevs that make up the "backup" pool, so I'm not sure how it's coming up with more than 3 vdev GUIDs). What's my next step in diagnosing this? -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Thu Jun 8 21:53:03 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFF44D84AF4 for ; Thu, 8 Jun 2017 21:53:03 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from mail.inparadise.se (h-246-50.A444.priv.bahnhof.se [155.4.246.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 655B5735CE for ; Thu, 8 Jun 2017 21:53:02 +0000 (UTC) (envelope-from karli@inparadise.se) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id C1B814794C; Thu, 8 Jun 2017 23:44:39 +0200 (CEST) Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id hhbfv6iE8cdT; Thu, 8 Jun 2017 23:44:38 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id 802724794D; Thu, 8 Jun 2017 23:44:38 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.inparadise.se 802724794D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inparadise.se; s=ECF0F226-2F14-11E7-BBE9-ECFEB9BC1D67; t=1496958278; bh=WvVEWM1aJi3r7ZvbDz5xlpbkeO5WDGxJz6K6YEROHOc=; h=Date:Message-ID:To:From:MIME-Version; b=MiQaSAU+TvlaRAa14aLiT666LF3DvFqotuZEVuLIZc2ruh7oMf71PG9t8nmC4VCpE HIG237IzNZ4eEWqxnNA0soMAn+vtFGzVpUZFoFIKrl8/13+We7P44Vc7/v15jWR9Hp q6dzC4/njFmmMJtKrOk2pIRvj8ChfXYtCgt8dSAACcyMfvKe4AWZWi/Z2B4QWRs7c/ Cu4zPYI92IfsW+xBclcsA9aTKIcKhTL1lHANIDlVA2VjNoaaSCDKM70UOs5fGTbUp4 t3/iGj8sAXvmM7GjOBn+A7oFSIYRsvpXciGMGnMz4k7Lh0lIF3/mUScGl4KIvcYmC3 jBZJoVVNk1dDw== X-Virus-Scanned: amavisd-new at inparadise.se Received: from mail.inparadise.se ([127.0.0.1]) by localhost (mail.inparadise.se [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Gv-u_j6ClAFl; Thu, 8 Jun 2017 23:44:38 +0200 (CEST) Received: from mail.inparadise.se (localhost [127.0.0.1]) by mail.inparadise.se (Postfix) with ESMTP id 5AD384794C; Thu, 8 Jun 2017 23:44:38 +0200 (CEST) Date: Thu, 8 Jun 2017 23:44:37 +0200 (CEST) Subject: Re: ZFS Commands In "D" State Message-ID: <7e8aab1f-faba-4a80-83c8-9bf71946e44c@email.android.com> X-Android-Message-ID: <7e8aab1f-faba-4a80-83c8-9bf71946e44c@email.android.com> To: Tim Gustafson Cc: freebsd-fs@freebsd.org Importance: Normal X-Priority: 3 X-MSMail-Priority: Normal From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= X-Originating-IP: [172.16.1.154, 127.0.0.1] X-Mailer: Zimbra 8.7.1_GA_1670 (Android-Mail/7.5.7.156101332.release(...883836) devip=172.16.1.154 ZPZB/66) Thread-Index: bzex0/r21Shx83ylBz/c9yZgxTYHiA== Thread-Topic: ZFS Commands In "D" State MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 21:53:03 -0000 From owner-freebsd-fs@freebsd.org Thu Jun 8 22:13:21 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31A75D851C5 for ; Thu, 8 Jun 2017 22:13:21 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x22d.google.com (mail-lf0-x22d.google.com [IPv6:2a00:1450:4010:c07::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1FD47422D for ; Thu, 8 Jun 2017 22:13:20 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x22d.google.com with SMTP id o83so23229045lff.3 for ; Thu, 08 Jun 2017 15:13:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=k/9Nlc3TFxXNseBaupmXb9I1G32+u798rrX+nDUu3WY=; b=SiMKMc2JL10jS6Q7FPL0gkdyMWHhVF7DShVOwNXAajjkFHJbTky3xF8uBjX90tw1V7 Gnc0IzTSDM2E6j1n/edkGdeMHezzDXjnsUXtOd0dE8g5s3lOBSqosqSr/KSGX3Exh8Wl hE3E6EF8R9tt7hDrpPIChOw7gBLYrBrHnv4/M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=k/9Nlc3TFxXNseBaupmXb9I1G32+u798rrX+nDUu3WY=; b=BOglM0IR1X3HJU65i97sbu09+gaRunyYBjOUlj6EGVxFGaqGJZAhzLRjLia3hLPDzA RxvHKLFgdFJ/YoBU4xcD7W7M76QPbs3pLOGPydV0DFAqb0AmMoqnOX8FDhlcjMDmxgRx hrJxEWDE+cJwDxV1Y3EGqBVifMKuQvDbZ8M5vwynNApp9Qm3QOgto9bFCxzfIMfIBb2z F8jBeYjwpyt+xknFAXQYTa6mHez1Rdb54rgoJ9fJfQEZlfa6Baggf70DGzLZKgjtqVdy Jp3lk1mrTFGXVLheUTII1aHOwT6ZKkfV7UOMyl7vTADrPx3E4+QPmeSg9GxNvmUJNI22 YFFg== X-Gm-Message-State: AODbwcBqGJwP44CZKN2dd7nkDgJL1zMs/E503uHo4isYiA/xfNgMutId tBCDF4U2YTEY5ozmLUc50XPEPluRtV0o X-Received: by 10.46.77.88 with SMTP id a85mr11209025ljb.103.1496959998727; Thu, 08 Jun 2017 15:13:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Thu, 8 Jun 2017 15:13:18 -0700 (PDT) In-Reply-To: <7e8aab1f-faba-4a80-83c8-9bf71946e44c@email.android.com> References: <7e8aab1f-faba-4a80-83c8-9bf71946e44c@email.android.com> From: Tim Gustafson Date: Thu, 8 Jun 2017 15:13:18 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: =?UTF-8?Q?Karli_Sj=C3=B6berg?= Cc: freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 22:13:21 -0000 >> A few days ago, our nightly ZFS send failed. When I looked at the >> server, I saw that the "zfs receive" command was in a "D" wait state: > > Dedup? No, that's a kernel process state; it's got nothing at all to do with ZFS. -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Thu Jun 8 22:57:45 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3297BD85F11 for ; Thu, 8 Jun 2017 22:57:45 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-it0-x232.google.com (mail-it0-x232.google.com [IPv6:2607:f8b0:4001:c0b::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EFBCD75393 for ; Thu, 8 Jun 2017 22:57:44 +0000 (UTC) (envelope-from delphij@gmail.com) Received: by mail-it0-x232.google.com with SMTP id m62so130603957itc.0 for ; Thu, 08 Jun 2017 15:57:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=oQ1MrVVzUcXs7QTJWSDHbZ1tV/tCSz2ut6mN8OMF3QY=; b=IMqJC7qgLHy9vkOOnsPJCJpn9/rMKqfr6yRX47V62PQxSEUIMn6Koqwt49ck4kXJPP Ad6wZzPvwhj4m5oP+oBnlCU5W2I4YQ9VuNNNeTi++BeqBcjKL9WOFLJ/WlXrtYRcRl5c VFzgNCAn6aeHW67AG//d16GgnbZNP23WhWRFiuUI6rbf60RXb6F1/UGiGgVBf8TBerTP DXaqIGdrNvVe/bn5cIIfK0h7OGSNFOvHvi464Ozap3I/fbJ8ecDnWpYEYaqE11LXU42B 81zGJeLTp4o9SDsFPInEcyZeqrCfw6Do67JlVrt1MkdWuJDOL3KOTJuhIQQDQbe06rq5 xK7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=oQ1MrVVzUcXs7QTJWSDHbZ1tV/tCSz2ut6mN8OMF3QY=; b=ALcjUgQdbU0moQS9eugMDTjHyxBG0dCCadWeflFRQcTPTLJiwiQwrDzlzQNwMImvAS LYYmhodlgvmz0Do0CIULW55M0XpY7dOyMZ73rN8RqPH0BQQuhdKCK08O8cPMPdKQFh5i Hha3l5OXKvSpuQgt9YKKa4emPx9zJcAVAujGNX+TEC+h3ZU56/BhtYOXGDLZH6J2Mdsu bHfn9gsO+MPD1l7wvNdb/UQGBq90JIMnrH+aVAxSik6KyYSZNG1QZXkowN79vANQmNCw IoauKaby+VE++6oIJla04cDag6AhOYX8bv9nfMm8t6TNFlj8/vN4IfqoxBPUz9X5Jum+ XreA== X-Gm-Message-State: AODbwcAlFp4GMqgfgCTGi/hdxYA0/bPVzZ96mH1Ktm+C+KlbL9Wm50GV 3TxuA1OOtFtb0IJ7hrCRikqG0MyMWg== X-Received: by 10.36.65.18 with SMTP id x18mr8245106ita.88.1496962664346; Thu, 08 Jun 2017 15:57:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.4.134 with HTTP; Thu, 8 Jun 2017 15:57:43 -0700 (PDT) In-Reply-To: References: From: Xin LI Date: Thu, 8 Jun 2017 15:57:43 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: Tim Gustafson Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 22:57:45 -0000 procstat -kk 1425 ? (or whatever PID's that is stuck in "D" state) On Thu, Jun 8, 2017 at 2:13 PM, Tim Gustafson wrote: > We have a ZFS server that we've been running for a few months now. > The server is a backup server that receives ZFS sends from its primary > daily. This mechanism has been working for us on several pairs of > servers for years in general, and for several months with this > particular piece of hardware. > > A few days ago, our nightly ZFS send failed. When I looked at the > server, I saw that the "zfs receive" command was in a "D" wait state: > > 1425 - D 0:02.75 /sbin/zfs receive -v -F backup/export > > I rebooted the system, checked that "zpool status" and "zfs list" both > came back correctly (which they did) and then re-started the "zfs > send" on the master server. At first, the "zfs receive" command did > not enter the "D" state, but once the master server started sending > actual data (which I was able to ascertain because I was doing "zfs > send" with the -v option), the receiving process entered the "D" state > again, and another reboot was required. Only about 2MB of data got > sent before this happened. > > I've rebooted several times, always with the same result. I did a > "zpool scrub os" (there's a separate zpool for the OS to live on) and > that completed in a few minutes, but when I did a "zpool scrub > backup", that process immediately went into the "D+" state: > > 895 0 D+ 0:00.04 zpool scrub backup > > We run smartd on this device, and that is showing no disk errors. The > devd process is logging some stuff, but it doesn't appear to be very > helpful: > > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=11754027336427262018 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=11367786800631979308 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=18407069648425063426 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=9496839124651172990 > Jun 8 13:52:49 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=332784898986906736 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=16384086680948393578 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=10762348983543761591 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=8585274278710252761 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=17456777842286400332 > Jun 8 13:52:50 backup ZFS: vdev state changed, > pool_guid=2176924632732322522 vdev_guid=10533897485373019500 > > No word on which state it changed "from" or "to". Also, the system > only has three vdevs (the OS one, and then two raidz2 vdevs that make > up the "backup" pool, so I'm not sure how it's coming up with more > than 3 vdev GUIDs). > > What's my next step in diagnosing this? > > -- > > Tim Gustafson > BSOE Computing Director > tjg@ucsc.edu > 831-459-5354 > Baskin Engineering, Room 313A > > To request BSOE IT support, please visit https://support.soe.ucsc.edu/ > or send e-mail to help@soe.ucsc.edu. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Jun 8 23:08:54 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37E4DD86152 for ; Thu, 8 Jun 2017 23:08:54 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x22f.google.com (mail-lf0-x22f.google.com [IPv6:2a00:1450:4010:c07::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5E9F757D7 for ; Thu, 8 Jun 2017 23:08:53 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x22f.google.com with SMTP id a136so23584619lfa.0 for ; Thu, 08 Jun 2017 16:08:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=m5pjVfB1NuYNiHCRihGKdckjQxDL5FF1NgLSmJHunCQ=; b=VbglMniraAzcw1wrNo6XO+/3f3IziuOu6wmn7m7Z78p2xt70f/glWR8Y236iYvCwSU 3jicMYclfvfFA85qSoxmo61lcTX3T66+cl5zHlUVaXIO1f1GgeEsnGmcszMQ44Qm2OlU lyTzkqCu4O9Bmpbi0WlJs96Q1qfmWkYV8dIFM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=m5pjVfB1NuYNiHCRihGKdckjQxDL5FF1NgLSmJHunCQ=; b=brpxbwb6+7XoJJ/vOSplYZycDauC0fTU9l4BlpSWXck08vhA36V+tFfcTAJYRH/aOC 1pAVU4XzoIwqNVvBEai1Il8kd2L++6HH8rSQKmq/XkDkDpSiKBQhq66veGHoIIAeg2ex m0Y7dgQi4yqqwoZofDXxR8UjtJ6UGYTFsj6/h7DhqvLunQF5HkdmuNzUf4qR4lXDCbUU yPDBYDmCTV6wQZROwtOb5u4/GHhZeEGTh3LsMeIJKjPsN25imYIquUKPB2koMHkqB6ih rhDZ6czFNrR4PuWQ/iqEWP+8vh8d1xJG/R7rZA+YfqaowAzQCTX2PsYLxzxA5M4DKehm rcOw== X-Gm-Message-State: AODbwcDHVLoHj85ntV2/LQ4bvBJD7AZPGMHP/lnoD3MHAQC1SVgmZVtj /nKp1PP0wpMRidZa5MbdRVas92Ctb6E8 X-Received: by 10.25.145.88 with SMTP id y24mr413326lfj.2.1496963331741; Thu, 08 Jun 2017 16:08:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Thu, 8 Jun 2017 16:08:51 -0700 (PDT) In-Reply-To: References: From: Tim Gustafson Date: Thu, 8 Jun 2017 16:08:51 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: Xin LI Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 23:08:54 -0000 > procstat -kk 1425 ? (or whatever PID's that is stuck in "D" state) I went over to my terminal to get that information for you, and found that the "zpool scrub" command I had started when I first posted this question had *finally* completed, and the scrub is now running, but not making much progress: scan: scrub in progress since Thu Jun 8 15:15:59 2017 1 scanned out of 103T at 1/s, (scan is slow, no estimated time) 0 repaired, 0.00% done So, in 50 minutes it's scanned 1 byte out of 103T. Since the scrub command completed, I decided to try another ZFS send. That did enter the D state as soon as it started transmitting data again, and the output of the command you suggested is: PID TID COMM TDNAME KSTACK 1646 101975 zfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 zio_wait+0x5b arc_read+0x8f0 dmu_objset_open_impl+0xed dmu_objset_from_ds+0x65 zfs_ioc_snapshot_list_next+0x178 zfsdev_ioctl+0x5f5 devfs_ioctl_f+0x13f kern_ioctl+0x2d4 sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Thu Jun 8 23:21:01 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42559D86493 for ; Thu, 8 Jun 2017 23:21:01 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x232.google.com (mail-lf0-x232.google.com [IPv6:2a00:1450:4010:c07::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF52575DA3 for ; Thu, 8 Jun 2017 23:21:00 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x232.google.com with SMTP id v20so23653214lfa.1 for ; Thu, 08 Jun 2017 16:21:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=RFsR6ubos0aICLkF67SG++TZo/B/FPXj6vLMFcbACjs=; b=XJF4muVEE7fecfIOAiKdaZo5bFWas7k7m+CRkcdHQqPwm3EZOc6KTi3vTchZgNdnUZ XO41vtkjXpRN0hyQF0ugoI1fqo/HCDKTb4Aut9qHevxSlQwE2gHpd3cIx2r1kITd+Nfg /G9UtDSNwDU7KoFV1ZXJr+2QFsWLXACTXgbz4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RFsR6ubos0aICLkF67SG++TZo/B/FPXj6vLMFcbACjs=; b=TuCw3qR+mOPi+NgfZyuUdC8swqdVG+0DUppPBl09bk/1EWZeV0gZoR4oYKZfJr71xG rLyZfcgFywU8wYtvrrIFEWTWMoCEUGqzlJMS2u5CwavlUaI73wlcyr7OJJmGvTiA03U8 k/eBVS5MB/p9w3baGAM8vC7SbYLG4kdTxs/Lo+4d298/FvkCBqw+20DVaE5E98Ithpv6 M888a1nK0/qIOrS9kI3NNkoVxsZMOEieTY+TO1OhaIhngsnIBWY+iNGUGcGc3Up0rdHa 6awvym2h+QRVE0ZIMPQ5ejpVEYpbbOpdggry34UbjmHSVb2xFaCoza/melaL73TKJjzq vkfw== X-Gm-Message-State: AODbwcA7lcbupy1yMd4LYB7nBL2fmor7kcvsBsCNB5jtWWVJe6VbMBo0 s2G0TQcVaKfXT16U+l7DFo2AmtVG9Ch9HPfipQ== X-Received: by 10.46.5.130 with SMTP id 124mr11179542ljf.95.1496964058790; Thu, 08 Jun 2017 16:20:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Thu, 8 Jun 2017 16:20:58 -0700 (PDT) In-Reply-To: References: From: Tim Gustafson Date: Thu, 8 Jun 2017 16:20:58 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: Xin LI Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jun 2017 23:21:01 -0000 By the way, the output of the procstat command changed after it ran for a while (but it's still in the "D" state): PID TID COMM TDNAME KSTACK 1646 101975 zfs - mi_switch+0xd2 sleepq_wait+0x3a _cv_wait+0x194 txg_wait_synced+0x85 dsl_sync_task+0x205 dsl_destroy_snapshot+0x87 zfs_ioc_destroy+0x3c zfsdev_ioctl+0x5f5 devfs_ioctl_f+0x13f kern_ioctl+0x2d4 sys_ioctl+0x171 amd64_syscall+0x4ce Xfast_syscall+0xfb -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Fri Jun 9 03:18:14 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82EE3D89AE1 for ; Fri, 9 Jun 2017 03:18:14 +0000 (UTC) (envelope-from mckay@freebsd.org) Received: from ipmail05.adl6.internode.on.net (ipmail05.adl6.internode.on.net [150.101.137.143]) by mx1.freebsd.org (Postfix) with ESMTP id D5D897BDE6; Fri, 9 Jun 2017 03:18:13 +0000 (UTC) (envelope-from mckay@freebsd.org) Message-Id: <0fc687$sij78n@ipmail05.adl6.internode.on.net> X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ApKgAOEjpZ/7u1Ag5eHAEBBAEBCgEBgmtuUoEdg3SKYSqQeYMikmCCEYYeBAICgnpCFgECAQEBAQEBAWsohRgBAQEBAgEjMyMQCAMYAgImAgI5HgYTiiMHsSiCJotrAQEIAoExiiI0h3wSgk8BBIlGiEqMLKVBlGgmBiuBCoEBCIdnLjaJSgECAw Received: from ppp14-2-181-187.bras1.hba2.internode.on.net (HELO localhost) ([14.2.181.187]) by ipmail05.adl6.internode.on.net with ESMTP; 09 Jun 2017 12:48:05 +0930 Content-Type: text/plain; charset=UTF-8 From: Stephen McKay To: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= cc: Stephen McKay , freebsd-fs@freebsd.org Subject: Re: Problem with zpool remove of log device References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net><4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> In-Reply-To: <4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> from =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= at "Wed, 07 Jun 2017 12:56:30 +0200" Date: Fri, 09 Jun 2017 13:18:04 +1000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 03:18:14 -0000 On Wednesday, 7th June 2017, =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= wrote: >W dniu 2017-06-07 o 10:21, Stephen McKay pisze: > >> On Friday, 26th May 2017, lukasz@wasikowski.net wrote: >> >>> I cant remove log device from pool - operation ends ok, but log device >>> is still in the pool (bug?). >> Does your system still write to the log? Use "zfs iostat -v 1" to >> check. I think it is probably no longer be in use and only the final >> disconnection failed. > >zpool iostat -v 1 shows no activity on the log drive. I tried to remove >it after booting to livecd - no luck. I hope I haven't mislead you there (more than misspelling zpool, that is). In normal operation the log is idle unless there are synchronous writes to log. So, to be sure your log is active, use something like this (within the target pool) to generate sync writes: while : do date > foo fsync foo done With this running, my system does 600 writes per second to the log according to zpool iostat. That drops to zero once I kill the script. >> I had this problem when I tried to remove a log mirror from a pool >> I have been using for years. I solved it by tweaking the zfsonlinux >> hack a bit and slotting it into 9.3. > >Thank you for this info. It seems that it's no longer an issue for me - >this box will be soon retired. I'll move some data from it and wipe it down. If you are feeling public spirited this is an ideal opportunity to check the theory against reality. I'm curious about how widespread this accounting bug is, so if you can keep the machine intact for a while after it has been decommissioned, you could build a patched kernel and try to remove the log again. All it would cost in this case is a little time. Cheers, Stephen. From owner-freebsd-fs@freebsd.org Fri Jun 9 10:15:42 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E446BF14D6 for ; Fri, 9 Jun 2017 10:15:42 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:2:1276::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3709F1BCD; Fri, 9 Jun 2017 10:15:42 +0000 (UTC) (envelope-from lukasz@wasikowski.net) Received: from mail.freebsd.systems (mail.freebsd.systems [IPv6:2001:41d0:2:1276::1]) by mail.freebsd.systems (Postfix) with ESMTP id 87557316; Fri, 9 Jun 2017 12:15:40 +0200 (CEST) X-Virus-Scanned: amavisd-new at freebsd.systems Received: from mail.freebsd.systems ([IPv6:2001:41d0:2:1276::1]) by mail.freebsd.systems (scan.freebsd.systems [IPv6:2001:41d0:2:1276::1]) (amavisd-new, port 10026) with ESMTP id j83_mLhyBAPH; Fri, 9 Jun 2017 12:15:40 +0200 (CEST) Received: from [192.168.138.100] (unknown [194.181.68.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.freebsd.systems (Postfix) with ESMTPSA id A8723313; Fri, 9 Jun 2017 12:15:39 +0200 (CEST) Authentication-Results: mail.freebsd.systems; dmarc=none header.from=wasikowski.net Authentication-Results: mail.freebsd.systems; spf=pass smtp.mailfrom=lukasz@wasikowski.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wasikowski.net; s=default; t=1497003339; bh=P0fCTh/j5/9ezO5OhyVPNavpQDH7D3EX3jh0ECLKOfI=; h=To:Cc:References:From:Date:In-Reply-To; b=G+9HUMWiyUP7N7IwIPpvBFgal4x3GdMuRT5UzfLG+cEKfXC7ly37Rhka/gnmOrBfz 95S577a6ByRnGMSYbEkCpZQEe5R7ILgQt5NMcNoypeYfKZkokThE9T8D201E3qJ5Nc lMTq/AFvxKqgK2WuQTKRi37m5pVeKE2m70QW03iPcZTGZNQVsMVfZScTJMhAB4AX8i JxPo2pt/1k4to0RFiJU131JCNYEH6XDLorBvuoTRX30sO/roJzWQIJs0WXTYkX+V/P att9MKNyX0m+UKHSYxPxFnzh0cRME5nmtx9Vv1YBZaHY+WRo9C60fpjmYB4+ssVb38 oXZey/t6CJ5EA== Subject: Re: Problem with zpool remove of log device To: Stephen McKay Cc: freebsd-fs@freebsd.org References: <9188a169-cd81-f64d-6b9e-0e3c6b4af1bb@wasikowski.net> <0410af$1dldvp4@ipmail04.adl6.internode.on.net> <4df1ea6d-148e-f3ab-d204-9277c513a468@wasikowski.net> <0fc687$sij78n@ipmail05.adl6.internode.on.net> From: =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= Message-ID: Date: Fri, 9 Jun 2017 12:15:32 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <0fc687$sij78n@ipmail05.adl6.internode.on.net> Content-Type: text/plain; charset=utf-8 Content-Language: pl Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 10:15:42 -0000 W dniu 2017-06-09 o 05:18, Stephen McKay pisze: > On Wednesday, 7th June 2017, =?UTF-8?Q?=c5=81ukasz_W=c4=85sikowski?= wrote: > >> W dniu 2017-06-07 o 10:21, Stephen McKay pisze: >> >>> On Friday, 26th May 2017, lukasz@wasikowski.net wrote: >>> >>>> I cant remove log device from pool - operation ends ok, but log device >>>> is still in the pool (bug?). > >>> Does your system still write to the log? Use "zfs iostat -v 1" to >>> check. I think it is probably no longer be in use and only the final >>> disconnection failed. >> >> zpool iostat -v 1 shows no activity on the log drive. I tried to remove >> it after booting to livecd - no luck. > > I hope I haven't mislead you there (more than misspelling zpool, that is). > > In normal operation the log is idle unless there are synchronous writes > to log. So, to be sure your log is active, use something like this (within > the target pool) to generate sync writes: > > while : > do > date > foo > fsync foo > done > > With this running, my system does 600 writes per second to the log > according to zpool iostat. That drops to zero once I kill the script. Zero, so no writes to log are performed during execution of this script. >>> I had this problem when I tried to remove a log mirror from a pool >>> I have been using for years. I solved it by tweaking the zfsonlinux >>> hack a bit and slotting it into 9.3. >> >> Thank you for this info. It seems that it's no longer an issue for me - >> this box will be soon retired. I'll move some data from it and wipe it down. > > If you are feeling public spirited this is an ideal opportunity to check > the theory against reality. I'm curious about how widespread this accounting > bug is, so if you can keep the machine intact for a while after it has > been decommissioned, you could build a patched kernel and try to remove the > log again. All it would cost in this case is a little time. I applied this patch to 11.1-PRERELASE, nothing changed. Still zpool remove exits with errcode 0, but log device is still attached to pool. -- Z poważaniem, Łukasz Wąsikowski From owner-freebsd-fs@freebsd.org Fri Jun 9 15:46:48 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A040FBF7DBD for ; Fri, 9 Jun 2017 15:46:48 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x22a.google.com (mail-lf0-x22a.google.com [IPv6:2a00:1450:4010:c07::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 37F2B71F34 for ; Fri, 9 Jun 2017 15:46:48 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x22a.google.com with SMTP id v20so31392663lfa.1 for ; Fri, 09 Jun 2017 08:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=6mi0pnboUpg4gpCEP9nIS9/MzS/63upZEyIq4yRXUqI=; b=H3wQaPS9S6XkYHBFJjXbYodg/J7z21fdy8o9SRu/Gvl6y599hQ54BdI2Ys1JRb/AaX HJEsg14Gfuvq6KPNM8m3lOod/l87jNMZwddKPsCG2DKxKCnGdpvQdawPNMEsByzLrAm0 Y06ikKllyArb31TPWM2Zuy4+idOqIU8mMeOT0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=6mi0pnboUpg4gpCEP9nIS9/MzS/63upZEyIq4yRXUqI=; b=ccgn6tSh1ASNtZpAmNbUpsBDT5wtoMMMY4TG7uLcc2ru572jN/O7VmlRodNZDeZDwT eHPE8TpUkAUaFdDDaNtUZe2kP8eraJ5FNJfAH81JN5z9WCHa/aLOd+mWV+MQWjpZK9Uj huMGykllGvLDNnOJIeopZ5IOUr/OhRu+MReKpfpGfseqHmUadiPb7Ikixvagsjo7To2o YZYm8X87px12sdo/xSDsFqGMKAnznX/A/5FmhDDOWsQjSDNMOVwxkL9r0Fzm/raZ7Gto 7ZFCLMQO7D2zhkvkc3JNlQmv1sKHTAmhAMeonJVw/cQTiwg286Bx2hrq0dbAzM0/IB5q D7JQ== X-Gm-Message-State: AODbwcDffjwy3Znkzm0rgBhIYr/nmChz6rwC5ltppA9mEjRXjyHFp6hf +gkU43GxpUeT+0Z5yILzTlWiR4R8fTCZa+xJ5g== X-Received: by 10.46.21.11 with SMTP id s11mr12086417ljd.2.1497023205978; Fri, 09 Jun 2017 08:46:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Fri, 9 Jun 2017 08:46:45 -0700 (PDT) In-Reply-To: References: From: Tim Gustafson Date: Fri, 9 Jun 2017 08:46:45 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 15:46:48 -0000 I wanted to follow up with some new status on this. So, the zpool scrub is going still, but it's going *very* slowly: pool: backup state: ONLINE scan: scrub in progress since Thu Jun 8 15:15:59 2017 644G scanned out of 103T at 10.5M/s, (scan is slow, no estimated time) 0 repaired, 0.61% done I started another zfs send/receive before I left yesterday, and it did complete successfully. I'm re-running last night's transfer again (it failed because the one I started before I left yesterday was still going), and it seems to be going at a reasonable rate of speed. So, I'm really not sure what is happening here. I'm still concerned that the zpool scrub is going so slowly. Usually we can scrub that amount of data in 3-4 days; at this rate, it will probably take a month or two. Since there isn't a userland process for the scrub, is there another way I can procstat that process to see what it's doing? -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Fri Jun 9 23:24:56 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C9C9BFF29A for ; Fri, 9 Jun 2017 23:24:56 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: from mail-lf0-x22e.google.com (mail-lf0-x22e.google.com [IPv6:2a00:1450:4010:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0483A7FCA9 for ; Fri, 9 Jun 2017 23:24:56 +0000 (UTC) (envelope-from tjg@ucsc.edu) Received: by mail-lf0-x22e.google.com with SMTP id v20so34883448lfa.1 for ; Fri, 09 Jun 2017 16:24:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=GuVbal+sQt7Md3/Mlwobprw1j6sNMfwBBXmvZVBxMwM=; b=Kw15KJ7hPmzkoMuKx71SqMtXYRRy6qkM8LdA9fcxA854Ny/abCR3Zvqg2GMpOW+xCN Wqx/q306hmK/p8JHyvQm3OOzYcgZhBYcZy/Arb/G4FfvX7+TfRzW6DEjbHcsMPN9fFxU nu7wxqXEAMbTLySQMz8J/h3giodeJWx2G6h9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=GuVbal+sQt7Md3/Mlwobprw1j6sNMfwBBXmvZVBxMwM=; b=UpSrGOtk9Y9/ib59SRkb/qtInlsUTXKpUOUZKlyfAGe9q26em+a4br6pbk7+S59Afi QZJP3NXiDa+/Pwt6RpcvHT8b6+S8SMqUhNsM2LzjcysyhC8GNH39cBrPgc8aRF+NQds2 zisRvJweBTZB6+2YbjWvQNzH8LgE7Lz3T3kAyqYoo9dw25STcCBe65h0gFndZQJLsjim WR1kld0LpV5fqvAjEmwydv83DWQnHJ9zs8TSZO8KCIx0e3/nL9Tnyzn3HXxx1AePBx51 8m4d3KiVUAnGAHb1RJLat9w3OTnZT4arpg6isZ7KYUGBduZ+4dLfNcYHVtFthTW5bm6r z78A== X-Gm-Message-State: AODbwcB/VYTl7v4bpRfE+9KG24ALsKKpCRuAoVtDY6kalg/oWYr+IYQU SXt/tcPntyTXtToLBUtdFZnsUVhlFwFeoSvlXQ== X-Received: by 10.25.0.209 with SMTP id 200mr14771302lfa.117.1497050693444; Fri, 09 Jun 2017 16:24:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.67.4 with HTTP; Fri, 9 Jun 2017 16:24:52 -0700 (PDT) In-Reply-To: References: From: Tim Gustafson Date: Fri, 9 Jun 2017 16:24:52 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 23:24:56 -0000 > I'm really not sure what is happening here. I'm still concerned that > the zpool scrub is going so slowly. Someone else suggested off-list that this might be the LSI HBA controller running its "partrol" of the disks in the system. Our HBA model is: Avago Technologies (LSI) SAS3008 We're using the mpr drive, but mprutil doesn't seem to have any way to query or set the status of the "patrol" function. I rebooted the machine into the BIOS config for the HBA, but could not find any reference to enabling or disabling "patrol" there either. Am I missing something? Does this adapter actually do any patrolling? I did notice that all the drive activity lights are lit on this server nearly all the time, but I can't tell if that means "activity" or "on-line" for that controller in a SuperMicro chassis. -- Tim Gustafson BSOE Computing Director tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A To request BSOE IT support, please visit https://support.soe.ucsc.edu/ or send e-mail to help@soe.ucsc.edu. From owner-freebsd-fs@freebsd.org Fri Jun 9 23:51:15 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62809BFF663 for ; Fri, 9 Jun 2017 23:51:15 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-it0-x231.google.com (mail-it0-x231.google.com [IPv6:2607:f8b0:4001:c0b::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A17E80510 for ; Fri, 9 Jun 2017 23:51:15 +0000 (UTC) (envelope-from delphij@gmail.com) Received: by mail-it0-x231.google.com with SMTP id m47so2785809iti.1 for ; Fri, 09 Jun 2017 16:51:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6MaecrtZK6bomDHR56DgcGH0F3ZTy+HbWiESnd8vxNo=; b=fWrXwGUASwmorjwY+munSL50qANCBoRQHULR0K6M1m7upRygUBXBlSN/IRrxU9NYeR aKe4C39DT0RZk/nGyqvN0QiQgyIr2fL2ICN90CqIBpoUT9hddl2+5kok1jcnTO0r7KXo KzYpiq+2rpHGiUD81lyHSCcukrahfdS/qNlAKAIs1b/4SfNSq6P+eKQR7tf73V0GzbJz 5Bg7cu6jjFeLq0FuoFjGrBNOaL53gLzF2UbF3o86B2D+x7jypbVf3mYYBHQDvD/2VERM 08U9OYT6skqKAI6TbxStUUCCbHx7PnZDsf+Z60ID8SjNoOiiA3acxp6/usWBn8J14OXa xhDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6MaecrtZK6bomDHR56DgcGH0F3ZTy+HbWiESnd8vxNo=; b=NzB/sC8M+bzNINZSqy7uN1/RfqYtU4rvTEXLNzklfNI6EZZpqUYHT91cy7p0DVwWjL zZEx0rE7ASpsNBZQuI9BAKOVC0OmcP4n3SRawo1nCz18BEYmMzFWQTYXYVBPs1xOdaAz JKjvOZmUbvwL29IolEcZp/hNsLX8yba8wTCKez8Y9vihrEY3GaOJv3ZHphUXpc/uyl1Q gXVhDD/Zg3FGXfkrPBNswDYJQrLTuEVec/dFSXRTccvsEmX58QpBXIGRLQv8RwjKefwE 2EzDmle3z0b0t031Qqc/i+EamxEEM3DxVhHvKlAgjnS33qkO3Ae4ZbPeCLso+ZWBUF3m 2NbQ== X-Gm-Message-State: AODbwcCtElKdvDwv49pNfXAORQahh2xz5xa+XeJRSJaEu7r0ldNP7wm2 9swfTmp9OVcOOLX9Pu1O7ItaEbUT+Q== X-Received: by 10.36.118.11 with SMTP id z11mr2153669itb.88.1497052274311; Fri, 09 Jun 2017 16:51:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.4.134 with HTTP; Fri, 9 Jun 2017 16:51:13 -0700 (PDT) In-Reply-To: References: From: Xin LI Date: Fri, 9 Jun 2017 16:51:13 -0700 Message-ID: Subject: Re: ZFS Commands In "D" State To: Tim Gustafson Cc: freebsd-fs Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 23:51:15 -0000 Yes, you can observe the progress with: dtrace -qn 'zfs-dbgmsg{printf("%s\n", stringof(arg0))}' . I'm not 100% sure if that would be helpful for diagnosing your issue, though. What's the FreeBSD release? Maybe you have a lot of memory fragments? ( bugs.freebsd.org/194513 may be related?) On Fri, Jun 9, 2017 at 8:46 AM, Tim Gustafson wrote: > I wanted to follow up with some new status on this. > > So, the zpool scrub is going still, but it's going *very* slowly: > > pool: backup > state: ONLINE > scan: scrub in progress since Thu Jun 8 15:15:59 2017 > 644G scanned out of 103T at 10.5M/s, (scan is slow, no estimated time) > 0 repaired, 0.61% done > > I started another zfs send/receive before I left yesterday, and it did > complete successfully. I'm re-running last night's transfer again (it > failed because the one I started before I left yesterday was still > going), and it seems to be going at a reasonable rate of speed. So, > I'm really not sure what is happening here. I'm still concerned that > the zpool scrub is going so slowly. Usually we can scrub that amount > of data in 3-4 days; at this rate, it will probably take a month or > two. > > Since there isn't a userland process for the scrub, is there another > way I can procstat that process to see what it's doing? > > -- > > Tim Gustafson > BSOE Computing Director > tjg@ucsc.edu > 831-459-5354 > Baskin Engineering, Room 313A > > To request BSOE IT support, please visit https://support.soe.ucsc.edu/ > or send e-mail to help@soe.ucsc.edu. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sat Jun 10 22:52:33 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14942BF7CE0 for ; Sat, 10 Jun 2017 22:52:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ECEEE65A3D for ; Sat, 10 Jun 2017 22:52:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v5AMqW4T021373 for ; Sat, 10 Jun 2017 22:52:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 219760] ZFS iSCSI w/ Win10 Initiator Causes pool corruption Date: Sat, 10 Jun 2017 22:52:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: henric_jungheim@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jun 2017 22:52:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D219760 --- Comment #5 from Henric Jungheim --- (In reply to Edward Tomasz Napierala from comment #4) The ada4 and ada3 disks were added because the existing mirror set in the p= ool was full (as in "getting out of disk space errors", not "uncomfortably high= "), so they are not likely to see any significant writes. This is a home NAS s= etup and while I have backups in the cloud and some on LTO3 tapes, I don't have anywhere local I can store what is on those drives so I'm really reluctant = to pull those drives. I was not able to find any firmware updates for those drives. Both long and short SMART tests have been run w/o finding anything. Writing multiple 1TB files to files with "dd if=3D/dev/random" hasn't caused any problems. I'll= try doing those "dd"s to zdevs next. The "write a couple TB, then scrub," test= ing takes a while; any thoughts on other non-destructive testing I could do? Is there some sane way to get a list of what the actual errors are in the p= ool (offset on the raw disk or such)? When I've done the backup, both drives h= ave always reported the same number of errors. If both drives report exactly t= he same (bad) data at the same locations, then that could perhaps suggest something useful? I have both the chipset SATA ports and SAS/SATA ports from the motherboards= LSI SAS controller. I might be able to move the drives from one controller to = the other. Another device--a Server 2008 R2 box--has been doing similar iSCSI backups = to another pool in the system for years. It, however, has much less to write = and does so from spinning rust instead of SSD (although, I would assume the GigE link is still the bottleneck). The drives for this other pool are on the s= ame controller as ada3 and ada4. I'll see about having it do an extra backup or two to the "tank" pool to see if it causes any grief there. For ada3: smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-PRERELEASE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org =3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D Model Family: HGST Deskstar NAS Device Model: HGST HDN726060ALE610 Serial Number: NCGTEJ2S LU WWN Device Id: 5 000cca 24dcb1ba7 Firmware Version: APGNT517 User Capacity: 6,001,175,126,016 bytes [6.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Jun 10 15:49:55 2017 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled =3D=3D=3D START OF READ SMART DATA SECTION =3D=3D=3D SMART overall-health self-assessment test result: PASSED For ada4: smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.1-PRERELEASE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org =3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D Model Family: HGST Deskstar NAS Device Model: HGST HDN726060ALE610 Serial Number: NCGTES1S LU WWN Device Id: 5 000cca 24dcb1c7f Firmware Version: APGNT517 User Capacity: 6,001,175,126,016 bytes [6.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Jun 10 15:46:34 2017 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled =3D=3D=3D START OF READ SMART DATA SECTION =3D=3D=3D SMART overall-health self-assessment test result: PASSED --=20 You are receiving this mail because: You are the assignee for the bug.=