From owner-freebsd-fs@freebsd.org Sun Aug 9 21:00:12 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BE1099D956 for ; Sun, 9 Aug 2015 21:00:12 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EAF7F1E6D for ; Sun, 9 Aug 2015 21:00:11 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t79L0BSA035608 for ; Sun, 9 Aug 2015 21:00:11 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201508092100.t79L0BSA035608@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 09 Aug 2015 21:00:11 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2015 21:00:12 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 3 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Aug 10 04:43:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AAAFF99D107 for ; Mon, 10 Aug 2015 04:43:02 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 68EF76D0 for ; Mon, 10 Aug 2015 04:43:02 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from [192.168.0.170] (laptop1.herveybayaustralia.com.au [192.168.0.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id BD71361F87 for ; Mon, 10 Aug 2015 14:42:56 +1000 (EST) To: freebsd-fs@freebsd.org From: Da Rock Subject: Urgently need some help solving lost space due to snapshots during receive op Message-ID: <55C82BCD.2050404@herveybayaustralia.com.au> Date: Mon, 10 Aug 2015 14:42:53 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 04:43:02 -0000 I'm trying to move a pool from one system to another - exact same hdd and config, different in other areas. Both register same space available, and so should be no issue. The old system is quite full - still at least 5% free though. I have done a recursive snapshot on the system and a send/receive over ssh. Near the end, it final croaks and says no space left. The snapshot it is sending is nowhere to be found on the receiving system. Ok, checked free space - should be enough - so I try a simple cp op on the last dataset, but then it again croaks no space near the end, but at least I have some data. Checking space again, it says free space of an expected value, but still won't allow further data. I can't seem to figure out why this is not working, I'm currently trying using compression on the dataset to see if it can't squeeze it on there, but I still have a couple of no space messages so far. Where has my space gone?! TIA From owner-freebsd-fs@freebsd.org Mon Aug 10 08:12:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1969D9982D7 for ; Mon, 10 Aug 2015 08:12:28 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F3AB7CEB for ; Mon, 10 Aug 2015 08:12:27 +0000 (UTC) (envelope-from julian@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id F322A9982D6; Mon, 10 Aug 2015 08:12:27 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2AE09982D5 for ; Mon, 10 Aug 2015 08:12:27 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C20ABCEA for ; Mon, 10 Aug 2015 08:12:27 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (50-196-156-133-static.hfc.comcastbusiness.net [50.196.156.133]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t7A8CGrH012102 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 10 Aug 2015 01:12:21 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: Using SSDs as swap To: Konstantin Belousov , Willem Jan Withagen References: <55C5D48E.6010605@digiware.nl> <20150808102900.GA2072@kib.kiev.ua> <20150808103810.GB2072@kib.kiev.ua> <55C5E697.4080102@digiware.nl> <20150808114107.GD2072@kib.kiev.ua> Cc: fs@freebsd.org From: Julian Elischer Message-ID: <55C85CDB.80600@freebsd.org> Date: Mon, 10 Aug 2015 16:12:11 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150808114107.GD2072@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 08:12:28 -0000 On 8/8/15 7:41 PM, Konstantin Belousov wrote: > On Sat, Aug 08, 2015 at 01:23:03PM +0200, Willem Jan Withagen wrote: >> On 8-8-2015 12:38, Konstantin Belousov wrote: >>> On Sat, Aug 08, 2015 at 01:29:00PM +0300, Konstantin Belousov wrote: >>>> On Sat, Aug 08, 2015 at 12:06:06PM +0200, Willem Jan Withagen wrote: >>>>> one of the following commits just passed with this in the log, and it >>>>> triggered again a question I've been having for some time again already. >>>>> >>>>> ---- >>>>> Log: >>>>> Enable BIO_DELETE passthru in GELI, so TRIM/UNMAP can work as expected >>>>> when >>>>> GELI is used on a SSD or inside virtual machine, so that guest can tell >>>>> host that it is no longer using some of the storage. >>>>> ----- >>>>> >>>>> In ZFS I slice my SSD's into log and caches, but on a a server with >>>>> little memory (which can't be grown) I use a partion on each ssd as swap >>>>> as well. So swappinging does not have to seek, and has faster loading >>>>> time. To allocate a few GB on aan SSD to swap is not really all that >>>>> painfull, given current sizes, but the speed difference with regular >>>>> spindels is impressive. >>>>> >>>>> But the questions are: >>>>> 1) Does the swap driver understand that backing-store needs a TRIM? >>>> No. >>>> >>>>> 1a) if not would it be useful, and what would it take to implement? >>>> One good thing is that it is simply the question of coding: the VM >>>> already has a place where it informs the swap pager that the page copy >>>> in swap is no longer needed. this is the vm_pager_page_unswapped() call >>>> and swap pager method swap_pager_unswapped(). swp_pager_meta_ctl() would >>>> need to issue BIO_DELETE to the backing storage. >>>> >>>> On the other hand, note that this would increase the amount of work >>>> performed, even for the swap volumes located on the rotating media, >>>> which is more typical and reasonable setup. >>>> >>>> I think an implementation and a knob to turn it off, or configure per >>>> swap partition, would be reasonable. >>> One additional thing: while BIO_DELETE is in progress, the swap block >>> cannot be marked free, since otherwise we could write other page and >>> get it obliterated with the TRIM. This can be done async, but the >>> consequence is that swap space would be released and usable some time >>> after the page-in. This will affect loads which are close to OOM. >> Sort of makes sense to me... >> >> I take it that BIO_DELETE fires and returns before TRIM is completed? >> But then the SSD accepts writes to a TRIMmed block, but then mixes this >> up? Possibly deleting a write to a to be trimmed block? This sort of >> strikes me as odd, but then I do not know the full intricate details of >> TRIM on SSD >> >> Would it be possible to be notified that a TRIM has completed, only then >> to actually free the swap sectors? > This is exactly what I wrote above. Having worked on the other side of the dotted line at an ssd vendor, Trim can be both easy and hard depending on many implementation details of how they do SSD. Part of the hard part is that trim needs to be persistent, except when it doesn't. In the case of swap, it could be non persistent if the system were to do a huge trim on the whole of swap when it starts up. We tried to make our TRIM implementable in a single fast action so there wouldn't need to be any notification of 'completion' as such. (well there would be at the lowest level, but we would be talking a couple of uSec). The persistence is hard because you want it to remember the trim if power is lost immediately after it happens. Which requires a write, which uses space, and time ,which is leads to the odd situation of being unable to trim because you are short of space. which means you need to have reserves you can use to trim, in order to free space.. I was often pushing for a "non persistent trim" which in fact would have been really easy for us and incredibly fast. Would have been great for things like swap. > >> And then perhaps the swap bookkeeping does not yet accommodate for a >> possible extra state? > It does not need to. The in-flight BIO_DELETE remembers the intermediate > state, the swap block should be freed only after the storage reported the > BIO_DELETE as finished. It is exactly the same as UFS handles trimming > of the free blocks, the bitmap of the used/freed blocks is only updated > after the BIO_DELETE is finished, not when the inode drops reference to > the block. >> Speaking about blocks.... Does Swap take into account that disks could >> be of a sectorsize other than 512 bytes. I would guess so, since we >> could have a 4K disk as swap disk, and doing read-modify-write for swap >> is sure going to kill performance. > swap performs i/o in the page-sized chunks at least, which are min 4k on > all supported platforms (even on arms, where we do not support smaller > pages AFAIK). > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 10 08:42:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3949D998B4B for ; Mon, 10 Aug 2015 08:42:36 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED974A13 for ; Mon, 10 Aug 2015 08:42:35 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 396091534C6; Mon, 10 Aug 2015 10:42:31 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eTIfZxO3bfV5; Mon, 10 Aug 2015 10:42:06 +0200 (CEST) Received: from [192.168.10.10] (asus [192.168.10.10]) by smtp.digiware.nl (Postfix) with ESMTPA id 439761534C0; Mon, 10 Aug 2015 10:42:06 +0200 (CEST) Subject: Re: Urgently need some help solving lost space due to snapshots during receive op To: Da Rock , freebsd-fs@freebsd.org References: <55C82BCD.2050404@herveybayaustralia.com.au> From: Willem Jan Withagen Message-ID: <55C863DE.3000200@digiware.nl> Date: Mon, 10 Aug 2015 10:42:06 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55C82BCD.2050404@herveybayaustralia.com.au> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 08:42:36 -0000 On 10-8-2015 06:42, Da Rock wrote: > I'm trying to move a pool from one system to another - exact same hdd > and config, different in other areas. Both register same space > available, and so should be no issue. The old system is quite full - > still at least 5% free though. Hi, Are the versions of your FreeBSD also equal? Because in newer version, the free space reservation is significantly bigger. So if disks are equal size, then with newer FreeBSDs you have less usable space available. Don't know the exact SVN commit where it happened. But there are more question on this topic in the list, and they all boil down to the same thing: ZFS needs more reserved space to be able to do certain things without freezing the system. I've not read any suggestions that you can circumvent this setting. As a second note I think is is more or less general understanding that a ZFS systeem needs a lot more space left free than 5% to actual be able to perform. Last number I remember as sensible maximum fill level is around 70%. If you go over it, your system is going to be busy with ZFS bookkeeping instead of data storage. (One of the things I remember from the ZFS lecture by Kirk McKusik, which I no longer can find on YouTube :( ) --WjW > I have done a recursive snapshot on the system and a send/receive over > ssh. Near the end, it final croaks and says no space left. The snapshot > it is sending is nowhere to be found on the receiving system. > > Ok, checked free space - should be enough - so I try a simple cp op on > the last dataset, but then it again croaks no space near the end, but at > least I have some data. > > Checking space again, it says free space of an expected value, but still > won't allow further data. > > I can't seem to figure out why this is not working, I'm currently trying > using compression on the dataset to see if it can't squeeze it on there, > but I still have a couple of no space messages so far. > > Where has my space gone?! > > TIA > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Aug 10 09:46:46 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFC3E99813D for ; Mon, 10 Aug 2015 09:46:46 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F857A67 for ; Mon, 10 Aug 2015 09:46:46 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from [192.168.0.170] (laptop1.herveybayaustralia.com.au [192.168.0.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id 2100061F85; Mon, 10 Aug 2015 19:46:40 +1000 (EST) Subject: Re: Urgently need some help solving lost space due to snapshots during receive op To: Willem Jan Withagen , freebsd-fs@freebsd.org References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> From: Da Rock Message-ID: <55C872FD.1060608@herveybayaustralia.com.au> Date: Mon, 10 Aug 2015 19:46:37 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55C863DE.3000200@digiware.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 09:46:46 -0000 On 10/08/2015 18:42, Willem Jan Withagen wrote: > On 10-8-2015 06:42, Da Rock wrote: >> I'm trying to move a pool from one system to another - exact same hdd >> and config, different in other areas. Both register same space >> available, and so should be no issue. The old system is quite full - >> still at least 5% free though. > Hi, > > Are the versions of your FreeBSD also equal? No. 9.1 to 10.1. > > Because in newer version, the free space reservation is significantly > bigger. So if disks are equal size, then with newer FreeBSDs you have > less usable space available. > Don't know the exact SVN commit where it happened. But there are more > question on this topic in the list, and they all boil down to the same > thing: ZFS needs more reserved space to be able to do certain things > without freezing the system. > > I've not read any suggestions that you can circumvent this setting. Crap. So I guess my best will be to compress as best I can for the moment or something like that. > > As a second note I think is is more or less general understanding that a > ZFS systeem needs a lot more space left free than 5% to actual be able > to perform. > Last number I remember as sensible maximum fill level is around 70%. > If you go over it, your system is going to be busy with ZFS bookkeeping > instead of data storage. Yeah, I noticed that. Something I'm working on... backlog of sorting activities really :) Once I'm done it'll drop to well below 50% I'd say. > (One of the things I remember from the ZFS lecture by Kirk McKusik, > which I no longer can find on YouTube :( ) > > --WjW Thanks for the info - I owe you a beer! At least I know why now and eases my growing headache from banging my head on the brick wall. > >> I have done a recursive snapshot on the system and a send/receive over >> ssh. Near the end, it final croaks and says no space left. The snapshot >> it is sending is nowhere to be found on the receiving system. >> >> Ok, checked free space - should be enough - so I try a simple cp op on >> the last dataset, but then it again croaks no space near the end, but at >> least I have some data. >> >> Checking space again, it says free space of an expected value, but still >> won't allow further data. >> >> I can't seem to figure out why this is not working, I'm currently trying >> using compression on the dataset to see if it can't squeeze it on there, >> but I still have a couple of no space messages so far. >> >> Where has my space gone?! >> >> TIA >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Aug 10 09:56:13 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1CFF998348 for ; Mon, 10 Aug 2015 09:56:12 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CE1CE0D for ; Mon, 10 Aug 2015 09:56:12 +0000 (UTC) (envelope-from email.ahmedkamal@googlemail.com) Received: by wijp15 with SMTP id p15so128632021wij.0 for ; Mon, 10 Aug 2015 02:56:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=7HocLFOqrdR+KTzW1WrHLXlwmXZhBnMjcUiputaC85E=; b=Cb5wvuQs7osI8Hp6cYFSj6x4RWVSfuV7ZyMazmIZNkddU+DbQ6aX0yD83dZfGXZG69 uN8ooEc6G9V9LMpCP3DH5uVMB/C9GZHWjupU8SUpseL1gApaLBhwEhWIEEl82XjWjhr7 Z09Gbhtc2AFNzYCss0ywfvSF9eMLLGgXfMpYzO9WXtO2XnIMp6qJFYIM/38cCJ2t4vSa FINRrA1ryRD5IJR3SKlG4pfmxCEU8tvCgjd4bG1CSRRt8/xxzNLjuQsXvZeE9Sj3VKVT LfF7XVRzCL0PcHKKIWOe5PpbFw1GOLjQfHdE9X5lRnze2yTp7RpEcORmjqx/gL/UoQ6i OGew== X-Received: by 10.180.21.244 with SMTP id y20mr22920099wie.65.1439200571020; Mon, 10 Aug 2015 02:56:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.6.143 with HTTP; Mon, 10 Aug 2015 02:55:51 -0700 (PDT) In-Reply-To: <55C872FD.1060608@herveybayaustralia.com.au> References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> <55C872FD.1060608@herveybayaustralia.com.au> From: Ahmed Kamal Date: Mon, 10 Aug 2015 11:55:51 +0200 Message-ID: Subject: Re: Urgently need some help solving lost space due to snapshots during receive op To: Da Rock Cc: Willem Jan Withagen , Ahmed Kamal via freebsd-fs Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 09:56:13 -0000 Try setting compression=lz4 on the target pool .. that might shave off some space especially if the source wasn't compressed On Mon, Aug 10, 2015 at 11:46 AM, Da Rock < freebsd-fs@herveybayaustralia.com.au> wrote: > On 10/08/2015 18:42, Willem Jan Withagen wrote: > >> On 10-8-2015 06:42, Da Rock wrote: >> >>> I'm trying to move a pool from one system to another - exact same hdd >>> and config, different in other areas. Both register same space >>> available, and so should be no issue. The old system is quite full - >>> still at least 5% free though. >>> >> Hi, >> >> Are the versions of your FreeBSD also equal? >> > No. 9.1 to 10.1. > >> >> Because in newer version, the free space reservation is significantly >> bigger. So if disks are equal size, then with newer FreeBSDs you have >> less usable space available. >> Don't know the exact SVN commit where it happened. But there are more >> question on this topic in the list, and they all boil down to the same >> thing: ZFS needs more reserved space to be able to do certain things >> without freezing the system. >> >> I've not read any suggestions that you can circumvent this setting. >> > Crap. So I guess my best will be to compress as best I can for the moment > or something like that. > >> >> As a second note I think is is more or less general understanding that a >> ZFS systeem needs a lot more space left free than 5% to actual be able >> to perform. >> Last number I remember as sensible maximum fill level is around 70%. >> If you go over it, your system is going to be busy with ZFS bookkeeping >> instead of data storage. >> > Yeah, I noticed that. Something I'm working on... backlog of sorting > activities really :) Once I'm done it'll drop to well below 50% I'd say. > >> (One of the things I remember from the ZFS lecture by Kirk McKusik, >> which I no longer can find on YouTube :( ) >> >> --WjW >> > Thanks for the info - I owe you a beer! At least I know why now and eases > my growing headache from banging my head on the brick wall. > > >> I have done a recursive snapshot on the system and a send/receive over >>> ssh. Near the end, it final croaks and says no space left. The snapshot >>> it is sending is nowhere to be found on the receiving system. >>> >>> Ok, checked free space - should be enough - so I try a simple cp op on >>> the last dataset, but then it again croaks no space near the end, but at >>> least I have some data. >>> >>> Checking space again, it says free space of an expected value, but still >>> won't allow further data. >>> >>> I can't seem to figure out why this is not working, I'm currently trying >>> using compression on the dataset to see if it can't squeeze it on there, >>> but I still have a couple of no space messages so far. >>> >>> Where has my space gone?! >>> >>> TIA >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Mon Aug 10 10:37:09 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A1E0998F58 for ; Mon, 10 Aug 2015 10:37:09 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.29.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E0A4CB25 for ; Mon, 10 Aug 2015 10:37:08 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.130.244] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1ZOkP4-0005f6-0t; Mon, 10 Aug 2015 12:34:02 +0200 Date: Mon, 10 Aug 2015 12:34:53 +0200 From: Fabian Keil To: Willem Jan Withagen Cc: freebsd-fs@freebsd.org Subject: Re: Urgently need some help solving lost space due to snapshots during receive op Message-ID: <60563a3f.3d0a7128@fabiankeil.de> In-Reply-To: <55C863DE.3000200@digiware.nl> References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/cdFuXofdsTrXI7cHWY/.W6B"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 10:37:09 -0000 --Sig_/cdFuXofdsTrXI7cHWY/.W6B Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Willem Jan Withagen wrote: > On 10-8-2015 06:42, Da Rock wrote: > > I'm trying to move a pool from one system to another - exact same hdd > > and config, different in other areas. Both register same space > > available, and so should be no issue. The old system is quite full - > > still at least 5% free though. > Are the versions of your FreeBSD also equal? >=20 > Because in newer version, the free space reservation is significantly > bigger. So if disks are equal size, then with newer FreeBSDs you have > less usable space available. > Don't know the exact SVN commit where it happened. But there are more > question on this topic in the list, and they all boil down to the same > thing: ZFS needs more reserved space to be able to do certain things > without freezing the system. >=20 > I've not read any suggestions that you can circumvent this setting. You can increase the usable space by increasing vfs.zfs.spa_slop_shift: fk@r500 ~ $sysctl vfs.zfs.spa_slop_shift vfs.zfs.spa_slop_shift: 6 fk@r500 ~ $zfs list tank NAME USED AVAIL REFER MOUNTPOINT tank 190G 34.4G 136K /tank fk@r500 ~ $sudo sysctl vfs.zfs.spa_slop_shift=3D7 vfs.zfs.spa_slop_shift: 6 -> 7 fk@r500 ~ $zfs list tank NAME USED AVAIL REFER MOUNTPOINT tank 190G 36.2G 136K /tank Fabian --Sig_/cdFuXofdsTrXI7cHWY/.W6B Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlXIfkoACgkQBYqIVf93VJ3EnACgjF1IaDtJFAX3qxTVaUI202lW qdIAn0VKfJtTGMPSDsW8ouj69sumVl+j =Rhh7 -----END PGP SIGNATURE----- --Sig_/cdFuXofdsTrXI7cHWY/.W6B-- From owner-freebsd-fs@freebsd.org Mon Aug 10 11:38:24 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0E7E99EDCF for ; Mon, 10 Aug 2015 11:38:24 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 822131FF for ; Mon, 10 Aug 2015 11:38:24 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 4BCBB153466; Mon, 10 Aug 2015 13:38:15 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BU_McFat9Nk2; Mon, 10 Aug 2015 13:37:56 +0200 (CEST) Received: from [IPv6:2001:4cb8:3:1:21eb:d0a9:8ec7:27cb] (unknown [IPv6:2001:4cb8:3:1:21eb:d0a9:8ec7:27cb]) by smtp.digiware.nl (Postfix) with ESMTP id 4CFD7153401; Mon, 10 Aug 2015 13:37:56 +0200 (CEST) Message-ID: <55C88D16.8090700@digiware.nl> Date: Mon, 10 Aug 2015 13:37:58 +0200 From: Willem Jan Withagen Organization: Digiware Management b.v. User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Fabian Keil CC: freebsd-fs@freebsd.org Subject: Re: Urgently need some help solving lost space due to snapshots during receive op References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> <60563a3f.3d0a7128@fabiankeil.de> In-Reply-To: <60563a3f.3d0a7128@fabiankeil.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 11:38:24 -0000 On 10-8-2015 12:34, Fabian Keil wrote: > Willem Jan Withagen wrote: > >> On 10-8-2015 06:42, Da Rock wrote: >>> I'm trying to move a pool from one system to another - exact same hdd >>> and config, different in other areas. Both register same space >>> available, and so should be no issue. The old system is quite full - >>> still at least 5% free though. > >> Are the versions of your FreeBSD also equal? >> >> Because in newer version, the free space reservation is significantly >> bigger. So if disks are equal size, then with newer FreeBSDs you have >> less usable space available. >> Don't know the exact SVN commit where it happened. But there are more >> question on this topic in the list, and they all boil down to the same >> thing: ZFS needs more reserved space to be able to do certain things >> without freezing the system. >> >> I've not read any suggestions that you can circumvent this setting. > > You can increase the usable space by increasing vfs.zfs.spa_slop_shift: > > fk@r500 ~ $sysctl vfs.zfs.spa_slop_shift > vfs.zfs.spa_slop_shift: 6 > fk@r500 ~ $zfs list tank > NAME USED AVAIL REFER MOUNTPOINT > tank 190G 34.4G 136K /tank > fk@r500 ~ $sudo sysctl vfs.zfs.spa_slop_shift=7 > vfs.zfs.spa_slop_shift: 6 -> 7 > fk@r500 ~ $zfs list tank > NAME USED AVAIL REFER MOUNTPOINT > tank 190G 36.2G 136K /tank Ah, Really nice info... Bet that that is going to "rescue" some of the migrating ZFS users. --WjW From owner-freebsd-fs@freebsd.org Mon Aug 10 13:01:56 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7808199D772 for ; Mon, 10 Aug 2015 13:01:56 +0000 (UTC) (envelope-from rcv-cmlja0BoYXZva21vbi5jb20AQDE0MzkyMTEzMTM=@vfemail.net) Received: from smtp101-5.vfemail.net (eightfive.vfemail.net [96.30.253.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 497A1848 for ; Mon, 10 Aug 2015 13:01:55 +0000 (UTC) (envelope-from rcv-cmlja0BoYXZva21vbi5jb20AQDE0MzkyMTEzMTM=@vfemail.net) Received: (qmail 30083 invoked by uid 89); 10 Aug 2015 12:55:13 -0000 Received: by simscan 1.4.0 ppid: 30076, pid: 30079, t: 0.0771s scanners:none Received: from unknown (HELO d3d3MTEwQDE0MzkyMTEzMTM=) (cmlja0BoYXZva21vbi5jb21AMTQzOTIxMTMxMw==@MTcyLjE2LjEwMC45MkAxNDM5MjExMzEz) by 172.16.100.61 with ESMTPA; 10 Aug 2015 12:55:13 -0000 Date: Mon, 10 Aug 2015 07:55:13 -0500 Message-ID: <20150810075513.Horde.dA37JPvUlgePL8j1izaLPg7@www.vfemail.net> From: Rick Romero To: freebsd-fs@freebsd.org Subject: Re: Urgently need some help solving lost space due to snapshots during receive op References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> <55C872FD.1060608@herveybayaustralia.com.au> In-Reply-To: <55C872FD.1060608@herveybayaustralia.com.au> User-Agent: Internet Messaging Program (IMP) H5 (6.2.2) X-VFEmail-Originating-IP: MTIuMzEuMTAwLjE0Ng== X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 13:01:56 -0000 Quoting Da Rock : > On 10/08/2015 18:42, Willem Jan Withagen wrote: >> On 10-8-2015 06:42, Da Rock wrote: >>> I'm trying to move a pool from one system to another - exact same hdd >>> and config, different in other areas. Both register same space >>> available, and so should be no issue. The old system is quite full - >>> still at least 5% free though. >> >> Hi, >> >> Are the versions of your FreeBSD also equal? > > No. 9.1 to 10.1. >> Because in newer version, the free space reservation is significantly >> bigger. So if disks are equal size, then with newer FreeBSDs you have >> less usable space available. >> Don't know the exact SVN commit where it happened. But there are more >> question on this topic in the list, and they all boil down to the same >> thing: ZFS needs more reserved space to be able to do certain things >> without freezing the system. >> >> I've not read any suggestions that you can circumvent this setting. > > Crap. So I guess my best will be to compress as best I can for the > moment or something like that. >> As a second note I think is is more or less general understanding that a >> ZFS systeem needs a lot more space left free than 5% to actual be able >> to perform. >> Last number I remember as sensible maximum fill level is around 70%. >> If you go over it, your system is going to be busy with ZFS bookkeeping >> instead of data storage. > > Yeah, I noticed that. Something I'm working on... backlog of sorting > activities really :) Once I'm done it'll drop to well below 50% I'd say. >> (One of the things I remember from the ZFS lecture by Kirk McKusik, >> which I no longer can find on YouTube :( ) >> >> --WjW > > Thanks for the info - I owe you a beer! At least I know why now and > eases my growing headache from banging my head on the brick wall. Also check out ashift.  I have hundreds of thousands of small files and the ashift change ate up 25% additional space. Rick From owner-freebsd-fs@freebsd.org Mon Aug 10 14:04:15 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22FED99E5B3 for ; Mon, 10 Aug 2015 14:04:15 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from cpanel61.fastdnsservers.com (server61.fastdnsservers.com [216.51.232.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 03627AAB for ; Mon, 10 Aug 2015 14:04:14 +0000 (UTC) (envelope-from paul@pk1048.com) Received: from mail.thecreativeadvantage.com ([96.236.20.34]:56403 helo=mbp-1.thecreativeadvantage.com) by cpanel61.fastdnsservers.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.85) (envelope-from ) id 1ZOngN-0028Ny-L3; Mon, 10 Aug 2015 09:04:07 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Urgently need some help solving lost space due to snapshots during receive op From: PK1048 In-Reply-To: <55C863DE.3000200@digiware.nl> Date: Mon, 10 Aug 2015 09:20:35 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <43B00232-46B7-4B96-B80C-D0111C6D8DA2@pk1048.com> References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> To: FreeBSD Filesystems X-Mailer: Apple Mail (2.1878.6) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cpanel61.fastdnsservers.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pk1048.com X-Get-Message-Sender-Via: cpanel61.fastdnsservers.com: authenticated_id: info@pk1048.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 14:04:15 -0000 On Aug 10, 2015, at 4:42, Willem Jan Withagen wrote: > Last number I remember as sensible maximum fill level is around 70%. > If you go over it, your system is going to be busy with ZFS = bookkeeping > instead of data storage. Fortunately or not, that number is _very_ dependent on your specific = data and how it is loaded. If it is static data, loaded once and not = modified, you can run a much higher % Used, for lots of changes on an = ongoing basis you need much more free space. The issue is that since ZFS = is Copy on Write, every write (even a modification to an existing file = or disk block) needs to find space. ZFS tries to reduce fragmentation. = As the zpool gets more and more full it takes longer and longer to find = the best possible location to put the new write. It also becomes harder = and harder to find contiguous blocks to put the new write into. The = existence of snapshots makes this problem even worse, as previously = written data is not cleared until the snapshot is destroyed. From owner-freebsd-fs@freebsd.org Mon Aug 10 19:41:27 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69B7399E559 for ; Mon, 10 Aug 2015 19:41:27 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3322BE38 for ; Mon, 10 Aug 2015 19:41:26 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id 41A2318FFA for ; Mon, 10 Aug 2015 21:41:23 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id 3A2407B888; Mon, 10 Aug 2015 21:41:23 +0200 (CEST) Date: Mon, 10 Aug 2015 21:41:23 +0200 From: Kristof Provost To: freebsd-fs@freebsd.org Subject: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302 Message-ID: <20150810194122.GE48727@vega.codepro.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 19:41:27 -0000 Hi, Booting CURRENT (r286582) earlier I saw the following panic: panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 302 cpuid = 1 Uptime: 41s ... (kgdb) #0 doadump (textdump=1) at pcpu.h:221 #1 0xffffffff80a19e65 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:329 #2 0xffffffff80a1a458 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:626 #3 0xffffffff80a1a4a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:557 #4 0xffffffff8242622a in assfail (a=, f=, l=) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 #5 0xffffffff8213f8be in vdev_queue_io (zio=0xfffff8000ede1000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:302 #6 0xffffffff82163406 in zio_vdev_io_start (zio=0xfffff8000ede1000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2774 #7 0xffffffff8215eaa9 in zio_execute (zio=0xfffff8000ede1000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1496 #8 0xffffffff8215de5b in zio_nowait (zio=0xfffff8000ede1000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1550 #9 0xffffffff8219f23c in trim_map_commit (spa=0xfffffe00032b7000, zio=0xfffff8000eda6760, vd=0xfffff8000e7dd800) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:476 #10 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, zio=0xfffff8000eda6760, vd=0xfffff8000e9b4800) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 #11 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, zio=0xfffff8000eda6760, vd=0xfffff8000e9b4000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 #12 0xffffffff8219ed9f in trim_thread (arg=0xfffffe00032b7000) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:579 #13 0xffffffff809e0374 in fork_exit ( callout=0xffffffff8219ecb0 , arg=0xfffffe00032b7000, frame=0xfffffe0239c09c00) at /usr/src/sys/kern/kern_fork.c:1006 #14 0xffffffff80e5e7ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:610 #15 0x0000000000000000 in ?? () This is a regular amd64 machine with 4 disks in raidz. I'd pulled one drive and replaced it with an empty one. It's possible that's related to the panic. Regards, Kristof From owner-freebsd-fs@freebsd.org Mon Aug 10 19:46:05 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A527C99E70F for ; Mon, 10 Aug 2015 19:46:05 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 89C821F8 for ; Mon, 10 Aug 2015 19:46:05 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 7E7AF1FAA for ; Mon, 10 Aug 2015 19:46:05 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 3A20D10060 for ; Mon, 10 Aug 2015 19:46:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id kyALYtf4hHyu for ; Mon, 10 Aug 2015 19:46:02 +0000 (UTC) Subject: Re: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302 DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com F2E1E1005A To: freebsd-fs@freebsd.org References: <20150810194122.GE48727@vega.codepro.be> From: Bryan Drewery Organization: FreeBSD Message-ID: <55C8FF76.7060202@FreeBSD.org> Date: Mon, 10 Aug 2015 12:45:58 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20150810194122.GE48727@vega.codepro.be> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 19:46:05 -0000 On 8/10/15 12:41 PM, Kristof Provost wrote: > Hi, > > Booting CURRENT (r286582) earlier I saw the following panic: > panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 302 > cpuid = 1 > Uptime: 41s > > ... > (kgdb) #0 doadump (textdump=1) at pcpu.h:221 > #1 0xffffffff80a19e65 in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:329 > #2 0xffffffff80a1a458 in vpanic (fmt=, > ap=) at /usr/src/sys/kern/kern_shutdown.c:626 > #3 0xffffffff80a1a4a3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:557 > #4 0xffffffff8242622a in assfail (a=, > f=, l=) > at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 > #5 0xffffffff8213f8be in vdev_queue_io (zio=0xfffff8000ede1000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:302 > #6 0xffffffff82163406 in zio_vdev_io_start (zio=0xfffff8000ede1000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2774 > #7 0xffffffff8215eaa9 in zio_execute (zio=0xfffff8000ede1000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1496 > #8 0xffffffff8215de5b in zio_nowait (zio=0xfffff8000ede1000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1550 > #9 0xffffffff8219f23c in trim_map_commit (spa=0xfffffe00032b7000, > zio=0xfffff8000eda6760, vd=0xfffff8000e7dd800) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:476 > #10 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, > zio=0xfffff8000eda6760, vd=0xfffff8000e9b4800) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 > #11 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, > zio=0xfffff8000eda6760, vd=0xfffff8000e9b4000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 > #12 0xffffffff8219ed9f in trim_thread (arg=0xfffffe00032b7000) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:579 > #13 0xffffffff809e0374 in fork_exit ( > callout=0xffffffff8219ecb0 , arg=0xfffffe00032b7000, > frame=0xfffffe0239c09c00) at /usr/src/sys/kern/kern_fork.c:1006 > #14 0xffffffff80e5e7ae in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:610 > #15 0x0000000000000000 in ?? () > > This is a regular amd64 machine with 4 disks in raidz. I'd pulled one drive and > replaced it with an empty one. It's possible that's related to the panic. > Same here... panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 302 cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe35545c6780 vpanic() at vpanic+0x189/frame 0xfffffe35545c6800 panic() at panic+0x43/frame 0xfffffe35545c6860 assfail() at assfail+0x1a/frame 0xfffffe35545c6870 vdev_queue_io() at vdev_queue_io+0x17e/frame 0xfffffe35545c68c0 zio_vdev_io_start() at zio_vdev_io_start+0x446/frame 0xfffffe35545c6920 zio_execute() at zio_execute+0x2e9/frame 0xfffffe35545c6970 zio_nowait() at zio_nowait+0x6b/frame 0xfffffe35545c6990 trim_map_commit() at trim_map_commit+0x2dc/frame 0xfffffe35545c6a30 trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6ad0 trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6b70 trim_thread() at trim_thread+0xef/frame 0xfffffe35545c6bb0 fork_exit() at fork_exit+0x84/frame 0xfffffe35545c6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe35545c6bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 6 tid 100684 ] Stopped at kdb_enter+0x3e: movq $0,kdb_why 2 disk mirror + an SSD log/cache. I have no way to debug it currently, rolling back is hard enough on this system. -- Regards, Bryan Drewery From owner-freebsd-fs@freebsd.org Mon Aug 10 20:01:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A40799EC4E for ; Mon, 10 Aug 2015 20:01:36 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 2658B225 for ; Mon, 10 Aug 2015 20:01:36 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 1AEBB1838 for ; Mon, 10 Aug 2015 20:01:36 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id B65231010C for ; Mon, 10 Aug 2015 20:01:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id Y3Z5O6wWydBf for ; Mon, 10 Aug 2015 20:01:33 +0000 (UTC) Subject: Re: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302 DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com CAF3710103 To: freebsd-fs@freebsd.org References: <20150810194122.GE48727@vega.codepro.be> <55C8FF76.7060202@FreeBSD.org> From: Bryan Drewery Organization: FreeBSD Message-ID: <55C9031C.60901@FreeBSD.org> Date: Mon, 10 Aug 2015 13:01:32 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55C8FF76.7060202@FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 20:01:36 -0000 On 8/10/15 12:45 PM, Bryan Drewery wrote: > On 8/10/15 12:41 PM, Kristof Provost wrote: >> Hi, >> >> Booting CURRENT (r286582) earlier I saw the following panic: >> panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, line: 302 >> cpuid = 1 >> Uptime: 41s >> >> ... >> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >> #1 0xffffffff80a19e65 in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:329 >> #2 0xffffffff80a1a458 in vpanic (fmt=, >> ap=) at /usr/src/sys/kern/kern_shutdown.c:626 >> #3 0xffffffff80a1a4a3 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:557 >> #4 0xffffffff8242622a in assfail (a=, >> f=, l=) >> at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 >> #5 0xffffffff8213f8be in vdev_queue_io (zio=0xfffff8000ede1000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:302 >> #6 0xffffffff82163406 in zio_vdev_io_start (zio=0xfffff8000ede1000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2774 >> #7 0xffffffff8215eaa9 in zio_execute (zio=0xfffff8000ede1000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1496 >> #8 0xffffffff8215de5b in zio_nowait (zio=0xfffff8000ede1000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1550 >> #9 0xffffffff8219f23c in trim_map_commit (spa=0xfffffe00032b7000, >> zio=0xfffff8000eda6760, vd=0xfffff8000e7dd800) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:476 >> #10 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, >> zio=0xfffff8000eda6760, vd=0xfffff8000e9b4800) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 >> #11 0xffffffff8219f046 in trim_map_commit (spa=0xfffffe00032b7000, >> zio=0xfffff8000eda6760, vd=0xfffff8000e9b4000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:532 >> #12 0xffffffff8219ed9f in trim_thread (arg=0xfffffe00032b7000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c:579 >> #13 0xffffffff809e0374 in fork_exit ( >> callout=0xffffffff8219ecb0 , arg=0xfffffe00032b7000, >> frame=0xfffffe0239c09c00) at /usr/src/sys/kern/kern_fork.c:1006 >> #14 0xffffffff80e5e7ae in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:610 >> #15 0x0000000000000000 in ?? () >> >> This is a regular amd64 machine with 4 disks in raidz. I'd pulled one drive and >> replaced it with an empty one. It's possible that's related to the panic. >> > > > Same here... > > panic: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c, > line: 302 > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe35545c6780 > vpanic() at vpanic+0x189/frame 0xfffffe35545c6800 > panic() at panic+0x43/frame 0xfffffe35545c6860 > assfail() at assfail+0x1a/frame 0xfffffe35545c6870 > vdev_queue_io() at vdev_queue_io+0x17e/frame 0xfffffe35545c68c0 > zio_vdev_io_start() at zio_vdev_io_start+0x446/frame 0xfffffe35545c6920 > zio_execute() at zio_execute+0x2e9/frame 0xfffffe35545c6970 > zio_nowait() at zio_nowait+0x6b/frame 0xfffffe35545c6990 > trim_map_commit() at trim_map_commit+0x2dc/frame 0xfffffe35545c6a30 > trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6ad0 > trim_map_commit() at trim_map_commit+0xe6/frame 0xfffffe35545c6b70 > trim_thread() at trim_thread+0xef/frame 0xfffffe35545c6bb0 > fork_exit() at fork_exit+0x84/frame 0xfffffe35545c6bf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe35545c6bf0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > KDB: enter: panic > [ thread pid 6 tid 100684 ] > Stopped at kdb_enter+0x3e: movq $0,kdb_why > > > 2 disk mirror + an SSD log/cache. > > I have no way to debug it currently, rolling back is hard enough on this > system. > > Booting with vfs.zfs.trim.enabled=0 seems to work fine as a workaround until mav@ has a fix. -- Regards, Bryan Drewery From owner-freebsd-fs@freebsd.org Mon Aug 10 20:38:39 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5EE699D541 for ; Mon, 10 Aug 2015 20:38:39 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6FDDE8E0; Mon, 10 Aug 2015 20:38:39 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by lbbsx3 with SMTP id sx3so18232811lbb.0; Mon, 10 Aug 2015 13:38:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:content-transfer-encoding; bh=e9aYHKErVgXYqavFS+BadX3rAaivCQrS8qXAM3sZf5E=; b=Hn5C5tFqkO2XBEE2NzKROp7Q1VYeQiAfOHTY6jjSfWXgqwKobNnROXkGg6EGIKsyqA Lot2Arl766ryyGQ1Zbzrp6iT5GKqvuAa623WKKgLlRK+UrrXyMqUyFbPs8266g5XJqKG gz0i+zQXy+Elatnqi6rRnkJzDKpOgT7D3+nAYINCPQUHlA+89S970n54XyomAyHKJJlR WUgFv3wdltSUNSmagG10czs11L9QWJl6u7D8SqMON1yElfjp9upUxDjZ8ShXUCIY0zgT RUTUtVFmTcS+Uw+13HKBxqYCbxmW+EQWvra9W6Y0m/1/ZFI3vbyMTHxh0hy0aBvfBq6j W4iw== X-Received: by 10.152.4.98 with SMTP id j2mr21585551laj.67.1439239117571; Mon, 10 Aug 2015 13:38:37 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([134.249.139.101]) by smtp.googlemail.com with ESMTPSA id z1sm4487530lbj.11.2015.08.10.13.38.35 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Aug 2015 13:38:36 -0700 (PDT) Sender: Alexander Motin Message-ID: <55C90BCA.4080106@FreeBSD.org> Date: Mon, 10 Aug 2015 23:38:34 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Kristof Provost CC: freebsd-fs@FreeBSD.org Subject: Re: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 20:38:40 -0000 Hi. r286593 should fix the issue. I'm sorry. -- Alexander Motin From owner-freebsd-fs@freebsd.org Mon Aug 10 21:54:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE5D399EB52 for ; Mon, 10 Aug 2015 21:54:58 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 771211B7; Mon, 10 Aug 2015 21:54:58 +0000 (UTC) (envelope-from kp@vega.codepro.be) Received: from vega.codepro.be (unknown [172.16.1.3]) by venus.codepro.be (Postfix) with ESMTP id 92AC4B075; Mon, 10 Aug 2015 23:54:53 +0200 (CEST) Received: by vega.codepro.be (Postfix, from userid 1001) id 7D2D57BCDD; Mon, 10 Aug 2015 23:54:53 +0200 (CEST) Date: Mon, 10 Aug 2015 23:54:53 +0200 From: Kristof Provost To: Alexander Motin Cc: freebsd-fs@FreeBSD.org Subject: Re: solaris assert: t == ZIO_TYPE_READ || t == ZIO_TYPE_WRITE, file: ...zfs/vdev_queue.c, line: 302 Message-ID: <20150810215453.GF48727@vega.codepro.be> References: <55C90BCA.4080106@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <55C90BCA.4080106@FreeBSD.org> X-Checked-By-NSA: Probably User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Aug 2015 21:54:58 -0000 On 2015-08-10 23:38:34 (+0300), Alexander Motin wrote: > r286593 should fix the issue. I'm sorry. > The system is now happily running r286593. Looks good! Thanks, Kristof From owner-freebsd-fs@freebsd.org Tue Aug 11 10:19:30 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 388AD99EEC0 for ; Tue, 11 Aug 2015 10:19:30 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from mail.unitedinsong.com.au (mail.unitedinsong.com.au [150.101.178.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E53351D25 for ; Tue, 11 Aug 2015 10:19:29 +0000 (UTC) (envelope-from freebsd-fs@herveybayaustralia.com.au) Received: from [192.168.0.170] (laptop1.herveybayaustralia.com.au [192.168.0.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.unitedinsong.com.au (Postfix) with ESMTPSA id 9EAB961F85 for ; Tue, 11 Aug 2015 20:19:17 +1000 (EST) Subject: Re: Urgently need some help solving lost space due to snapshots during receive op To: freebsd-fs@freebsd.org References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> <60563a3f.3d0a7128@fabiankeil.de> From: Da Rock Message-ID: <55C9CC22.7060802@herveybayaustralia.com.au> Date: Tue, 11 Aug 2015 20:19:14 +1000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <60563a3f.3d0a7128@fabiankeil.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 10:19:30 -0000 On 10/08/2015 20:34, Fabian Keil wrote: > Willem Jan Withagen wrote: > >> On 10-8-2015 06:42, Da Rock wrote: >>> I'm trying to move a pool from one system to another - exact same hdd >>> and config, different in other areas. Both register same space >>> available, and so should be no issue. The old system is quite full - >>> still at least 5% free though. >> Are the versions of your FreeBSD also equal? >> >> Because in newer version, the free space reservation is significantly >> bigger. So if disks are equal size, then with newer FreeBSDs you have >> less usable space available. >> Don't know the exact SVN commit where it happened. But there are more >> question on this topic in the list, and they all boil down to the same >> thing: ZFS needs more reserved space to be able to do certain things >> without freezing the system. >> >> I've not read any suggestions that you can circumvent this setting. > You can increase the usable space by increasing vfs.zfs.spa_slop_shift: > > fk@r500 ~ $sysctl vfs.zfs.spa_slop_shift > vfs.zfs.spa_slop_shift: 6 > fk@r500 ~ $zfs list tank > NAME USED AVAIL REFER MOUNTPOINT > tank 190G 34.4G 136K /tank > fk@r500 ~ $sudo sysctl vfs.zfs.spa_slop_shift=7 > vfs.zfs.spa_slop_shift: 6 -> 7 > fk@r500 ~ $zfs list tank > NAME USED AVAIL REFER MOUNTPOINT > tank 190G 36.2G 136K /tank > > Fabian What version is that on? I don't have the OID. From owner-freebsd-fs@freebsd.org Tue Aug 11 10:42:14 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19E0D99E696 for ; Tue, 11 Aug 2015 10:42:14 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay04.ispgateway.de (smtprelay04.ispgateway.de [80.67.31.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCD26A60 for ; Tue, 11 Aug 2015 10:42:13 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [84.44.152.119] (helo=fabiankeil.de) by smtprelay04.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1ZP6xQ-0004KC-V7; Tue, 11 Aug 2015 12:39:01 +0200 Date: Tue, 11 Aug 2015 12:38:58 +0200 From: Fabian Keil To: Da Rock Cc: freebsd-fs@freebsd.org Subject: Re: Urgently need some help solving lost space due to snapshots during receive op Message-ID: <12ccbf2f.401bf1f3@fabiankeil.de> In-Reply-To: <55C9CC22.7060802@herveybayaustralia.com.au> References: <55C82BCD.2050404@herveybayaustralia.com.au> <55C863DE.3000200@digiware.nl> <60563a3f.3d0a7128@fabiankeil.de> <55C9CC22.7060802@herveybayaustralia.com.au> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/31b=w1ZlrYj=7m7F8sxH8K2"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 10:42:14 -0000 --Sig_/31b=w1ZlrYj=7m7F8sxH8K2 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Da Rock wrote: > On 10/08/2015 20:34, Fabian Keil wrote: > > Willem Jan Withagen wrote: > > > >> On 10-8-2015 06:42, Da Rock wrote: > >>> I'm trying to move a pool from one system to another - exact same hdd > >>> and config, different in other areas. Both register same space > >>> available, and so should be no issue. The old system is quite full - > >>> still at least 5% free though. > >> Are the versions of your FreeBSD also equal? > >> > >> Because in newer version, the free space reservation is significantly > >> bigger. So if disks are equal size, then with newer FreeBSDs you have > >> less usable space available. > >> Don't know the exact SVN commit where it happened. But there are more > >> question on this topic in the list, and they all boil down to the same > >> thing: ZFS needs more reserved space to be able to do certain things > >> without freezing the system. > >> > >> I've not read any suggestions that you can circumvent this setting. > > You can increase the usable space by increasing vfs.zfs.spa_slop_shift: > > > > fk@r500 ~ $sysctl vfs.zfs.spa_slop_shift > > vfs.zfs.spa_slop_shift: 6 > > fk@r500 ~ $zfs list tank > > NAME USED AVAIL REFER MOUNTPOINT > > tank 190G 34.4G 136K /tank > > fk@r500 ~ $sudo sysctl vfs.zfs.spa_slop_shift=3D7 > > vfs.zfs.spa_slop_shift: 6 -> 7 > > fk@r500 ~ $zfs list tank > > NAME USED AVAIL REFER MOUNTPOINT > > tank 190G 36.2G 136K /tank > What version is that on? I don't have the OID. It's available in 11-CURRENT (obviously), 10-STABLE and will be in the 10.2 release: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D275490 Fabian --Sig_/31b=w1ZlrYj=7m7F8sxH8K2 Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlXJ0LIACgkQBYqIVf93VJ0PLgCcC9VU0vNNVmELJWjq0DSMcXY7 470AoLWZlMIsjZU7rsH4Me8IRfAPUdsj =5JGF -----END PGP SIGNATURE----- --Sig_/31b=w1ZlrYj=7m7F8sxH8K2-- From owner-freebsd-fs@freebsd.org Tue Aug 11 15:38:46 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3085299F51A for ; Tue, 11 Aug 2015 15:38:46 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC441E6 for ; Tue, 11 Aug 2015 15:38:45 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by ioeg141 with SMTP id g141so205996624ioe.3 for ; Tue, 11 Aug 2015 08:38:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=zCmEPHDmpLhHxZio7D0+7ivdbFdj2GoQhRBVI8iS0og=; b=ag4p5/SnBHedYEu5VLouWl1cmLJpp7JnpiUZxPncJS1rTomVBYcNqFhC4/TDkTq11Q TmlgvTuRz0BUPy/065dixSTMUcGoF5viyGVeviB/EW2vubaN8G8E9XZykiR8YjuE6ZzI VZQcWgEtTIV6p7x9kXiCvAAPg5/mL97DtegCVF10kxCC2ZWPUVygHXy13ZhybdM4yKXL DT2WYHOczTHMlHZEx7wyEBF4VU28yKRk6RlxL03oCz7dle4F4/g/P3FkqKY5McDh8qcf YQL8q6NmHRuwS3+kBqG9EbgrS+eHzGXZzUphAXUvWxXH+BC672ffWI8N9VlpAnFe4K91 6k9g== MIME-Version: 1.0 X-Received: by 10.107.131.168 with SMTP id n40mr31182572ioi.47.1439307525223; Tue, 11 Aug 2015 08:38:45 -0700 (PDT) Received: by 10.36.34.77 with HTTP; Tue, 11 Aug 2015 08:38:45 -0700 (PDT) In-Reply-To: <55A6C4BD.5090500@denninger.net> References: <55A3A800.5060904@denninger.net> <55A4D5B7.2030603@freebsd.org> <55A4E5AB.8060909@netlabs.org> <1436989410.1427298.324703241.421E814B@webmail.messagingengine.com> <55A6C0A4.1030300@digiware.nl> <55A6C4BD.5090500@denninger.net> Date: Tue, 11 Aug 2015 12:38:45 -0300 Message-ID: Subject: Re: FreeBSD 10.1 Memory Exhaustion From: Christopher Forgeron To: Karl Denninger Cc: FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2015 15:38:46 -0000 Hello All, Any reports on success / issues with the patch? I'm reporting a good increase in uptime and stability. I have two production machines with heavy load running, and I've made it almost a month without a crash on one machine. The other had a swap-related crash that wasn't really a fault/issue of the patch, but even with that one crash, it's been more stable than a non-patched system. I'm curious to everyone else's findings. On Wed, Jul 15, 2015 at 5:38 PM, Karl Denninger wrote: > > On 7/15/2015 15:20, Willem Jan Withagen wrote: > > On 15/07/2015 21:43, Mark Felder wrote: > >> > >> On Tue, Jul 14, 2015, at 10:10, Sean Chittenden wrote: > >>> I think the reason this is not seen more often is because people > >>> frequently > >>> throw limits on the arc in /boot/loader.conf: > >>> > >>> vfs.zfs.arc_min="18G" > >>> vfs.zfs.arc_max="149G" > >>> > >>> ZFS ARC *should* not require those settings, but does currently for > mixed > >>> workloads (i.e. databases) in order to be "stable". By setting fixed > >>> sizes > >>> on the ARC, UMA and ARC are much more cooperative in that they have > their > >>> own memory regions to manage so this behavior is not seen as often. > >>> > >>> To be clear, however, it should not be necessary to set parameters like > >>> these in /boot/loader.conf in order to obtain consistent operational > >>> behavior. I'd be curious to know if someone running 10.2 BETA without > >>> patches is able to trigger this behavior or not. There was work done > >>> that > >>> reported helped with this between 10.1 and now. To what extent it > >>> helped, > >>> however, I don't have any advice yet. > >>> > >> I was about to email "I have 12TB at home and 4GB of RAM with a very > >> erratic workload and never run into any issues" and then I looked at > >> /boot/loader.conf and saw vfs.zfs.arc_max="2G" > >> > >> Now I'm too scared to turn it off... :-) > > Same here. > > Just a leftover of all the advise to limit arc in the past. > > Just bit the bullit: installed BETA1, killed the settings and rebooted. > > We'll see what comes of it. > > > > --WjW > If you get bit I have refactored the patch for 10.2-BETA1 to get rid of > the fudges and i386 linker problem and uploaded it, so if you run into > trouble try applying that and see if that fixes it. > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ > From owner-freebsd-fs@freebsd.org Fri Aug 14 10:17:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 747979B8D25 for ; Fri, 14 Aug 2015 10:17:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 605C31319 for ; Fri, 14 Aug 2015 10:17:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7EAHKFU074710 for ; Fri, 14 Aug 2015 10:17:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 161424] [nullfs] __getcwd() calls fail when used on nullfs mount Date: Fri, 14 Aug 2015 10:17:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 8.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: fullermd@over-yonder.net X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 10:17:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=161424 fullermd@over-yonder.net changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fullermd@over-yonder.net --- Comment #9 from fullermd@over-yonder.net --- I've just run into what looks like this (after lots of tracking and puzzled-looking) myself on a stable/10 system (in a jail, but it happens outside of jails too). What's happening is that, when you're on a nullfs, if you don't have read permissions to all the directories in the path up to the root, it blows up, whereas on a regular FS, it does just fine. e.g., I have a "/thome/matt/tmp/wt". /thome and /thome/matt are root:wheel 751; the tmp/wt dirs are 755. I use www as a convenient user, changing its shell to /bin/sh. % (cd /thome/matt/tmp/wt ; su www -c /bin/pwd) /thome/matt/tmp/wt Yep, that's what it should get. So then I null mount /thome off into /tmp: % mount_nullfs /thome /tmp/th/ Then: % ( cd /tmp/th/matt/tmp/wt ; su www -c /bin/pwd ) pwd: .: Permission denied If I chmod 755 /thome and /thome/matt, then % ( cd /tmp/th/matt/tmp/wt ; su www -c /bin/pwd ) /tmp/th/matt/tmp/wt -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Aug 14 13:01:17 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9F349B8648 for ; Fri, 14 Aug 2015 13:01:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D6C3F1690 for ; Fri, 14 Aug 2015 13:01:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t7ED1HqZ027591 for ; Fri, 14 Aug 2015 13:01:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 161424] [nullfs] __getcwd() calls fail when used on nullfs mount Date: Fri, 14 Aug 2015 13:01:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 8.2-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 13:01:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=161424 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kib@FreeBSD.org --- Comment #10 from Konstantin Belousov --- (In reply to fullermd from comment #9) What you described is the expected behaviour, it is just a situation that it always happens for nullfs, but only sporadically for other filesystems. More precisely, nullfs does not use namecache for good reasons (we cannot provide cache consistency between nullfs entries and lower filesystems without some drastic measures). If a filesystem uses namecache, then getcwd call first tries to resolve the path using namecache, and only when the cache failed to provide an entry, it falls down to read the ".." directory and searching the child entry name by inode number. So typically for fs like ufs or zfs, the reading of ".." does not happen. On the other hand, since nullfs does not use namecache, ".." is always scanned and there the directory permissions starts to play. So the fix, if we ever would change the observed behaviour, is to make getcwd to honor the directory permissions even when the request is satisfied by the namecache, making failure deterministic. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Aug 14 13:32:49 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D6B29B8E40 for ; Fri, 14 Aug 2015 13:32:49 +0000 (UTC) (envelope-from webmailservice@update.com) Received: from mailserver.ikf.co.in (mailserver.ikf.co.in [188.138.8.83]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "localhost", Issuer "www.qmailtoaster.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F30A01C03 for ; Fri, 14 Aug 2015 13:32:47 +0000 (UTC) (envelope-from webmailservice@update.com) Received: (qmail 9555 invoked by uid 89); 14 Aug 2015 13:26:03 -0000 Received: by simscan 1.4.0 ppid: 9421, pid: 9551, t: 0.1191s scanners: attach: 1.4.0 clamav: 0.97.5/m:55/d:20788 Received: from unknown (HELO ?198.12.85.165?) (systems.bangalore@surinauto.com@198.12.85.165) by mailserver.ikf.co.in with ESMTPA; 14 Aug 2015 13:26:03 -0000 MIME-Version: 1.0 Subject: Email Update To: freebsd-fs@FreeBSD.org From: "Mail Team" Date: Fri, 14 Aug 2015 06:25:18 -0700 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Description: Mail message body X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 13:32:49 -0000 Hi freebsd-fs@FreeBSD.org = You are running very low of data volume (95% Storage Low). = Avoid account malfunction, and retrieve pending mails from clicking on the= below link = ALLOCATE MORE DATA TO [freebsd-fs@FreeBSD.org] = We will not be responsible for any mail malfunction or account locked up i= f after this warning no response from you. = Mail Team From owner-freebsd-fs@freebsd.org Fri Aug 14 14:46:27 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73EB39B934B; Fri, 14 Aug 2015 14:46:27 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FD1319F5; Fri, 14 Aug 2015 14:46:26 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-227-250.lns20.per1.internode.on.net [121.45.227.250]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id t7EEkGBM008245 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 14 Aug 2015 07:46:19 -0700 (PDT) (envelope-from julian@freebsd.org) To: freebsd-current , "freebsd-fs@freebsd.org" From: Julian Elischer Subject: futimens and utimensat vs birthtime Message-ID: <55CDFF32.7050601@freebsd.org> Date: Fri, 14 Aug 2015 22:46:10 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 14:46:27 -0000 So, currently the method of setting birthtime on a file is an awkward combination of tricks. These break down in a number of situations, but the one I'm specifically interested in is as follows: windows robocopy running through samba, onto a ZFS filesystem. (may happen for UFS2 as well.) In order to mark a file as 'incomplete' during transfer, windows sets the modification time to Jan 1, 1980. This triggers code in our system to set the birthtime to Jan 1 1980. it then, on completion of the file sets the modification and birth times to the correct values. This fails becasue you can not make a birthtime later than what it is already set to. As the Man page for utimensat() says: "Ideally a new system call will be added that allows the setting of all three times at once." I would like to implement this call. but would like input as to it's nature. The code inside the system would already appear to support handling three elements, though it needs some scrutiny, so all that is needed is a system call with the ability to set the birthtime directly. Whether it should take the form of the existing calls but expecting three items is up for discussion. Maybe teh addition of a flags argument to specify which items are present and which to set. ideas? Julian From owner-freebsd-fs@freebsd.org Fri Aug 14 18:26:01 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADAF39B92E1; Fri, 14 Aug 2015 18:26:01 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6C72117D2; Fri, 14 Aug 2015 18:26:01 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (75-48-78-19.lightspeed.cncrca.sbcglobal.net [75.48.78.19]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 483F3B97B; Fri, 14 Aug 2015 14:26:00 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Cc: Julian Elischer , "freebsd-fs@freebsd.org" , 'Jilles Tjoelker' Subject: Re: futimens and utimensat vs birthtime Date: Fri, 14 Aug 2015 10:39:41 -0700 Message-ID: <2405496.WdPSxGzEuT@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-PRERELEASE; KDE/4.14.3; amd64; ; ) In-Reply-To: <55CDFF32.7050601@freebsd.org> References: <55CDFF32.7050601@freebsd.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 14 Aug 2015 14:26:00 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 18:26:01 -0000 On Friday, August 14, 2015 10:46:10 PM Julian Elischer wrote: > I would like to implement this call. but would like input as to it's > nature. > The code inside the system would already appear to support handling > three elements, though it needs some scrutiny, > so all that is needed is a system call with the ability to set the > birthtime directly. > > Whether it should take the form of the existing calls but expecting > three items is up for discussion. > Maybe teh addition of a flags argument to specify which items are > present and which to set. > > ideas? I believe these should be new calls. Only utimensat() provides a flag argument, but it is reserved for AT_* flags. I would be fine with something like futimens3() and utimensat3() (where 3 means "three timespecs"). Jilles implemented futimens() and utimensat(), so he might have ideas as well. I would probably stick the birth time in the third (final) timespec slot to make it easier to update new code (you can use an #ifdef just around ts[2] without having to #ifdef the entire block). -- John Baldwin From owner-freebsd-fs@freebsd.org Fri Aug 14 20:07:12 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E91559BA69E; Fri, 14 Aug 2015 20:07:12 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A87981239; Fri, 14 Aug 2015 20:07:12 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 58BD1B809B; Fri, 14 Aug 2015 22:07:09 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 413DD28494; Fri, 14 Aug 2015 22:07:09 +0200 (CEST) Date: Fri, 14 Aug 2015 22:07:09 +0200 From: Jilles Tjoelker To: John Baldwin Cc: freebsd-current@freebsd.org, Julian Elischer , "freebsd-fs@freebsd.org" Subject: Re: futimens and utimensat vs birthtime Message-ID: <20150814200709.GB88901@stack.nl> References: <55CDFF32.7050601@freebsd.org> <2405496.WdPSxGzEuT@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2405496.WdPSxGzEuT@ralph.baldwin.cx> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 20:07:13 -0000 On Fri, Aug 14, 2015 at 10:39:41AM -0700, John Baldwin wrote: > On Friday, August 14, 2015 10:46:10 PM Julian Elischer wrote: > > I would like to implement this call. but would like input as to it's > > nature. > > The code inside the system would already appear to support handling > > three elements, though it needs some scrutiny, > > so all that is needed is a system call with the ability to set the > > birthtime directly. > > Whether it should take the form of the existing calls but expecting > > three items is up for discussion. > > Maybe teh addition of a flags argument to specify which items are > > present and which to set. > > ideas? > I believe these should be new calls. Only utimensat() provides a flag > argument, but it is reserved for AT_* flags. I would be fine with > something like futimens3() and utimensat3() (where 3 means "three > timespecs"). Jilles implemented futimens() and utimensat(), so he > might have ideas as well. I would probably stick the birth time in > the third (final) timespec slot to make it easier to update new code > (you can use an #ifdef just around ts[2] without having to #ifdef the > entire block). Without adding new syscalls, it is possible to use the first tv_nsec to indicate that a new birth time is present. In that case, times[0].tv_nsec == UTIME_WITHBIRTHTIME would indicate that times has 4 instead of 2 elements. Whether you want to do this instead of adding two more system calls is a different question. Also note that, in some sense, the inability to set the birthtime forward is a feature. -- Jilles Tjoelker From owner-freebsd-fs@freebsd.org Fri Aug 14 21:22:27 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45AD79B936D for ; Fri, 14 Aug 2015 21:22:27 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [IPv6:2001:5a8:4:7e72:d250:99ff:fe57:4030]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22D6919F7; Fri, 14 Aug 2015 21:22:27 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.14.9/8.14.9) with ESMTP id t7ELMPjR002452; Fri, 14 Aug 2015 14:22:25 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <201508142122.t7ELMPjR002452@chez.mckusick.com> From: Kirk McKusick To: Julian Elischer cc: "freebsd-fs@freebsd.org" , "'Jilles Tjoelker'" , John Baldwin Subject: Re: futimens and utimensat vs birthtime In-reply-to: <2405496.WdPSxGzEuT@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <2450.1439587345.1@chez.mckusick.com> Date: Fri, 14 Aug 2015 14:22:25 -0700 X-Spam-Status: No, score=0.1 required=5.0 tests=MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Aug 2015 21:22:27 -0000 > From: John Baldwin > To: freebsd-current@freebsd.org > Subject: Re: futimens and utimensat vs birthtime > Date: Fri, 14 Aug 2015 10:39:41 -0700 > Cc: "freebsd-fs@freebsd.org" , > "'Jilles Tjoelker'" > > On Friday, August 14, 2015 10:46:10 PM Julian Elischer wrote: >> I would like to implement this call. but would like input as to it's >> nature. >> The code inside the system would already appear to support handling >> three elements, though it needs some scrutiny, >> so all that is needed is a system call with the ability to set the >> birthtime directly. >> >> Whether it should take the form of the existing calls but expecting >> three items is up for discussion. >> Maybe teh addition of a flags argument to specify which items are >> present and which to set. >> >> ideas? > > I believe these should be new calls. Only utimensat() provides a flag > argument, but it is reserved for AT_* flags. I would be fine with > something like futimens3() and utimensat3() (where 3 means "three > timespecs"). Jilles implemented futimens() and utimensat(), so he > might have ideas as well. I would probably stick the birth time in > the third (final) timespec slot to make it easier to update new code > (you can use an #ifdef just around ts[2] without having to #ifdef the > entire block). > > -- > John Baldwin I concur with John's suggestion. Add a new system call with three argument set of times specifying birthtime as the last one. I proposed doing this when I added birthtime, but did not as the sentiment at the time was that it would gratuitously make FreeBSD written applications less portable if they used this new non-standard system call. Kirk McKusick From owner-freebsd-fs@freebsd.org Sat Aug 15 18:01:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2735A9BA1F4 for ; Sat, 15 Aug 2015 18:01:19 +0000 (UTC) (envelope-from karl@denninger.net) Received: from fs.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "NewFS.denninger.net", Issuer "NewFS.denninger.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DC8C79C9 for ; Sat, 15 Aug 2015 18:01:18 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (localhost [127.0.0.1]) by fs.denninger.net (8.15.2/8.14.8) with ESMTP id t7FHd0sJ044744 for ; Sat, 15 Aug 2015 12:39:00 -0500 (CDT) (envelope-from karl@denninger.net) Received: from [192.168.1.40] [192.168.1.40] (Via SSLv3 AES128-SHA) ; by Spamblock-sys (LOCAL/AUTH) Sat Aug 15 12:39:00 2015 Message-ID: <55CF7926.1030901@denninger.net> Date: Sat, 15 Aug 2015 12:38:46 -0500 From: Karl Denninger User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) References: <55BB443E.8040801@denninger.net> In-Reply-To: <55BB443E.8040801@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms060503000503080205070604" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Aug 2015 18:01:19 -0000 This is a cryptographically signed message in MIME format. --------------ms060503000503080205070604 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Update: This /appears /to be related to attempting to send or receive a /cloned /snapshot. I use /beadm /to manage boot environments and the crashes have all come while send/recv-ing the root pool, which is the one where these clones get created. It is /not /consistent within a given snapshot when it crashes and a second attempt (which does a "recovery" send/receive) succeeds every time -- I've yet to have it panic twice sequentially. I surmise that the problem comes about when a file in the cloned snapshot is modified, but this is a guess at this point. I'm going to try to force replication of the problem on my test system. On 7/31/2015 04:47, Karl Denninger wrote: > I have an automated script that runs zfs send/recv copies to bring a > backup data set into congruence with the running copies nightly. The > source has automated snapshots running on a fairly frequent basis > through zfs-auto-snapshot. > > Recently I have started having a panic show up about once a week during= > the backup run, but it's inconsistent. It is in the same place, but I > cannot force it to repeat. > > The trap itself is a page fault in kernel mode in the zfs code at > zfs_unmount_snap(); here's the traceback from the kvm (sorry for the > image link but I don't have a better option right now.) > > I'll try to get a dump, this is a production machine with encrypted swa= p > so it's not normally turned on. > > Note that the pool that appears to be involved (the backup pool) has > passed a scrub and thus I would assume the on-disk structure is ok.....= > but that might be an unfair assumption. It is always occurring in the > same dataset although there are a half-dozen that are sync'd -- if this= > one (the first one) successfully completes during the run then all the > rest will as well (that is, whenever I restart the process it has alway= s > failed here.) The source pool is also clean and passes a scrub. > > traceback is at http://www.denninger.net/kvmimage.png; apologies for th= e > image traceback but this is coming from a remote KVM. > > I first saw this on 10.1-STABLE and it is still happening on FreeBSD > 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to see if= > the problem was something that had been addressed. > > --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060503000503080205070604 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGXzCC BlswggRDoAMCAQICASkwDQYJKoZIhvcNAQELBQAwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQI EwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEg U3lzdGVtcyBMTEMgQ0EwHhcNMTUwNDIxMDIyMTU5WhcNMjAwNDE5MDIyMTU5WjBaMQswCQYD VQQGEwJVUzEQMA4GA1UECBMHRmxvcmlkYTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1zIExMQzEe MBwGA1UEAxMVS2FybCBEZW5uaW5nZXIgKE9DU1ApMIICIjANBgkqhkiG9w0BAQEFAAOCAg8A MIICCgKCAgEAuYRY+EB2mGtZ3grlVO8TmnEvduVFA/IYXcCmNSOC1q+pTVjylsjcHKBcOPb9 TP1KLxdWP+Q1soSORGHlKw2/HcVzShDW5WPIKrvML+Ry0XvIvNBu9adTiCsA9nci4Cnf98XE hVpenER0qbJkBUOGT1rP4iAcfjet0lEgzPEnm+pAxv6fYSNp1WqIY9u0b1pkQiaWrt8hgNOc rJOiLbc8CeQ/DBP6rUiQjYNO9/aPNauEtHkNNfR9RgLSfGUdZuOCmJqnIla1HsrZhA5p69Bv /e832BKiNPaH5wF6btAiPpTr2sRhwQO8/IIxcRX1Vxd1yZbjYtJGw+9lwEcWRYAmoxkzKLPi S6Zo/6z5wgNpeK1H+zOioMoZIczgI8BlX1iHxqy/FAvm4PHPnC8s+BLnJLwr+jvMNHm82QwL J9hC5Ho8AnFU6TkCuq+P2V8/clJVqnBuvTUKhYMGSm4mUp+lAgR4L+lwIEqSeWVsxirIcE7Z OKkvI7k5x3WeE3+c6w74L6PfWVAd84xFlo9DKRdU9YbkFuFZPu21fi/LmE5brImB5P+jdqnK eWnVwRq+RBFLy4kehCzMXooitAwgP8l/JJa9VDiSyd/PAHaVGiat2vCdDh4b8cFL7SV6jPA4 k0MgGUA/6Et7wDmhZmCigggr9K6VQCx8jpKB3x1NlNNiaWECAwEAAaOB9DCB8TA3BggrBgEF BQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDALBgNVHQ8EBAMCBeAwLAYJYIZIAYb4QgENBB8W HU9wZW5TU0wgR2VuZXJhdGVkIENlcnRpZmljYXRlMB0GA1UdDgQWBBTFHJQt6cloXBdG1Pv1 o2YgH+7lWTAfBgNVHSMEGDAWgBQkcZudhX383d29sMqSlAOh+tNtNTAdBgNVHREEFjAUgRJr YXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAE9/dxi2YqjCYYhiybp4GKcm 7tBVa/GLW+qcHPcoT4dqmqghlLz8+iUH+HCJjRQATVGyMEnvISOKFVHC6aZIG+Sg7J8bfS4+ fjKDi9smRH2VPPx3bV8+yFYRNroMGHaPHZB/Xctmmvc+PZ9O2W7rExgrODtxIOB3Zs6wkYf+ ty+9r1KmTHlV+rRHI6timH1uiyFE3cPi1taAEBxf0851cJV8k40PGF8G48ewnq8SY9sCf5cv liXbpdgU+I4ND5BuTjg63WS32zuhLd1VSuH3ZC/QbcncMX5W3oLXmcQP5/5uTiBJy74kdPtG MSZ9rXwZPwNxP/8PXMSR7ViaFvjUkf4bJlyENFa2PGxLk4EUzOuO7t3brjMlQW1fuInfG+ko 3tVxko20Hp0tKGPe/9cOxBVBZeZH/VgpZn3cLculGzZjmdh2fqAQ6kv9Z9AVOG1+dq0c1zt8 2zm+Oi1pikGXkfz5UJq60psY6zbX25BuEZkthO/qiS4pxjxb7gQkS0rTEHTy+qv0l3QVL0wa NAT74Zaj7l5DEW3qdQQ0dtVieyvptg9CxkfQJE3JyBMb0zBj9Qhc5/hbTfhSlHzZMEbUuIyx h9vxqFAmGzfB1/WfOKkiNHChkpPW8ZeH9yPeDBKvrgZ96dREHFoVkDk7Vpw5lSM+tFOfdyLg xxhb/RZVUDeUMYIE4zCCBN8CAQEwgZYwgZAxCzAJBgNVBAYTAlVTMRAwDgYDVQQIEwdGbG9y aWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBMTEMxHDAa BgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG9w0BCQEWE0N1ZGEgU3lzdGVt cyBMTEMgQ0ECASkwCQYFKw4DAhoFAKCCAiEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTUwODE1MTczODQ2WjAjBgkqhkiG9w0BCQQxFgQUsj301L0lWilS XmM0jDEJSHKqQh8wbAYJKoZIhvcNAQkPMV8wXTALBglghkgBZQMEASowCwYJYIZIAWUDBAEC MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN BggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBAgEpMIGpBgsqhkiG9w0BCRACCzGBmaCBljCBkDELMAkGA1UE BhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQ Q3VkYSBTeXN0ZW1zIExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqG SIb3DQEJARYTQ3VkYSBTeXN0ZW1zIExMQyBDQQIBKTANBgkqhkiG9w0BAQEFAASCAgAeJoPD 8I8TU6f4UIYY5o9vfFQyoUIY7wBCQWN6TzFNTmg4vVnG2MPDz6iGL1JvY0avvPS2fdpgby2G 0iGQTzuivKXW3tqIRfE3yDocV/9ruJPz92ji6B6Omj7uostdpafD/SdHKuV4ZdwmtmVByP8O 9d7jPD6S1uSw0gj8NpgCZ/Xx4pgaVN6WqIOr7F+6rwTTK+A2zydDsy94qTAfSoHThTd6XN/i 4g+Hl8kwPADBCHcfRIZy9v8Qm7MHcoVYUpeeY+530L9usmp5egLoB0culFAIf4Yh3//xLbed wqgXyZnX9b1xcJdKeKVyhxlDGDGayXBUOLHNy5Vr0bLWWHX7YlRn1HjJ5LOLiD9He99CAdTa P90cLcPF+VGVsO8dteGJvmd72e02QTFlxv/jgbXwtXfTPeZJxPq5pYAPY3q1GjqX1cqXCCAo XuRVpAl0VhFRajKZQYwKiMJEaUxYhZAAiNhppGMBZozXrc11t8J2Q1l4BsSNK43u+bQUzKn8 wu/J7qe6PC1QX6lJSMdYMfnvkpd+OiJCsQ7qnRzt2CsDr1Oyz1NYT2TBNTM2abD8m/RuJ/HQ +EzN0JosGP0Q70/668VslP9czOZ4xc8o4EXRHvOag9q4PzzW+zQ6IQ0eMHVHkgicVUEcL3aC XRu0KMO0pxxR6OA1OJIx/lmUNUuZCwAAAAAAAA== --------------ms060503000503080205070604--