From owner-freebsd-stable@FreeBSD.ORG Sun Oct 12 08:34:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9585F853 for ; Sun, 12 Oct 2014 08:34:58 +0000 (UTC) Received: from mail.webmatic.de (mail.webmatic.de [212.78.101.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.webmatic.de", Issuer "PositiveSSL CA 2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 404CB8E1 for ; Sun, 12 Oct 2014 08:34:57 +0000 (UTC) Received: from [192.168.178.27] (p548A745E.dip0.t-ipconnect.de [84.138.116.94]) by mail.webmatic.de (Postfix) with ESMTPSA id 23E1F8A06A for ; Sun, 12 Oct 2014 10:34:55 +0200 (CEST) Message-ID: <543A3D2C.80503@chef-ingenieur.de> Date: Sun, 12 Oct 2014 10:34:52 +0200 From: Thomas Krause User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: 3 TB USB disk References: <5438FAC6.3000807@chef-ingenieur.de> <729FD171-6362-4CA3-A0BA-2BF827A3EFB8@eternamente.info> In-Reply-To: <729FD171-6362-4CA3-A0BA-2BF827A3EFB8@eternamente.info> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 08:34:58 -0000 Am 11.10.2014 15:29, schrieb Nenhum_de_Nos: > Could you tell the hardware model and manufacturer? Thanks, Matheus Okay, my mistake. The enclosure supports only 2 TB HDDs. Its a fantec DB-ALU2 Thanks! Regards, Thomas. From owner-freebsd-stable@FreeBSD.ORG Sun Oct 12 10:08:30 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DA34CB54 for ; Sun, 12 Oct 2014 10:08:30 +0000 (UTC) Received: from nm7-vm1.access.bullet.mail.bf1.yahoo.com (nm7-vm1.access.bullet.mail.bf1.yahoo.com [216.109.114.160]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8031413F for ; Sun, 12 Oct 2014 10:08:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bellsouth.net; s=s1024; t=1413108368; bh=IL7f5L0rRauGA+/VOl+2B6W6+VgdvtpXFDX4WpC8ZLA=; h=Date:From:To:Subject:References:From:Subject; b=aY4vRLcMZkQrGXDx5/xgh8fMJy4lO3xQwYjDUz4SGewuOQ2lerqoyUFZBqEi/Y5z5bKgZexticM/6n8zaiRcprLzz/GNcselAzCtRd5s5SCg0z1yqaTtQmZAYe3U2EPAfGK8KmmCDCyDZjisBEgjl0SXZhd++p/tsZNS/goOudM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=bellsouth.net; b=iew7cFG+oeAmwTL25lSR++7Khg09X8aBby6wx4jVsryrD/bcjRBpXGNDtUj0CzomCjaOKWx4ltS7B6r8u4m/XjEZj0yt/uxMlMFEn6W6moxURNUgmBKQDS5wY9cgdKCK0PDNG0JvpCTrDChP3WoRQ8x2Mjsw5pb+fZ8vOVBBHHw=; Received: from [66.196.81.156] by nm7.access.bullet.mail.bf1.yahoo.com with NNFMP; 12 Oct 2014 10:06:08 -0000 Received: from [98.138.104.99] by tm2.access.bullet.mail.bf1.yahoo.com with NNFMP; 12 Oct 2014 10:06:08 -0000 Received: from [127.0.0.1] by smtp119.sbc.mail.ne1.yahoo.com with NNFMP; 12 Oct 2014 10:06:08 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bellsouth.net; s=s1024; t=1413108368; bh=IL7f5L0rRauGA+/VOl+2B6W6+VgdvtpXFDX4WpC8ZLA=; h=X-Yahoo-Newman-Id:Message-ID:Date:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:From:To:Subject:References; b=R5cASp7fle4jis/gyNCKBCBptGffiF5Vhv8NHgrW8N4XO0jMZyOZk3/isYRT2o+kPFa5+O1JM6Um65hy74bLynwKs+zwL6CVNi1WR2sYOf1xnGuUuU6eE2Ip9kFdwQVxSuroT105RpEmVTEmB2cjDQAphNeJyBtO8T2JgdRLerY= X-Yahoo-Newman-Id: 677159.30270.bm@smtp119.sbc.mail.ne1.yahoo.com Message-ID: <677159.30270.bm@smtp119.sbc.mail.ne1.yahoo.com> Date: Sun, 12 Oct 2014 10:06:08 +0000 (UTC) X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: fhBtGocVM1nlVrJHHOXAhNTkx4Vxugv0yjsfH3UH4ERDpNn mdzIQ4kjE5HQahjsT35OiYGppGpd0K20Hsz882jDBB_gzIIJ4scX7T09pjCV 8VHhJWQqswJN5syI_BQUamdWeKHReWgqUg8QzBs2ZitFt_NCynTyOnvhwYmO F4bjMsIy_JzBr2SjCCFslCT2U2Uv9Cr6jpiKuOs58ACbpSp3xQ0bXHrpDxtW 1hXEnsa.h2FuRb47poi6EaBOHPp8PkVjgKQBxWAmIXmz43tIq8WLothfs_lc E78R8pYIQod7PFXa8hKD2K8RGXLyGdx.BRaiiO0Yvel1M9lG4SOvL.lBNsgw 5uooJoCnSDyVNZvwlS_8LS92B7BxpgmLYhyEYK5HNgfvE1ndY.xu_IHi7.ng qzKbSuN7Fa7Lawt4UES6j7Ocz5zm40CAKIMP0HBv_CLj2dqoWTfhiOb4GVy6 OlRbGoSHcgOaXYqIBd1itp66Lyx4h2aO7y2EvwrAH4rSR1DG78ih5o4e5PYQ T_dNCwx6mmSCa1J.E9eifmxpAe7C494o9tAI3ODLe4Jrt.6j84XaDXopWBCc - X-Yahoo-SMTP: Kz_aW1.swBBYof3zAD7.RWzXz9ZAQVDMml1VADsbgPT4Kq79LC0- From: "Thomas Mueller" To: freebsd-stable@freebsd.org Subject: Re: 3 TB USB disk References: <5438FAC6.3000807@chef-ingenieur.de> <543A3D2C.80503@chef-ingenieur.de> X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 10:08:30 -0000 from Thomas Krause: > I connected a 3TB USB disk to a FreeBSD 9.3 server. The drive is > recognized as 2 disks (da0: 2 TB and da1: 1TB). What is > wrong here? > Okay, my mistake. The enclosure supports only 2 TB HDDs. Its a fantec DB-ALU2 Your first post sounded like it was a USB hard drive as opposed to an enclosure. I had a 3 TB USB 3.0 hard drive, sectors were 4096 bytes, OK with FreeBSD and Linux (System Rescue CD). I also have a 3 TB hard drive in a Sabrent enclosure, USB 2.0 and eSATA. Partitioning scheme is GPT. With USB 2.0, NetBSD can see the partitions, but not FreeBSD or Linux (System Rescue CD), though FreeBSD and Linux see the disk. With eSATA, it looks just like an internal SATA hard drive to system (UEFI), and to NetBSD, FreeBSD and Linux, which can see the partitions. If your USB disk is really a SATA hard drive in an enclosure, you could try using eSATA if your enclosure has that feature. There are hard-drive enclosures on the market that support both USB 3.0 and eSATA. There are also eSATA cables and internal eSATA adapters and connectors. Tom From owner-freebsd-stable@FreeBSD.ORG Sun Oct 12 11:50:05 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5EF45CEC for ; Sun, 12 Oct 2014 11:50:05 +0000 (UTC) Received: from burnus.net (burnus.net [85.214.218.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C4B1D14 for ; Sun, 12 Oct 2014 11:50:04 +0000 (UTC) Received: from nullzonenkonverter.lan (p508C71AD.dip0.t-ipconnect.de [80.140.113.173]) by burnus.net (Postfix) with ESMTPSA id E33332872034 for ; Sun, 12 Oct 2014 13:50:00 +0200 (CEST) Message-ID: <543A6AE3.2080503@burnus.net> Date: Sun, 12 Oct 2014 13:49:55 +0200 From: Christian Alge User-Agent: Postbox 3.0.11 (Macintosh/20140602) MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: Server insists on wrong hostname References: <5438F7A4.4070908@burnus.net> <54392A57.7020704@sorbs.net> <54393142.4050005@burnus.net> <5439326B.3070508@sorbs.net> <54393C76.7090702@burnus.net> <54393DBD.4030602@sorbs.net> <5439485F.4000002@burnus.net> <54394AF1.7000807@sorbs.net> <54395BC2.7090204@burnus.net> <20141011222243.GA23187@anubis.morrow.me.uk> In-Reply-To: <20141011222243.GA23187@anubis.morrow.me.uk> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 11:50:05 -0000 Thanks all of you. You made me realise my stupidity. The issue was caused by the fact that I mount another filesystem over the /etc directory during boot time (I know that this is dirty as hell, but it just has to make do for legacy reasons at the moment). Therefore the config that was read on boot is not visible on the running system. Changing the config on the boot file system did the trick (and helped me solve some other problems as well). Cheers. From owner-freebsd-stable@FreeBSD.ORG Sun Oct 12 20:40:41 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6228441F; Sun, 12 Oct 2014 20:40:41 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 021FF643; Sun, 12 Oct 2014 20:40:41 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGFK31HN8zTC; Sun, 12 Oct 2014 22:40:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413146433; x=1415738434; bh=sjHenPGix6lixGKB+tLxm1xd3 8Hpi3a8Lwzk0a9KC4k=; b=oz+Tox6sfOfLQczdobwcpNkBOi1xenRbO8xfOoJZp lVsCcPMt6DYIpCYHNJYk4uJsh6krSOQ6ZdpXoYhT9ZRgfaHdrzi16MQUMb/qgdkZ QOPIo8ixeCriDlK3f1VkK3YfLWeJrCCFLInPD0HUr/yQtRxhY7dqmIX9mWkwHOaa 6k= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id du5XWCIkoZTI; Sun, 12 Oct 2014 22:40:33 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Sun, 12 Oct 2014 22:40:33 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGFJx09zbz1P5; Sun, 12 Oct 2014 22:40:32 +0200 (CEST) Message-ID: <543AE740.7000808@ijs.si> Date: Sun, 12 Oct 2014 22:40:32 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> In-Reply-To: <543731F3.8090701@ijs.si> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 20:40:41 -0000 I made available an image copy (dd) of my 4 GiB partition (compressed down to a 384 MiB file), holding one of my partitions (a small bsd root) that can no longer be imported into a 10.1-RC1 or 10.1-BETA3, as described in my first posting (below): http://www.ijs.si/usr/mark/bsd/ I would appreciate if it can be confirmed that such pool (one of several I have with this symptom) causes 'zpool import' to hang on 10.1 or 10-STABLE: - download, xz -d sys1boot.img.xz # mdconfig -f sys1boot.img # zpool import sys1boot ... and advise on a solution. Considering that 'zdb -e -cc' is happy and there were no other prior trouble with these pools, except for an upgrade to 10.1-BETA3/-RC1 from 10-STABLE as of cca. late September, it is my belief that these pools are still healthy, just non-importable. I'm reluctant to upgrade any other system from 10.0 to 10.1 without finding out what went wrong here. Mark On 10/10/2014 03:02, Steven Hartland wrote: > Sorry to be a pain but could you attach the full output or link it > somewhere as mail has messed up the formatting :( Now at http://www.ijs.si/usr/mark/bsd/procstat-kka-2074.txt On 2014-10-10 Mark Martinec wrote: > In short, after upgrading to 10.1-BETA3 or -RC1 I ended up with several > zfs pools that can no longer be imported. The zpool import command > (with no arguments) does show all available pools, but trying to > import one just hangs and the command cannot be aborted, although > the rest of the system is still alive and fine: > > # zpool import -f > > Typing ^T just shows an idle process, waiting on [tx->tx_sync_done_cv]: > > load: 0.20 cmd: zpool 939 [tx->tx_sync_done_cv] 5723.65r 0.01u 0.02s 0% 8220k > load: 0.16 cmd: zpool 939 [tx->tx_sync_done_cv] 5735.73r 0.01u 0.02s 0% 8220k > load: 0.14 cmd: zpool 939 [tx->tx_sync_done_cv] 5741.83r 0.01u 0.02s 0% 8220k > load: 0.13 cmd: zpool 939 [tx->tx_sync_done_cv] 5749.16r 0.01u 0.02s 0% 8220k > > ps shows (on a system re-booted to a LiveCD running FreeBSD-10.1-RC1): > > PID TID COMM TDNAME CPU PRI STATE WCHAN > 939 100632 zpool - 5 122 sleep tx->tx_s > UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND > 0 939 801 0 22 0 107732 8236 tx->tx_s D+ v0 0:00.04 > zpool import -f -o cachefile=/tmp/zpool.cache -R /tmp/sys0boot sys0boot > > NWCHAN > fffff8007b0f2a20 > > # procstat -kk 939 > > PID TID COMM TDNAME KSTACK > 939 100632 zpool - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d txg_wait_synced+0x85 spa_load+0x1cd1 spa_load_best+0x6f spa_import+0x1ff zfs_ioc_pool_import+0x137 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb > > > Background story: the system where this happened was being kept > to a fairly recent 10-STABLE. The last upgrade was very close to > a BETA3 release. There are a couple of zfs pools there, one on a > mirrored pair of SSDs mostly holding the OS, one with a mirrored > pair of large spindles, and three more small ones (4 GiB each), > mostly for boot redundancy or testing - these small ones are on > old smallish disks. These disks are different, and attached to > different SATA controllers (LSI and onboard Intel). Pools were > mostly kept up-to-date to the most recent zpool features set > through their lifetime (some starting their life with 9.0, some > with 10.0). > > About two weeks ago after a reboot to a 10-STABLE of the day > the small pools became unavailable, but the regular two large > pools were still normal. At first I wasn't giving much attention > to that, as these pools were on oldish disks and nonessential > for normal operation, blaming a potentially crappy hw. > > Today I needed to do a reboot (for unrelated reason), and the > machine was no longer able to mount the boot pool. > The first instinct was - disks are malfunctioning - but ... > > Booting it to a FreeBSD-10.1-RC1 LiveCD was successful. > smartmon disk test shows no problems. dd is able to read whole > partititions of each problematic pool. And most importantly, > running a 'zdb -e -cc' on each (non-imported) pool was churning > normally and steadily, producing a stats report at the end > and reported no errors. > > As a final proof that disks are fine I sacrificed one of the broken > 4 GiB GPT partitions with one of the problematic pools, and > did a fresh 10.1-RC1 install on it from a distribution ISO DVD. > The installation went fine and the system does boot and run > fine from the newly installed OS. Trying to import one of the > remaining old pools hangs the import command as before. > > As a final proof, I copied (with dd) one of the broken 4 GiB > partitions to a file on another system (running 10.1-BETA3, > which did not suffer from this problem), made a memory disk > out of this file, then run zfs import on this pool - and it hangs > there too! So hardware was not a problem - either these partitions > are truly broken (even though zdb -cc says they are fine), > or the new OS is somehow no longer able to import them. > > Please advise. > > I have a copy of the 4 GiB partition on a 400 MB compressed > file available, if somebody would be willing to play with it. > > Also have a ktrace of the 'zpool import' command. It's last > actions before it hangs are: > > 939 zpool RET madvise 0 > 939 zpool CALL madvise(0x80604e000,0x1000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL close(0x6) > 939 zpool RET close 0 > 939 zpool CALL ioctl(0x3,0xc0185a05,0x7fffffffbf00) > 939 zpool RET ioctl -1 errno 2 No such file or directory > 939 zpool CALL madvise(0x802c71000,0x10000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL madvise(0x802ca5000,0x1000,MADV_FREE) > 939 zpool RET madvise 0 > 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) > 939 zpool RET ioctl 0 > 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) > 939 zpool RET ioctl 0 > 939 zpool CALL stat(0x802c380e0,0x7fffffffbc58) > 939 zpool NAMI "/tmp" > 939 zpool STRU struct stat {dev=273, ino=2, mode=041777, nlink=8, uid=0, gid=0, rdev=96, atime=1412866648, stime=1412871393, ctime=1412871393, birthtime=1412866648, size=512, blksize=32768, blocks=8, flags=0x0 } > 939 zpool RET stat 0 > 939 zpool CALL ioctl(0x3,0xc0185a02,0x7fffffffbc60) > > > Mark From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 00:54:21 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE61A9DB; Mon, 13 Oct 2014 00:54:21 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 2CB3368; Mon, 13 Oct 2014 00:54:20 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 8D0BC20E708AC; Mon, 13 Oct 2014 00:54:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id A360420E708AA; Mon, 13 Oct 2014 00:54:09 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 01:54:05 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 00:54:22 -0000 I have reproduced the issue with your image. It seems like the import is waiting a transation group to complete. It looks like its waiting on the zio from dsl_pool_sync_mos which for some reason is never returning. The pool does seem to be happy importing read only with: zpool import -N -f -o readonly=on sys1boot print dp->dp_tx $3 = { tx_cpu = 0xfffffe0004a05000, tx_sync_lock = { lock_object = { lo_name = 0xffffffff815f2941 "tx->tx_sync_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0 }, sx_lock = 1 }, tx_open_txg = 11733519, tx_quiesced_txg = 0, tx_syncing_txg = 11733518, tx_synced_txg = 0, tx_open_time = 120562502889, tx_sync_txg_waiting = 11733518, tx_quiesce_txg_waiting = 11733519, tx_sync_more_cv = { cv_description = 0xffffffff815f2953 "tx->tx_sync_more_cv", cv_waiters = 0 }, tx_sync_done_cv = { cv_description = 0xffffffff815f2968 "tx->tx_sync_done_cv", cv_waiters = 1 }, tx_quiesce_more_cv = { cv_description = 0xffffffff815f297d "tx->tx_quiesce_more_cv", cv_waiters = 1 }, tx_quiesce_done_cv = { cv_description = 0xffffffff815f2995 "tx->tx_quiesce_done_cv", cv_waiters = 0 }, tx_timeout_cv = { cv_description = 0x0, cv_waiters = 0 }, tx_exit_cv = { cv_description = 0xffffffff815f29ad "tx->tx_exit_cv", cv_waiters = 0 }, tx_threads = 2 '\002', tx_exiting = 0 '\0', tx_sync_thread = 0xfffff80041505490, tx_quiesce_thread = 0xfffff80041509920, tx_commit_cb_taskq = 0x0 } Relavent threads: #0 sched_switch (td=0xfffff800446bc000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80021f4b220, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80021f4b220, lock=0xfffff80021f4b1b8) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff8151ec4b in txg_wait_synced (dp=0xfffff80021f4b000, txg=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:631 #6 0xffffffff81510937 in spa_load (spa=0xfffffe0003fe1000, state=SPA_LOAD_IMPORT, type=, mosconfig=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2777 #7 0xffffffff8150e5df in spa_load_best (spa=0xfffffe0003fe1000, state=SPA_LOAD_IMPORT, mosconfig=1, max_request=, rewind_flags=1) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2873 #8 0xffffffff8150e023 in spa_import (pool=0xfffffe0002e2a000 "sys1boot", config=0xfffff80044f24ba0, props=0x0, flags=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:4207 #9 0xffffffff81566d78 in zfs_ioc_pool_import (zc=0xfffffe0002e2a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1594 #10 0xffffffff81563c4f in zfsdev_ioctl (dev=, zcmd=, arg=, flag=, td=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6158 #11 0xffffffff805299eb in devfs_ioctl_f (fp=0xfffff8004f020aa0, com=3222821378, data=0xfffff80038123e60, cred=, td=0xfffff800446bc000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #12 0xffffffff8066201b in kern_ioctl (td=, fd=, com=) at file.h:320 #13 0xffffffff80661d9c in sys_ioctl (td=0xfffff800446bc000, uap=0xfffffe011e49fa40) at /usr/src/sys/kern/sys_generic.c:702 #14 0xffffffff8088defa in amd64_syscall (td=0xfffff800446bc000, traced=0) at subr_syscall.c:134 #15 0xffffffff80870ecb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 Thread 509 (Thread 100533): #0 sched_switch (td=0xfffff800446bf000, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064e323 in sleepq_timedwait (wchan=0xfffff80044e47210, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:652 #4 0xffffffff805b84f0 in _cv_timedwait_sbt (cvp=0xfffff80044e47210, lock=0xfffff80044e471b8, sbt=, pr=, flags=) at /usr/src/sys/kern/kern_condvar.c:325 #5 0xffffffff8151f72e in txg_thread_wait (tx=, cpr=0xfffffe011dc76a00, cv=0xfffff80044e47210, time=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:249 #6 0xffffffff8151e930 in txg_sync_thread (arg=0xfffff80044e47000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:483 #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 , arg=0xfffff80044e47000, frame=0xfffffe011dc76ac0) at /usr/src/sys/kern/kern_fork.c:996 #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #9 0x0000000000000000 in ?? () Thread 508 (Thread 100532): #0 sched_switch (td=0xfffff80048204920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044e47230, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044e47230, lock=0xfffff80044e471b8) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff8151f73b in txg_thread_wait (tx=, cpr=0xfffffe011dc6da00, cv=, time=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:251 #6 0xffffffff8151e450 in txg_quiesce_thread (arg=0xfffff80044e47000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:556 #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e030 , arg=0xfffff80044e47000, frame=0xfffffe011dc6dac0) at /usr/src/sys/kern/kern_fork.c:996 #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #9 0x0000000000000000 in ?? () Thread 512 (Thread 100338): #0 sched_switch (td=0xfffff80041505490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1945 #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:494 #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044acf320, pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044acf320, lock=0xfffff80044acf300) at /usr/src/sys/kern/kern_condvar.c:139 #5 0xffffffff81547c9b in zio_wait (zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1442 #6 0xffffffff814f056c in dsl_pool_sync (dp=0xfffff80021f4b000, txg=11733518) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:531 #7 0xffffffff81514030 in spa_sync (spa=0xfffffe0003fe1000, txg=11733518) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6604 #8 0xffffffff8151e6cd in txg_sync_thread (arg=0xfffff80021f4b000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:518 #9 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 , arg=0xfffff80021f4b000, frame=0xfffffe011e6fdac0) at /usr/src/sys/kern/kern_fork.c:996 #10 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #11 0x0000000000000000 in ?? () ----- Original Message ----- From: "Mark Martinec" To: ; Sent: Sunday, October 12, 2014 9:40 PM Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] >I made available an image copy (dd) of my 4 GiB partition > (compressed down to a 384 MiB file), holding one of my > partitions (a small bsd root) that can no longer be imported > into a 10.1-RC1 or 10.1-BETA3, as described in my first > posting (below): > > http://www.ijs.si/usr/mark/bsd/ > > I would appreciate if it can be confirmed that such pool > (one of several I have with this symptom) causes > 'zpool import' to hang on 10.1 or 10-STABLE: > > - download, xz -d sys1boot.img.xz > # mdconfig -f sys1boot.img > # zpool import sys1boot > > ... and advise on a solution. > > Considering that 'zdb -e -cc' is happy and there were no > other prior trouble with these pools, except for an upgrade > to 10.1-BETA3/-RC1 from 10-STABLE as of cca. late September, > it is my belief that these pools are still healthy, just > non-importable. I'm reluctant to upgrade any other system > from 10.0 to 10.1 without finding out what went wrong here. > > Mark > > > On 10/10/2014 03:02, Steven Hartland wrote: > > Sorry to be a pain but could you attach the full output or link it > > somewhere as mail has messed up the formatting :( > > Now at > http://www.ijs.si/usr/mark/bsd/procstat-kka-2074.txt > > On 2014-10-10 Mark Martinec wrote: >> In short, after upgrading to 10.1-BETA3 or -RC1 I ended up with several >> zfs pools that can no longer be imported. The zpool import command >> (with no arguments) does show all available pools, but trying to >> import one just hangs and the command cannot be aborted, although >> the rest of the system is still alive and fine: >> >> # zpool import -f >> >> Typing ^T just shows an idle process, waiting on [tx->tx_sync_done_cv]: >> >> load: 0.20 cmd: zpool 939 [tx->tx_sync_done_cv] 5723.65r 0.01u 0.02s 0% 8220k >> load: 0.16 cmd: zpool 939 [tx->tx_sync_done_cv] 5735.73r 0.01u 0.02s 0% 8220k >> load: 0.14 cmd: zpool 939 [tx->tx_sync_done_cv] 5741.83r 0.01u 0.02s 0% 8220k >> load: 0.13 cmd: zpool 939 [tx->tx_sync_done_cv] 5749.16r 0.01u 0.02s 0% 8220k >> >> ps shows (on a system re-booted to a LiveCD running FreeBSD-10.1-RC1): >> >> PID TID COMM TDNAME CPU PRI STATE WCHAN >> 939 100632 zpool - 5 122 sleep tx->tx_s >> UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND >> 0 939 801 0 22 0 107732 8236 tx->tx_s D+ v0 0:00.04 >> zpool import -f -o cachefile=/tmp/zpool.cache -R /tmp/sys0boot sys0boot >> >> NWCHAN >> fffff8007b0f2a20 >> >> # procstat -kk 939 >> >> PID TID COMM TDNAME KSTACK >> 939 100632 zpool - mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x16d txg_wait_synced+0x85 >> spa_load+0x1cd1 spa_load_best+0x6f spa_import+0x1ff zfs_ioc_pool_import+0x137 zfsdev_ioctl+0x6f0 devfs_ioctl_f+0x114 >> kern_ioctl+0x255 sys_ioctl+0x13c amd64_syscall+0x351 Xfast_syscall+0xfb >> >> >> Background story: the system where this happened was being kept >> to a fairly recent 10-STABLE. The last upgrade was very close to >> a BETA3 release. There are a couple of zfs pools there, one on a >> mirrored pair of SSDs mostly holding the OS, one with a mirrored >> pair of large spindles, and three more small ones (4 GiB each), >> mostly for boot redundancy or testing - these small ones are on >> old smallish disks. These disks are different, and attached to >> different SATA controllers (LSI and onboard Intel). Pools were >> mostly kept up-to-date to the most recent zpool features set >> through their lifetime (some starting their life with 9.0, some >> with 10.0). >> >> About two weeks ago after a reboot to a 10-STABLE of the day >> the small pools became unavailable, but the regular two large >> pools were still normal. At first I wasn't giving much attention >> to that, as these pools were on oldish disks and nonessential >> for normal operation, blaming a potentially crappy hw. >> >> Today I needed to do a reboot (for unrelated reason), and the >> machine was no longer able to mount the boot pool. >> The first instinct was - disks are malfunctioning - but ... >> >> Booting it to a FreeBSD-10.1-RC1 LiveCD was successful. >> smartmon disk test shows no problems. dd is able to read whole >> partititions of each problematic pool. And most importantly, >> running a 'zdb -e -cc' on each (non-imported) pool was churning >> normally and steadily, producing a stats report at the end >> and reported no errors. >> >> As a final proof that disks are fine I sacrificed one of the broken >> 4 GiB GPT partitions with one of the problematic pools, and >> did a fresh 10.1-RC1 install on it from a distribution ISO DVD. >> The installation went fine and the system does boot and run >> fine from the newly installed OS. Trying to import one of the >> remaining old pools hangs the import command as before. >> >> As a final proof, I copied (with dd) one of the broken 4 GiB >> partitions to a file on another system (running 10.1-BETA3, >> which did not suffer from this problem), made a memory disk >> out of this file, then run zfs import on this pool - and it hangs >> there too! So hardware was not a problem - either these partitions >> are truly broken (even though zdb -cc says they are fine), >> or the new OS is somehow no longer able to import them. >> >> Please advise. >> >> I have a copy of the 4 GiB partition on a 400 MB compressed >> file available, if somebody would be willing to play with it. >> >> Also have a ktrace of the 'zpool import' command. It's last >> actions before it hangs are: >> >> 939 zpool RET madvise 0 >> 939 zpool CALL madvise(0x80604e000,0x1000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL close(0x6) >> 939 zpool RET close 0 >> 939 zpool CALL ioctl(0x3,0xc0185a05,0x7fffffffbf00) >> 939 zpool RET ioctl -1 errno 2 No such file or directory >> 939 zpool CALL madvise(0x802c71000,0x10000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL madvise(0x802ca5000,0x1000,MADV_FREE) >> 939 zpool RET madvise 0 >> 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) >> 939 zpool RET ioctl 0 >> 939 zpool CALL ioctl(0x3,0xc0185a06,0x7fffffffbf60) >> 939 zpool RET ioctl 0 >> 939 zpool CALL stat(0x802c380e0,0x7fffffffbc58) >> 939 zpool NAMI "/tmp" >> 939 zpool STRU struct stat {dev=273, ino=2, mode=041777, nlink=8, uid=0, gid=0, rdev=96, atime=1412866648, >> stime=1412871393, ctime=1412871393, birthtime=1412866648, size=512, blksize=32768, blocks=8, flags=0x0 } >> 939 zpool RET stat 0 >> 939 zpool CALL ioctl(0x3,0xc0185a02,0x7fffffffbc60) >> >> >> Mark > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 03:54:33 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 847859EE; Mon, 13 Oct 2014 03:54:33 +0000 (UTC) Received: from mail-yh0-x232.google.com (mail-yh0-x232.google.com [IPv6:2607:f8b0:4002:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3351E3B5; Mon, 13 Oct 2014 03:54:33 +0000 (UTC) Received: by mail-yh0-f50.google.com with SMTP id a41so3265925yho.23 for ; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Y8Pu4pQN5ssvKh1dxtAFY4vwN4CziU13Dc6OZwmPSGQ=; b=Fh4FNi2ovq6/C87wRWOwZDVnYLzFhhL83XYs5HBdMbzOcLV3YYTN0suhbIhtA2goOq Mrrq+RrZgfnWtLsGzu08fGYllJvFR7W7to+9DqVmXyGyYSzT0Ms3L8Mux5CIJkzZYUbE duc/rYDlqaGZ1TqRFoHZh6j8EILsyyF2uNjWeKjq9QoN5kvXZK1EKH3cjzpStcFozZyE fg1NbsQGx0OFdY5wpd29gaRCW+0p9gDi/a8co82um/WD0e2aiV2M69215YlxCBlJJb9t q8Nl0q0oPWnvFp5pKiwIh49lUQGGRtulh7p0+6gxiqjC/09Otp4l/8DwtaWlNEqHaVYU 1EGg== MIME-Version: 1.0 X-Received: by 10.236.32.4 with SMTP id n4mr33142798yha.16.1413172472105; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sun, 12 Oct 2014 20:54:32 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> Date: Sun, 12 Oct 2014 20:54:32 -0700 X-Google-Sender-Auth: Ed-LGs5HF9Eo1bT7G0bE8hehL2Q Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 03:54:33 -0000 Relavent threads: > > #0 sched_switch (td=0xfffff800446bc000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80021f4b220, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80021f4b220, > lock=0xfffff80021f4b1b8) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff8151ec4b in txg_wait_synced (dp=0xfffff80021f4b000, > txg=) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:631 > #6 0xffffffff81510937 in spa_load (spa=0xfffffe0003fe1000, > state=SPA_LOAD_IMPORT, type=, mosconfig= optimized out>) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2777 > #7 0xffffffff8150e5df in spa_load_best (spa=0xfffffe0003fe1000, > state=SPA_LOAD_IMPORT, mosconfig=1, max_request=, > rewind_flags=1) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2873 > #8 0xffffffff8150e023 in spa_import (pool=0xfffffe0002e2a000 "sys1boot", > config=0xfffff80044f24ba0, props=0x0, flags=) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/spa.c:4207 > #9 0xffffffff81566d78 in zfs_ioc_pool_import (zc=0xfffffe0002e2a000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/zfs_ioctl.c:1594 > #10 0xffffffff81563c4f in zfsdev_ioctl (dev=, > zcmd=, arg=, flag= optimized out>, td=) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6158 > #11 0xffffffff805299eb in devfs_ioctl_f (fp=0xfffff8004f020aa0, > com=3222821378, data=0xfffff80038123e60, cred=, > td=0xfffff800446bc000) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 > #12 0xffffffff8066201b in kern_ioctl (td=, fd= optimized out>, com=) at file.h:320 > #13 0xffffffff80661d9c in sys_ioctl (td=0xfffff800446bc000, > uap=0xfffffe011e49fa40) at /usr/src/sys/kern/sys_generic.c:702 > #14 0xffffffff8088defa in amd64_syscall (td=0xfffff800446bc000, traced=0) > at subr_syscall.c:134 > #15 0xffffffff80870ecb in Xfast_syscall () at /usr/src/sys/amd64/amd64/ > exception.S:391 > > > Thread 509 (Thread 100533): > #0 sched_switch (td=0xfffff800446bf000, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064e323 in sleepq_timedwait (wchan=0xfffff80044e47210, > pri=0) at /usr/src/sys/kern/subr_sleepqueue.c:652 > #4 0xffffffff805b84f0 in _cv_timedwait_sbt (cvp=0xfffff80044e47210, > lock=0xfffff80044e471b8, sbt=, pr= out>, flags=) at /usr/src/sys/kern/kern_condvar.c:325 > #5 0xffffffff8151f72e in txg_thread_wait (tx=, > cpr=0xfffffe011dc76a00, cv=0xfffff80044e47210, time=) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:249 > #6 0xffffffff8151e930 in txg_sync_thread (arg=0xfffff80044e47000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:483 > #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 > , arg=0xfffff80044e47000, frame=0xfffffe011dc76ac0) at > /usr/src/sys/kern/kern_fork.c:996 > #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #9 0x0000000000000000 in ?? () > > Thread 508 (Thread 100532): > #0 sched_switch (td=0xfffff80048204920, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044e47230, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044e47230, > lock=0xfffff80044e471b8) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff8151f73b in txg_thread_wait (tx=, > cpr=0xfffffe011dc6da00, cv=, time= out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:251 > #6 0xffffffff8151e450 in txg_quiesce_thread (arg=0xfffff80044e47000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:556 > #7 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e030 > , arg=0xfffff80044e47000, frame=0xfffffe011dc6dac0) at > /usr/src/sys/kern/kern_fork.c:996 > #8 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #9 0x0000000000000000 in ?? () > > > Thread 512 (Thread 100338): > #0 sched_switch (td=0xfffff80041505490, newtd=, > flags=) at /usr/src/sys/kern/sched_ule.c:1945 > #1 0xffffffff806110d9 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:494 > #2 0xffffffff8064ddc2 in sleepq_switch (wchan=, > pri=) at /usr/src/sys/kern/subr_sleepqueue.c:538 > #3 0xffffffff8064dc23 in sleepq_wait (wchan=0xfffff80044acf320, pri=0) at > /usr/src/sys/kern/subr_sleepqueue.c:617 > #4 0xffffffff805b7c1a in _cv_wait (cvp=0xfffff80044acf320, > lock=0xfffff80044acf300) at /usr/src/sys/kern/kern_condvar.c:139 > #5 0xffffffff81547c9b in zio_wait (zio=) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/zio.c:1442 > #6 0xffffffff814f056c in dsl_pool_sync (dp=0xfffff80021f4b000, > txg=11733518) at /usr/src/sys/modules/zfs/../.. > /cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:531 > #7 0xffffffff81514030 in spa_sync (spa=0xfffffe0003fe1000, txg=11733518) > at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/spa.c:6604 > #8 0xffffffff8151e6cd in txg_sync_thread (arg=0xfffff80021f4b000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ > common/fs/zfs/txg.c:518 > #9 0xffffffff805d5bd4 in fork_exit (callout=0xffffffff8151e480 > , arg=0xfffff80021f4b000, frame=0xfffffe011e6fdac0) at > /usr/src/sys/kern/kern_fork.c:996 > #10 0xffffffff8087111e in fork_trampoline () at /usr/src/sys/amd64/amd64/ > exception.S:606 > #11 0x0000000000000000 in ?? () > A recent quick read of the code would lead me to believe that zio_wait not returning there means that the zio never reached the zio_done stage. Parent zios seem to yield in a couple of stages in the pipeline if they have incomplete children. They determine this by calling zio_wait_for_children with zio child types and their corresponding wait type. In so doing they set the io_stall to the count of the number of waiters of the first non-zero check. This parent I/O will be resumed by the last child zio of that type and wait state in zio_notify_parent. I'm sure you know all this - but I wrote it to preface asking for the following fields of the zio being waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children is pointed to) *io_stall, io_children[*][*], io_child_list (at a first glance just the addresses). The other alternative is that it reexecuting has gotten in to a bad place in the state machine so io_reexecute. Thanks. -K From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 03:56:39 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28612C99 for ; Mon, 13 Oct 2014 03:56:39 +0000 (UTC) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A641A3E0 for ; Mon, 13 Oct 2014 03:56:38 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id mc6so5956883lab.2 for ; Sun, 12 Oct 2014 20:56:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=r8wi/3dc3jYH9v8ocjhMsBlnc3MBpwdi6PEu/H9qbTY=; b=kPUkoGYqyX4BGKZNp5/pvO5FQ0vmBBBRdpHXHyPz01vJxMJlHH052Ok9WlIOpBAB2n HCnKvVwSG/1eq0H/3E8HCjwTZWg9y84NiZpQlenx1G0GFN1n1SPoj1RRUlwUCNcFdfYP Ao8jg/RSDjS3N2p08EDMff80Kyig1/c8imsl57T8XvmOdw/bI4sf+wMkcLVYqxfz1q/w SyLAJjqxTwR/EuFc00E5jmw16uK51TT6Ww/kdbSpkcUFLqU0Ui5Uy3M2IOVA+g0vRyQg Jiujra+wPcGNI9mR08fOSLzVaieWe/6eUbPJVk61GrjOtVw2SRfByn30I0ODvniZhSbk g1DQ== MIME-Version: 1.0 X-Received: by 10.152.6.71 with SMTP id y7mr493903lay.83.1413172596616; Sun, 12 Oct 2014 20:56:36 -0700 (PDT) Received: by 10.152.121.100 with HTTP; Sun, 12 Oct 2014 20:56:36 -0700 (PDT) Date: Sun, 12 Oct 2014 23:56:36 -0400 Message-ID: Subject: FreeBSD 10-STABLE Bhyve Ubuntu guest freeze From: Manas Bhatnagar To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 03:56:39 -0000 Hello, I am running r271243 I just installed a Ubuntu 14.04.1 server guest which went fine. After updating the system with "apt-get dist-upgrade" and rebooting, the VM does not complete the boot process. The last few lines displayed are: [ 0.002359] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.003757] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.004731] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.005267] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.006128] Initializing cgroup subsys memory [ 0.006493] Initializing cgroup subsys devices [ 0.006859] Initializing cgroup subsys freezer [ 0.007226] Initializing cgroup subsys blkio [ 0.007579] Initializing cgroup subsys perf_event [ 0.007966] Initializing cgroup subsys hugetlb [ 0.008665] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0 [ 0.008665] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32 [ 0.008665] tlb_flushall_shift: 2 [ 0.010121] Freeing SMP alternatives memory: 32K (ffffffff81e6e000 - ffffffff81e76000) [ 0.014325] ACPI: Core revision 20131115 [ 0.014888] ACPI: All ACPI Tables successfully acquired [ 0.015391] ftrace: allocating 28541 entries in 112 pages [ 0.048623] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (fam: 06, model: 3a, stepping: 09) Does anyone know what is preventing the VM from booting successfully? Thanks, Manas From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 06:14:49 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DCCF5AE; Mon, 13 Oct 2014 06:14:49 +0000 (UTC) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82ACC27D; Mon, 13 Oct 2014 06:14:48 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id ge10so6093050lab.10 for ; Sun, 12 Oct 2014 23:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:cc:content-type; bh=SXY9olLNRdQmN9wXDYsUB8DP+E9zWbp/iPgWRwEL9Iw=; b=LDHy7GQBX+jFUHphUkg+b9h8jj/uSHsLqmM7SFpieMcWPzrnxys7bQfyfFxbhx2yJu nbgRRqbgjIU5oXqpQZL0TtGXWB7DXRbe999id6UWxB09Qm/k+A3+P3g+gnUyUHSxEFu6 VBflNgyL5hRhlHZRhkpZY84lb/fy25wTAL7A1LJG59HXf+VioEXhhfqqS8L5g7Y0ZHhX PmgtynqZ0s14hESRt7+9rMBVdLuJUbn2tSSU+6R7ajLDVIGKMmjYeNSi2XZvrsbJWZYa wj19ctCQvn35gTHd4Pf0tv4gLBffTQYWL3QYkM2XbMQJ3f5FKuME0v++WLgrh7mP0NVs PqFg== MIME-Version: 1.0 X-Received: by 10.112.52.165 with SMTP id u5mr1166163lbo.80.1413180885134; Sun, 12 Oct 2014 23:14:45 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.131.66 with HTTP; Sun, 12 Oct 2014 23:14:45 -0700 (PDT) Date: Sun, 12 Oct 2014 23:14:45 -0700 X-Google-Sender-Auth: x4hPCYLIElEluSbZGKUNjM4-9C0 Message-ID: Subject: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: FreeBSD stable Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 06:14:49 -0000 Hi, I have created this Jenkins job, which you can see a graphical representation of: https://jenkins.freebsd.org/jenkins/view/FreeBSD_src_stable/job/FreeBSD_stable_10/848/BuildGraph/ (1) does a buildworld/buildkernel on amd64 when someone checks new code into the stable/10 branch (2) Creates a bootable UFS image with makefs (3) Boots the image under bhyve (4) Runs these commands inside the bhyve VM: cd /usr/tests kyua test kyua report-junit --output=test-output.xml (5) Shuts down the bhyve VM (6) imports test-output.xml into Jenkins. You can see a full test report here: https://jenkins.freebsd.org/jenkins/job/FreeBSD_stable_10-tests/4/testReport/ We already do the same thing for CURRENT. Hopefully by running the tests regularly, we can help improve the quality of FreeBSD. -- Craig From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 06:17:42 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3C6A3711; Mon, 13 Oct 2014 06:17:42 +0000 (UTC) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED8E02BA; Mon, 13 Oct 2014 06:17:41 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id bj1so5385471pad.29 for ; Sun, 12 Oct 2014 23:17:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=YAUKX8iaONTwNKnKwKh+fu1Qk44deLc826PFoEGYTmY=; b=oM2ifeeR59WTAx4COQYpOkeus+/1a+7M+6D3uRRqPrE56jotuR7zxF1T4WymDEd1QF cp4gMv0sJruolltd0qSLfSrlcEHOJQiTI0NBJSk9beRECs/1xaC6Qd0OzUdJ5hn8ENSO Nn3U5JzbrY7hEPD2XnClHsHjkxhqiAzv3RQ7i5YYShW85BjiJ6oAwn21KIl1vMPyYQO4 aHXVAnhuPcoHcRTAVQE1Mgj4EeMus32kmtZ318/Wcq30TEbV4yHLQD/x5AYartaYoNrA gPgzxKDCwVZ4TCGjST47NO4IIcMkHpdDw3DoyDnLdRNqf/7R8nR4gkTmzOfrgiXwUyC6 6nVA== X-Received: by 10.70.126.101 with SMTP id mx5mr21817658pdb.112.1413181061561; Sun, 12 Oct 2014 23:17:41 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:3158:a83c:dca7:c7bc? ([2601:8:ab80:7d6:3158:a83c:dca7:c7bc]) by mx.google.com with ESMTPSA id kk10sm10089234pdb.63.2014.10.12.23.17.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 12 Oct 2014 23:17:41 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_FE9F3DB8-2654-45AC-9B11-E2E54ADA38A3"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: Date: Sun, 12 Oct 2014 23:17:40 -0700 Message-Id: References: To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 06:17:42 -0000 --Apple-Mail=_FE9F3DB8-2654-45AC-9B11-E2E54ADA38A3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 12, 2014, at 23:14, Craig Rodrigues wrote: > Hi, >=20 > I have created this Jenkins job, which you can see a graphical > representation of: >=20 > = https://jenkins.freebsd.org/jenkins/view/FreeBSD_src_stable/job/FreeBSD_st= able_10/848/BuildGraph/ =85 Hi Craig! As much as everyone would like to take i386 out to pasture, = there=92s a large degree of value in running i386 tests on 11-CURRENT = and 10-STABLE (I=92ve caught some interesting build bugs and test bugs = by running on my i386/CURRENT VM). Are there any plans to have i386 = executors running tests anytime soon (does bhyve support i386?)? Thanks! -Garrett --Apple-Mail=_FE9F3DB8-2654-45AC-9B11-E2E54ADA38A3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUO26EAAoJEMZr5QU6S73e5iYIAKWNtVc+D9KflJ07hRtE/1mD bJZEGltByNu+IJITu3/wA6JVRH1mtZlMwh0YZ3fBsp82PQIVJbA7Q85EvLDa2Uy0 pF0Zu7sEsofe+9ptMIjcaUjRLWujy1n1o6s7IMKse0oZ77dCSXVHgh41eOAH932+ YcQ/J+lIyIHvMgykdSkX+UgUDRj41JbJOSB5xIuFJ27hIxgTDQFWdtaJEuo+D9lj qJr4sq/PKaL7yUMJdEhgUjU51HcHSuR9IGEKOQGGvcGcw0YmztZis1RmSVPDrRmc 0i//162LeYrZjAKtd71nmPBh9TBF/x3E64iGOvD5uEv5FSPMeHbhgLAEvQmHc4c= =mbYP -----END PGP SIGNATURE----- --Apple-Mail=_FE9F3DB8-2654-45AC-9B11-E2E54ADA38A3-- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 06:20:48 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B2C77891; Mon, 13 Oct 2014 06:20:48 +0000 (UTC) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 703E52ED; Mon, 13 Oct 2014 06:20:48 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTP id 994BA1278A; Mon, 13 Oct 2014 16:20:40 +1000 (EST) Received: from Peter-Grehans-MacBook-Pro-2.local (c-67-161-27-37.hsd1.ca.comcast.net [67.161.27.37]) by dommail.onthenet.com.au (MOS 4.4.4-GA) with ESMTP id BYY99736 (AUTH peterg@ptree32.com.au); Mon, 13 Oct 2014 16:20:38 +1000 Message-ID: <543B6F30.9060505@freebsd.org> Date: Sun, 12 Oct 2014 23:20:32 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Garrett Cooper , Craig Rodrigues Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 06:20:48 -0000 > (does bhyve support i386?)? Yes, and also PAE (props to jhb). later, Peter. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 08:06:12 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 58439D6F; Mon, 13 Oct 2014 08:06:12 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 17B23F08; Mon, 13 Oct 2014 08:06:11 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 86B5220E708CA; Mon, 13 Oct 2014 08:06:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 3A10F20E708C8; Mon, 13 Oct 2014 08:06:08 +0000 (UTC) Message-ID: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si><644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk><54372EBA.1000908@ijs.si><543731F3.8090701@ijs.si><543AE740.7000808@ijs.si> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 09:06:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:06:12 -0000 ----- Original Message ----- From: "K. Macy" > A recent quick read of the code would lead me to believe that zio_wait not > returning there means that the zio never reached the zio_done stage. Parent > zios seem to yield in a couple of stages in the pipeline if they have > incomplete children. They determine this by calling zio_wait_for_children > with zio child types and their corresponding wait type. In so doing they > set the io_stall to the count of the number of waiters of the first > non-zero check. This parent I/O will be resumed by the last child zio of > that type and wait state in zio_notify_parent. I'm sure you know all this - > but I wrote it to preface asking for the following fields of the zio being > waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children > is pointed to) *io_stall, io_children[*][*], io_child_list (at a first > glance just the addresses). The other alternative is that it reexecuting >has gotten in to a bad place in the state machine so io_reexecute. Yer I would have got the zio details but typically its "optimised out" by the compiler, so will need some effort to track that down unfortunately :( Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 08:11:49 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BD31F84; Mon, 13 Oct 2014 08:11:49 +0000 (UTC) Received: from mail-yh0-x234.google.com (mail-yh0-x234.google.com [IPv6:2607:f8b0:4002:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF485FDB; Mon, 13 Oct 2014 08:11:48 +0000 (UTC) Received: by mail-yh0-f52.google.com with SMTP id f10so3361152yha.39 for ; Mon, 13 Oct 2014 01:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=tA7AS5zwDI5HVrZ6Xn0uKpprexnAHw7Smzpb3vx6r8w=; b=UuV/zrbTF0VDXj8b8cNhvrUhQzZbyp1wxDYGZbiwOLNIpfR9HI0phq3iq95U7PkR1o c0J9v7P1BBJppCC7z8VwEkUoY34J3G7sfnkVlTyLtgC/YP5h+TfxUtNlFpGTn29IE0wk +TzZtazYauC7cQ3yKs2rssGhmy1P+pqTsEKapNtUBFNZOyP2Jak2NMXqKQIzaPYTPbDF ffvung5Vlb96C5JRdBYiCMD3HWMkjYmKUY6VCL4ukgZp/erICBI2ANrgy6huHn0Xeiq0 SCp6buFM/n++wmV3OmHBLvaq5c/pXRnlMoVlMSmGs4QHkB1Ivd5DeZDyMd2zyYtAdxzT 216A== MIME-Version: 1.0 X-Received: by 10.236.54.101 with SMTP id h65mr36033279yhc.47.1413187907802; Mon, 13 Oct 2014 01:11:47 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 01:11:47 -0700 (PDT) In-Reply-To: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 01:11:47 -0700 X-Google-Sender-Auth: FzUffdsgjnUxbaw-euX2JdaqQKQ Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:11:49 -0000 On Mon, Oct 13, 2014 at 1:06 AM, Steven Hartland wrote: > ----- Original Message ----- From: "K. Macy" > > A recent quick read of the code would lead me to believe that zio_wait not >> returning there means that the zio never reached the zio_done stage. >> Parent >> zios seem to yield in a couple of stages in the pipeline if they have >> incomplete children. They determine this by calling zio_wait_for_children >> with zio child types and their corresponding wait type. In so doing they >> set the io_stall to the count of the number of waiters of the first >> non-zero check. This parent I/O will be resumed by the last child zio of >> that type and wait state in zio_notify_parent. I'm sure you know all this >> - >> but I wrote it to preface asking for the following fields of the zio being >> waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children >> is pointed to) *io_stall, io_children[*][*], io_child_list (at a first >> glance just the addresses). The other alternative is that it reexecuting >> has gotten in to a bad place in the state machine so io_reexecute. >> > > Yer I would have got the zio details but typically its "optimised out" by > the > compiler, so will need some effort to track that down unfortunately :( > Well, let me know if you can. Re-creating a new 10.x VM is taking a while as it's taking me forever to checkout the sources. Things like that need to somehow continue to be accessible. Cheers. -K From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 08:35:47 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E57F3439; Mon, 13 Oct 2014 08:35:47 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B73623E; Mon, 13 Oct 2014 08:35:47 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s9D8ZZZT041611; Mon, 13 Oct 2014 10:35:35 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id ABB9F3037; Mon, 13 Oct 2014 10:35:34 +0200 (CEST) Message-ID: <543B8ED5.6040206@omnilan.de> Date: Mon, 13 Oct 2014 10:35:33 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Julian Elischer Subject: Re: Deleting IPv4 iface-routes from extra FIBs References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> In-Reply-To: <535771F3.4070007@freebsd.org> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6230EC5227E68E506CAFA445" X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 13 Oct 2014 10:35:35 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) Cc: "freebsd-net@freebsd.org" , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:35:48 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6230EC5227E68E506CAFA445 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Bez=C3=BCglich Julian Elischer's Nachricht vom 23.04.2014 09:55 (localti= me): > On 4/23/14, 4:38 AM, Nikolay Denev wrote: >> On Tue, Apr 22, 2014 at 5:37 PM, Harald Schmalzbauer >> wrote: >>> Hello, >>> >>> here, http://svnweb.freebsd.org/base?view=3Drevision&revision=3D24889= 5 >>> interface route protection was added (so the following problem arose >>> with 9.2). >>> >>> Unfortunately, in my case, I must be able to delete these routes; >>> not in >>> the default FIB, but in jail's fibs, because: >>> =C2=B7 Host is multihomed with multiple nics in different subnets. >>> =C2=B7 Jail's IP (no vnet) is from a different subnet than host's >>> default-router subnet =E2=80=93 jail has no ip in the range of host's= >>> default-router!!! >>> =C2=B7 FIB used by jail contains valid default-router. >>> >>> Problem: >>> If iface-routes exist in jail's FIB, answer-packets take the >>> iface-shortcut, not trespassing the router (default gateway); hence >>> 3way-handshake never finishes and firewall terminates (half-opened) T= CP >>> sessions. >>> >>> Workarround: >>> =C2=B7 Abuse packet filter doing some kind of route-to=E2=80=A6 >>> =C2=B7 Revert r248895, to be able to delete v4-iface-routes (inet6-ro= utes >>> can >>> be deleted without any hack) >>> >>> Desired solution: >>> =C2=B7 Allow deletion of v4-iface-routes if FIB!=3D0. >>> >>> Unfortunately my C skills don't allow me to implement this myself :-(= >>> I can't even follow the code, I guess that was originally considered,= >>> but possibly doesn't work bacause of a simple bug?!? I took the lazy >>> way >>> and simply reverted r248895 instead of trying to understand >>> rtrequest1_fib(). I wish I had the time to learn=E2=80=A6 >>> >>> Thanks for any help, >>> >>> -Harry >>> >> Hi, >> >> As it was suggested before as immediate workaround you can set >> net.add_addr_allfibs=3D0 so that the interface routes are added only i= n >> the default FIB. > > yes, we made two behaviours. > Add interface routes to all active FIBS or only add them to the first > fib and let the user populate other fibs as needed. > It appears you want the second behaviour, so I suggest you use that > option and set up all your routes manually. Hello, last time I had the iface-route problem, I just reverted r248895 (for 9.3). There was inconsitent behaviour with v6 iface routes and net.add_addr_allfibs=3D0. Now I checked with 10.1 ans it seems net.add_addr_allfibs=3D0 doesn't wor= k any more: netstat -f inet -nr Routing tables Internet: Destination Gateway Flags Netif Expire default 172.21.32.1 UGS egn 127.0.0.1 link#2 UH lo0 172.21.32.0/19 link#1 U egn 172.21.35.1 link#1 UHS lo0 netstat -F 1 -f inet -nr Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire 127.0.0.1 link#2 UH lo0 172.21.32.0/19 link#1 U egn 'sysctl net.add_addr_allfibs' net.add_addr_allfibs: 0 Shouldn't the routing table for fib1 stay empty? Can't remember the result when I testet that with 9.3 :-( Thanks, -Harry --------------enig6230EC5227E68E506CAFA445 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlQ7jtYACgkQLDqVQ9VXb8gJHQCfQKViwDAMUH8tgn4hfo0G93k3 mhUAoKjj8OX/wKIwfyQmvAwCWEfJdOja =zn/y -----END PGP SIGNATURE----- --------------enig6230EC5227E68E506CAFA445-- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 08:43:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38DDB600; Mon, 13 Oct 2014 08:43:58 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EBC9F33B; Mon, 13 Oct 2014 08:43:57 +0000 (UTC) Received: from [2a02:6b8:0:401:222:4dff:fe50:cd2f] (helo=ptichko.yndx.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XdXEm-000EED-6I; Mon, 13 Oct 2014 08:28:00 +0400 Message-ID: <543B9075.2000102@FreeBSD.org> Date: Mon, 13 Oct 2014 12:42:29 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Harald Schmalzbauer , Julian Elischer Subject: Re: Deleting IPv4 iface-routes from extra FIBs References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> In-Reply-To: <543B8ED5.6040206@omnilan.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-net@freebsd.org" , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 08:43:58 -0000 On 13.10.2014 12:35, Harald Schmalzbauer wrote: > Bezüglich Julian Elischer's Nachricht vom 23.04.2014 09:55 (localtime): >> On 4/23/14, 4:38 AM, Nikolay Denev wrote: >>> On Tue, Apr 22, 2014 at 5:37 PM, Harald Schmalzbauer >>> wrote: >>>> Hello, >>>> >>>> here, http://svnweb.freebsd.org/base?view=revision&revision=248895 >>>> interface route protection was added (so the following problem arose >>>> with 9.2). >>>> >>>> Unfortunately, in my case, I must be able to delete these routes; >>>> not in >>>> the default FIB, but in jail's fibs, because: >>>> · Host is multihomed with multiple nics in different subnets. >>>> · Jail's IP (no vnet) is from a different subnet than host's >>>> default-router subnet – jail has no ip in the range of host's >>>> default-router!!! >>>> · FIB used by jail contains valid default-router. >>>> >>>> Problem: >>>> If iface-routes exist in jail's FIB, answer-packets take the >>>> iface-shortcut, not trespassing the router (default gateway); hence >>>> 3way-handshake never finishes and firewall terminates (half-opened) TCP >>>> sessions. >>>> >>>> Workarround: >>>> · Abuse packet filter doing some kind of route-to… >>>> · Revert r248895, to be able to delete v4-iface-routes (inet6-routes >>>> can >>>> be deleted without any hack) >>>> >>>> Desired solution: >>>> · Allow deletion of v4-iface-routes if FIB!=0. >>>> >>>> Unfortunately my C skills don't allow me to implement this myself :-( >>>> I can't even follow the code, I guess that was originally considered, >>>> but possibly doesn't work bacause of a simple bug?!? I took the lazy >>>> way >>>> and simply reverted r248895 instead of trying to understand >>>> rtrequest1_fib(). I wish I had the time to learn… >>>> >>>> Thanks for any help, >>>> >>>> -Harry >>>> >>> Hi, >>> >>> As it was suggested before as immediate workaround you can set >>> net.add_addr_allfibs=0 so that the interface routes are added only in >>> the default FIB. >> yes, we made two behaviours. >> Add interface routes to all active FIBS or only add them to the first >> fib and let the user populate other fibs as needed. >> It appears you want the second behaviour, so I suggest you use that >> option and set up all your routes manually. > Hello, > > last time I had the iface-route problem, I just reverted r248895 (for > 9.3). There was inconsitent behaviour with v6 iface routes and > net.add_addr_allfibs=0. > Now I checked with 10.1 ans it seems net.add_addr_allfibs=0 doesn't work > any more: > netstat -f inet -nr > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 172.21.32.1 UGS egn > 127.0.0.1 link#2 UH lo0 > 172.21.32.0/19 link#1 U egn > 172.21.35.1 link#1 UHS lo0 > > netstat -F 1 -f inet -nr > Routing tables (fib: 1) > > Internet: > Destination Gateway Flags Netif Expire > 127.0.0.1 link#2 UH lo0 > 172.21.32.0/19 link#1 U egn > > 'sysctl net.add_addr_allfibs' > net.add_addr_allfibs: 0 Are you sure net.add_addr_allfibs was applied before interface address added? Can you check recent 10-STABLE code? It might have more fixes related to allfibs. > > Shouldn't the routing table for fib1 stay empty? Can't remember the > result when I testet that with 9.3 :-( Yes, it should. We're slowly moving to this direction > > Thanks, > > -Harry > > From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 09:16:39 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DD38C11; Mon, 13 Oct 2014 09:16:39 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D120D84E; Mon, 13 Oct 2014 09:16:38 +0000 (UTC) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s9D9GaZp042023; Mon, 13 Oct 2014 11:16:36 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 24CB63046; Mon, 13 Oct 2014 11:16:36 +0200 (CEST) Message-ID: <543B9873.3040605@omnilan.de> Date: Mon, 13 Oct 2014 11:16:35 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: "Alexander V. Chernikov" Subject: Re: Deleting IPv4 iface-routes from extra FIBs References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> <543B9075.2000102@FreeBSD.org> In-Reply-To: <543B9075.2000102@FreeBSD.org> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigDFEC5242399331AD23FB8855" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Mon, 13 Oct 2014 11:16:36 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) Cc: "freebsd-net@freebsd.org" , Julian Elischer , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 09:16:39 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigDFEC5242399331AD23FB8855 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Bez=C3=BCglich Alexander V. Chernikov's Nachricht vom 13.10.2014 10:42 (localtime): > On 13.10.2014 12:35, Harald Schmalzbauer wrote: >> Bez=C3=BCglich Julian Elischer's Nachricht vom 23.04.2014 09:55 >> (localtime): =2E.. >>> yes, we made two behaviours. >>> Add interface routes to all active FIBS or only add them to the first= >>> fib and let the user populate other fibs as needed. >>> It appears you want the second behaviour, so I suggest you use that >>> option and set up all your routes manually. >> Hello, >> >> last time I had the iface-route problem, I just reverted r248895 (for >> 9.3). There was inconsitent behaviour with v6 iface routes and >> net.add_addr_allfibs=3D0. >> Now I checked with 10.1 ans it seems net.add_addr_allfibs=3D0 doesn't = work >> any more: >> netstat -f inet -nr >> Routing tables >> >> Internet: >> Destination Gateway Flags Netif Expire >> default 172.21.32.1 UGS egn >> 127.0.0.1 link#2 UH lo0 >> 172.21.32.0/19 link#1 U egn >> 172.21.35.1 link#1 UHS lo0 >> >> netstat -F 1 -f inet -nr >> Routing tables (fib: 1) >> >> Internet: >> Destination Gateway Flags Netif Expire >> 127.0.0.1 link#2 UH lo0 >> 172.21.32.0/19 link#1 U egn >> >> 'sysctl net.add_addr_allfibs' >> net.add_addr_allfibs: 0 > Are you sure net.add_addr_allfibs was applied before interface address > added? Sorry, I messed it up. Forgot that on my production systems (where I tested), / is read-only with /etc as union-mount. Adding net.add_addr_allfibs=3D0 to the correct sysctl.conf made the inet routing table stay empty. But unfortunately not the inet6 routing table :-( So I still need to delete iface routes for my jail setups, hence need to revert r248895. For those having similar problems, here's how I currently solve my jail setups: jail.conf: jail { allow.set_hostname; =2E.. exec.fib =3D 1; exec.prestart =3D "/bin/sh /.JAIL$name/etc/rc.jails_fibprepare -f= 1 -i inop"; interface =3D inop; =2E.. =E2=80=93=E2=80=93=E2=80=93 rc.jails_fibprepare : #!/bin/sh # format FIB for JAIL usage (remove all but own interface routes) # Does only work if on FreeBSD-9.2 if r248895 was reverted, since deleting iface routes is prohibited by default. # TODO: extend jail (8) and jail.conf for routing parameters and delete this ugly hack! # TODO: Do it the other way, not deleleting, but adding if "sysctl net.add_addr_allfibs=3D0". # Last edited: 20140605.0 _help(){ echo "Usage: rc.jails_fibprepare -f FIBNUM -i IFACENAME [-4 defaultrouterIPv4] [-6 defaultrouterIPv6] [-h]" if [ "X$1" !=3D "X" ]; then if [ "$1" =3D "-h" ]; then echo "Prepare routing tabel of specified FIB for jail usage." echo "This removes all iface routes not belonging to own interface"= echo "and sets default route(s) if specified or automatically, if" echo "iface used is the same where fib 0 has set the default gatewa= y." echo " -f: FIBNUM is the number of the fib whose routing table will be altered." echo " -i: IFACENAME is the name of the interface we have our IP on." echo " -4: IP (v4) of the defaultrouter." echo " -6: IP (v6) of the defaultrouter." echo " -h: This help" echo else echo "ERROR:" echo " $1" echo exit 1 fi else echo "Type \"rc.jails_fibprepare -h\" for more help." exit 1 fi exit 0 } _find_unwanted_destinations(){ # first, generate complete destination lists (separate for v4+v6) dest4list=3D`setfib ${fibnum} netstat -f inet -nr | grep -E '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | cut -s -d ' ' -f 1` dest6list=3D`setfib ${fibnum} netstat -f inet6 -nr | grep -E '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | cut -s -d ' ' -f 1` # Create lists with wanted destinations (separate for v4+v6) for ifn in ${ifnames}; do link=3D`setfib ${fibnum} netstat -I ${ifn} | sed -n -E 's/^[[:print:]]+<[lL](ink#[[:digit:]]{1,2})>[[:print:]]+$/l\1/p'` dest4wanted=3D"`setfib ${fibnum} netstat -f inet -nr | grep -E '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' ' -f 1` ${dest4wanted:-}" dest6wanted=3D"`setfib ${fibnum} netstat -f inet6 -nr | grep -E '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' ' -f 1` ${dest6wanted:-}" done # remove wanted destinations from v4 list for dest in ${dest4wanted}; do dest4list=3D"`echo ${dest4list} | sed -E 's,'"${dest}"' *,,'`" done # remove wanted destinations from v6 list for dest in ${dest6wanted}; do dest6list=3D"`echo ${dest6list} | sed -E 's,'"${dest}"' *,,'`" done } _clean_fib(){ _find_unwanted_destinations || return 1 # extract default gateway IPv4 if it's on one of our interfaces and none is set already for ifn in ${ifnames}; do if [ "X${dv4gw}" =3D "X" ]; then dv4gw=3D"`netstat -f inet -nr | sed -n -E 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin= t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" fi done # extract default gateway IPv6 if it's on one of our interfaces and none is set already for ifn in ${ifnames}; do if [ "X${dv6gw}" =3D "X" ]; then dv6gw=3D"`netstat -f inet6 -nr | sed -n -E 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin= t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" fi done # remove v4 destinations for dest in ${dest4list}; do route -q delete -net -inet ${dest} -fib ${fibnum} || return 1 done # remove v6 destinations for dest in ${dest6list}; do route -q delete -net -inet6 ${dest} -fib ${fibnum} || return 1 done # Set v4 defaultrouter if [ "X${dv4gw}" !=3D "X" ]; then route -q add -net -inet default ${dv4gw} -fib ${fibnum} || return 1 fi # Set v6 defaultrouter if [ "X${dv6gw}" !=3D "X" ]; then route -q add -net -inet6 default ${dv6gw} -fib ${fibnum} || return 1 fi } if [ $# -gt 8 ]; then _help "Too many arguments!" else if [ $# -lt 4 ]; then _help "At least \"-f FIBUM\" and \"-i IFACENAME\" is required!" else if ! expr $# % 2 >/dev/null; then while [ $# -gt 0 ]; do case "$1" in -f) if ! setfib ${2} true; then _help "FIBNUM too high!" else fibnum=3D$2 fi ;; -i) if ! ifconfig ${2} >/dev/null 2>&1; then _help "No such interface: \"$2\"" else ifnames=3D"$2 ${ifnames:-}" fi ;; -4) dv4gw=3D"$2";; -6) dv6gw=3D"$2";; -h|*) _help "$1" esac shift 2 done _clean_fib && exit 0 else _help "Wrong number of arguments ($#), only even numbers can be valid!" fi fi fi exit 1 =E2=80=93=E2=80=93=E2=80=93 r248895-revert patch against 10.1: --- src/sys/net/if.c 2014-10-06 12:56:27.000000000 +0200 +++ src/sys/net/if.c 2014-10-13 10:47:51.000000000 +0200 @@ -1371,8 +1371,7 @@ return (0); =20 err =3D rtrequest_fib(RTM_DELETE, rt_key(rt), rt->rt_gateway, - rt_mask(rt), - rt->rt_flags|RTF_RNH_LOCKED|RTF_PINNED, + rt_mask(rt), rt->rt_flags|RTF_RNH_LOCKED, (struct rtentry **) NULL, rt->rt_fibnum); if (err) { log(LOG_WARNING, "if_rtdel: error %d\n", err); --- src/sys/net/route.c 2014-10-06 12:56:27.000000000 +0200 +++ src/sys/net/route.c 2014-10-13 10:47:51.000000000 +0200 @@ -1210,14 +1210,6 @@ error =3D 0; } #endif - if ((flags & RTF_PINNED) =3D=3D 0) { - /* Check if target route can be deleted */ - rt =3D (struct rtentry *)rnh->rnh_lookup(dst, - netmask, rnh); - if ((rt !=3D NULL) && (rt->rt_flags & RTF_PINNED)) - senderr(EADDRINUSE); - } - /* * Remove the item from the tree and return it. * Complain if it is not there and do no more processing. @@ -1521,7 +1513,6 @@ int didwork =3D 0; int a_failure =3D 0; static struct sockaddr_dl null_sdl =3D {sizeof(null_sdl), AF_LINK}; - struct radix_node_head *rnh; =20 if (flags & RTF_HOST) { dst =3D ifa->ifa_dstaddr; @@ -1580,6 +1571,7 @@ */ for ( fibnum =3D startfib; fibnum <=3D endfib; fibnum++) { if (cmd =3D=3D RTM_DELETE) { + struct radix_node_head *rnh; struct radix_node *rn; /* * Look up an rtentry that is in the routing tree and @@ -1626,8 +1618,7 @@ */ bzero((caddr_t)&info, sizeof(info)); info.rti_ifa =3D ifa; - info.rti_flags =3D flags | - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; + info.rti_flags =3D flags | (ifa->ifa_flags & ~IFA_RTSELF); info.rti_info[RTAX_DST] =3D dst; /* * doing this for compatibility reasons @@ -1639,33 +1630,6 @@ info.rti_info[RTAX_GATEWAY] =3D ifa->ifa_addr; info.rti_info[RTAX_NETMASK] =3D netmask; error =3D rtrequest1_fib(cmd, &info, &rt, fibnum); - - if ((error =3D=3D EEXIST) && (cmd =3D=3D RTM_ADD)) { - /* - * Interface route addition failed. - * Atomically delete current prefix generating - * RTM_DELETE message, and retry adding - * interface prefix. - */ - rnh =3D rt_tables_get_rnh(fibnum, dst->sa_family); - RADIX_NODE_HEAD_LOCK(rnh); - - /* Delete old prefix */ - info.rti_ifa =3D NULL; - info.rti_flags =3D RTF_RNH_LOCKED; - - error =3D rtrequest1_fib(RTM_DELETE, &info, NULL, fibnum); - if (error =3D=3D 0) { - info.rti_ifa =3D ifa; - info.rti_flags =3D flags | RTF_RNH_LOCKED | - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; - error =3D rtrequest1_fib(cmd, &info, &rt, fibnum); - } - - RADIX_NODE_HEAD_UNLOCK(rnh); - } - - if (error =3D=3D 0 && rt !=3D NULL) { /* * notify any listening routing agents of the change --- src/sys/net/route.h 2014-10-06 12:56:27.000000000 +0200 +++ src/sys/net/route.h 2014-10-13 10:43:59.000000000 +0200 @@ -148,7 +148,7 @@ /* 0x20000 unused, was RTF_WASCLONED */ #define RTF_PROTO3 0x40000 /* protocol specific routing flag *= / /* 0x80000 unused */ -#define RTF_PINNED 0x100000 /* route is immutable */ +#define RTF_PINNED 0x100000 /* future use (route is immutable, startintg with r248895) */ #define RTF_LOCAL 0x200000 /* route represents a local address= */ #define RTF_BROADCAST 0x400000 /* route represents a bcast address */ #define RTF_MULTICAST 0x800000 /* route represents a mcast address */ --------------enigDFEC5242399331AD23FB8855 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlQ7mHMACgkQLDqVQ9VXb8gOegCfXiznyHCmkyRMosVBO5uIUlzB 2yQAoKWEezWtKKwXzoBveGim6cb/E6y8 =10vS -----END PGP SIGNATURE----- --------------enigDFEC5242399331AD23FB8855-- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 09:22:46 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ADAA1EB1; Mon, 13 Oct 2014 09:22:46 +0000 (UTC) Received: from mail.ipfw.ru (mail.ipfw.ru [IPv6:2a01:4f8:120:6141::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2DD9D925; Mon, 13 Oct 2014 09:22:46 +0000 (UTC) Received: from [2a02:6b8:0:401:222:4dff:fe50:cd2f] (helo=ptichko.yndx.net) by mail.ipfw.ru with esmtpsa (TLSv1:DHE-RSA-AES128-SHA:128) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1XdXqK-000FRi-Bp; Mon, 13 Oct 2014 09:06:48 +0400 Message-ID: <543B998D.2020003@FreeBSD.org> Date: Mon, 13 Oct 2014 13:21:17 +0400 From: "Alexander V. Chernikov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Harald Schmalzbauer Subject: Re: Deleting IPv4 iface-routes from extra FIBs References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> <543B9075.2000102@FreeBSD.org> <543B9873.3040605@omnilan.de> In-Reply-To: <543B9873.3040605@omnilan.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-net@freebsd.org" , Julian Elischer , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 09:22:46 -0000 On 13.10.2014 13:16, Harald Schmalzbauer wrote: > Bezüglich Alexander V. Chernikov's Nachricht vom 13.10.2014 10:42 > (localtime): >> On 13.10.2014 12:35, Harald Schmalzbauer wrote: >>> Bezüglich Julian Elischer's Nachricht vom 23.04.2014 09:55 >>> (localtime): > ... >>>> yes, we made two behaviours. >>>> Add interface routes to all active FIBS or only add them to the first >>>> fib and let the user populate other fibs as needed. >>>> It appears you want the second behaviour, so I suggest you use that >>>> option and set up all your routes manually. >>> Hello, >>> >>> last time I had the iface-route problem, I just reverted r248895 (for >>> 9.3). There was inconsitent behaviour with v6 iface routes and >>> net.add_addr_allfibs=0. >>> Now I checked with 10.1 ans it seems net.add_addr_allfibs=0 doesn't work >>> any more: >>> netstat -f inet -nr >>> Routing tables >>> >>> Internet: >>> Destination Gateway Flags Netif Expire >>> default 172.21.32.1 UGS egn >>> 127.0.0.1 link#2 UH lo0 >>> 172.21.32.0/19 link#1 U egn >>> 172.21.35.1 link#1 UHS lo0 >>> >>> netstat -F 1 -f inet -nr >>> Routing tables (fib: 1) >>> >>> Internet: >>> Destination Gateway Flags Netif Expire >>> 127.0.0.1 link#2 UH lo0 >>> 172.21.32.0/19 link#1 U egn >>> >>> 'sysctl net.add_addr_allfibs' >>> net.add_addr_allfibs: 0 >> Are you sure net.add_addr_allfibs was applied before interface address >> added? > Sorry, I messed it up. Forgot that on my production systems (where I > tested), / is read-only with /etc as union-mount. > Adding net.add_addr_allfibs=0 to the correct sysctl.conf made the inet > routing table stay empty. > > But unfortunately not the inet6 routing table :-( > So I still need to delete iface routes for my jail setups, hence need to > revert r248895. Hm. If the problem happens with inet6 routes only, why do you need to revert r248895 ? > > Strage thing is that 'rcorder' shows nothing iface related before > mountcritlocal, where I resource /etc/rc.d, so the > 'net.add_addr_allfibs' in my union-mounted sysctl.conf should work!?! > But that's my homemade problem ;-) /> > > For those having similar problems, here's how I currently solve my jail > setups: > > jail.conf: > > jail { > allow.set_hostname; > ... > exec.fib = 1; > exec.prestart = "/bin/sh /.JAIL$name/etc/rc.jails_fibprepare -f > 1 -i inop"; > interface = inop; > ... > > ––– > rc.jails_fibprepare : > > #!/bin/sh > # format FIB for JAIL usage (remove all but own interface routes) > # Does only work if on FreeBSD-9.2 if r248895 was reverted, since > deleting iface routes is prohibited by default. > # TODO: extend jail (8) and jail.conf for routing parameters and delete > this ugly hack! > # TODO: Do it the other way, not deleleting, but adding if "sysctl > net.add_addr_allfibs=0". > # Last edited: 20140605.0 > > > _help(){ > echo "Usage: rc.jails_fibprepare -f FIBNUM -i IFACENAME [-4 > defaultrouterIPv4] [-6 defaultrouterIPv6] [-h]" > if [ "X$1" != "X" ]; then > if [ "$1" = "-h" ]; then > echo "Prepare routing tabel of specified FIB for jail usage." > echo "This removes all iface routes not belonging to own interface" > echo "and sets default route(s) if specified or automatically, if" > echo "iface used is the same where fib 0 has set the default gateway." > echo " -f: FIBNUM is the number of the fib whose routing > table will be altered." > echo " -i: IFACENAME is the name of the interface we have > our IP on." > echo " -4: IP (v4) of the defaultrouter." > echo " -6: IP (v6) of the defaultrouter." > echo " -h: This help" > echo > else > echo "ERROR:" > echo " $1" > echo > exit 1 > fi > else > echo "Type \"rc.jails_fibprepare -h\" for more help." > exit 1 > fi > exit 0 > } > > _find_unwanted_destinations(){ > # first, generate complete destination lists (separate for v4+v6) > dest4list=`setfib ${fibnum} netstat -f inet -nr | grep -E > '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | > cut -s -d ' ' -f 1` > dest6list=`setfib ${fibnum} netstat -f inet6 -nr | grep -E > '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | > cut -s -d ' ' -f 1` > # Create lists with wanted destinations (separate for v4+v6) > for ifn in ${ifnames}; do > link=`setfib ${fibnum} netstat -I ${ifn} | sed -n -E > 's/^[[:print:]]+<[lL](ink#[[:digit:]]{1,2})>[[:print:]]+$/l\1/p'` > dest4wanted="`setfib ${fibnum} netstat -f inet -nr | grep -E > '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d '' > -f 1` ${dest4wanted:-}" > dest6wanted="`setfib ${fibnum} netstat -f inet6 -nr | grep -E > '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d '' > -f 1` ${dest6wanted:-}" > done > # remove wanted destinations from v4 list > for dest in ${dest4wanted}; do > dest4list="`echo ${dest4list} | sed -E 's,'"${dest}"' *,,'`" > done > # remove wanted destinations from v6 list > for dest in ${dest6wanted}; do > dest6list="`echo ${dest6list} | sed -E 's,'"${dest}"' *,,'`" > done > } > > _clean_fib(){ > _find_unwanted_destinations || return 1 > # extract default gateway IPv4 if it's on one of our interfaces and > none is set already > for ifn in ${ifnames}; do > if [ "X${dv4gw}" = "X" ]; then > dv4gw="`netstat -f inet -nr | sed -n -E > 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:print:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" > fi > done > # extract default gateway IPv6 if it's on one of our interfaces and > none is set already > for ifn in ${ifnames}; do > if [ "X${dv6gw}" = "X" ]; then > dv6gw="`netstat -f inet6 -nr | sed -n -E > 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:print:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" > fi > done > # remove v4 destinations > for dest in ${dest4list}; do > route -q delete -net -inet ${dest} -fib ${fibnum} || return 1 > done > # remove v6 destinations > for dest in ${dest6list}; do > route -q delete -net -inet6 ${dest} -fib ${fibnum} || return 1 > done > # Set v4 defaultrouter > if [ "X${dv4gw}" != "X" ]; then > route -q add -net -inet default ${dv4gw} -fib ${fibnum} || return 1 > fi > # Set v6 defaultrouter > if [ "X${dv6gw}" != "X" ]; then > route -q add -net -inet6 default ${dv6gw} -fib ${fibnum} || return 1 > fi > } > > if [ $# -gt 8 ]; then > _help "Too many arguments!" > else > if [ $# -lt 4 ]; then > _help "At least \"-f FIBUM\" and \"-i IFACENAME\" is required!" > else > if ! expr $# % 2 >/dev/null; then > while [ $# -gt 0 ]; do > case "$1" in > -f) if ! setfib ${2} true; then > _help "FIBNUM too high!" > else > fibnum=$2 > fi > ;; > -i) if ! ifconfig ${2} >/dev/null 2>&1; then > _help "No such interface: \"$2\"" > else > ifnames="$2 ${ifnames:-}" > fi > ;; > -4) dv4gw="$2";; > -6) dv6gw="$2";; > -h|*) _help "$1" > esac > shift 2 > done > _clean_fib && exit 0 > else > _help "Wrong number of arguments ($#), only even numbers can be > valid!" > fi > fi > fi > exit 1 > > ––– > r248895-revert patch against 10.1: > > --- src/sys/net/if.c 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/if.c 2014-10-13 10:47:51.000000000 +0200 > @@ -1371,8 +1371,7 @@ > return (0); > > err = rtrequest_fib(RTM_DELETE, rt_key(rt), rt->rt_gateway, > - rt_mask(rt), > - rt->rt_flags|RTF_RNH_LOCKED|RTF_PINNED, > + rt_mask(rt), rt->rt_flags|RTF_RNH_LOCKED, > (struct rtentry **) NULL, rt->rt_fibnum); > if (err) { > log(LOG_WARNING, "if_rtdel: error %d\n", err); > --- src/sys/net/route.c 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/route.c 2014-10-13 10:47:51.000000000 +0200 > @@ -1210,14 +1210,6 @@ > error = 0; > } > #endif > - if ((flags & RTF_PINNED) == 0) { > - /* Check if target route can be deleted */ > - rt = (struct rtentry *)rnh->rnh_lookup(dst, > - netmask, rnh); > - if ((rt != NULL) && (rt->rt_flags & RTF_PINNED)) > - senderr(EADDRINUSE); > - } > - > /* > * Remove the item from the tree and return it. > * Complain if it is not there and do no more processing. > @@ -1521,7 +1513,6 @@ > int didwork = 0; > int a_failure = 0; > static struct sockaddr_dl null_sdl = {sizeof(null_sdl), AF_LINK}; > - struct radix_node_head *rnh; > > if (flags & RTF_HOST) { > dst = ifa->ifa_dstaddr; > @@ -1580,6 +1571,7 @@ > */ > for ( fibnum = startfib; fibnum <= endfib; fibnum++) { > if (cmd == RTM_DELETE) { > + struct radix_node_head *rnh; > struct radix_node *rn; > /* > * Look up an rtentry that is in the routing tree and > @@ -1626,8 +1618,7 @@ > */ > bzero((caddr_t)&info, sizeof(info)); > info.rti_ifa = ifa; > - info.rti_flags = flags | > - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; > + info.rti_flags = flags | (ifa->ifa_flags & ~IFA_RTSELF); > info.rti_info[RTAX_DST] = dst; > /* > * doing this for compatibility reasons > @@ -1639,33 +1630,6 @@ > info.rti_info[RTAX_GATEWAY] = ifa->ifa_addr; > info.rti_info[RTAX_NETMASK] = netmask; > error = rtrequest1_fib(cmd, &info, &rt, fibnum); > - > - if ((error == EEXIST) && (cmd == RTM_ADD)) { > - /* > - * Interface route addition failed. > - * Atomically delete current prefix generating > - * RTM_DELETE message, and retry adding > - * interface prefix. > - */ > - rnh = rt_tables_get_rnh(fibnum, dst->sa_family); > - RADIX_NODE_HEAD_LOCK(rnh); > - > - /* Delete old prefix */ > - info.rti_ifa = NULL; > - info.rti_flags = RTF_RNH_LOCKED; > - > - error = rtrequest1_fib(RTM_DELETE, &info, NULL, fibnum); > - if (error == 0) { > - info.rti_ifa = ifa; > - info.rti_flags = flags | RTF_RNH_LOCKED | > - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; > - error = rtrequest1_fib(cmd, &info, &rt, fibnum); > - } > - > - RADIX_NODE_HEAD_UNLOCK(rnh); > - } > - > - > if (error == 0 && rt != NULL) { > /* > * notify any listening routing agents of the change > --- src/sys/net/route.h 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/route.h 2014-10-13 10:43:59.000000000 +0200 > @@ -148,7 +148,7 @@ > /* 0x20000 unused, was RTF_WASCLONED */ > #define RTF_PROTO3 0x40000 /* protocol specific routing flag */ > /* 0x80000 unused */ > -#define RTF_PINNED 0x100000 /* route is immutable */ > +#define RTF_PINNED 0x100000 /* future use (route is immutable, > startintg with r248895) */ > #define RTF_LOCAL 0x200000 /* route represents a local address */ > #define RTF_BROADCAST 0x400000 /* route represents a bcast > address */ > #define RTF_MULTICAST 0x800000 /* route represents a mcast > address */ > > From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 09:42:12 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B8DC82F8; Mon, 13 Oct 2014 09:42:12 +0000 (UTC) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3E557AE1; Mon, 13 Oct 2014 09:42:12 +0000 (UTC) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id s9D9gAhu042272; Mon, 13 Oct 2014 11:42:10 +0200 (CEST) (envelope-from h.schmalzbauer@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id A81DA3053; Mon, 13 Oct 2014 11:42:09 +0200 (CEST) Message-ID: <543B9E70.9060609@omnilan.de> Date: Mon, 13 Oct 2014 11:42:08 +0200 From: Harald Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: "Alexander V. Chernikov" Subject: Re: Deleting IPv4 iface-routes from extra FIBs References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> <543B9075.2000102@FreeBSD.org> <543B9873.3040605@omnilan.de> <543B998D.2020003@FreeBSD.org> In-Reply-To: <543B998D.2020003@FreeBSD.org> X-Enigmail-Version: 1.1.2 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig81B20D416B47109DFDD66864" X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Mon, 13 Oct 2014 11:42:10 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) Cc: "freebsd-net@freebsd.org" , Julian Elischer , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 09:42:12 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig81B20D416B47109DFDD66864 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Bez=C3=BCglich Alexander V. Chernikov's Nachricht vom 13.10.2014 11:21 (localtime): > On 13.10.2014 13:16, Harald Schmalzbauer wrote: >> Bez=C3=BCglich Alexander V. Chernikov's Nachricht vom 13.10.2014 10:= 42 >> (localtime): >>> On 13.10.2014 12:35, Harald Schmalzbauer wrote: >>>> Bez=C3=BCglich Julian Elischer's Nachricht vom 23.04.2014 09:55 >>>> (localtime): >> ... >>>>> yes, we made two behaviours. >>>>> Add interface routes to all active FIBS or only add them to the fir= st >>>>> fib and let the user populate other fibs as needed. >>>>> It appears you want the second behaviour, so I suggest you use that= >>>>> option and set up all your routes manually. >>>> Hello, >>>> >>>> last time I had the iface-route problem, I just reverted r248895 (fo= r >>>> 9.3). There was inconsitent behaviour with v6 iface routes and >>>> net.add_addr_allfibs=3D0. >>>> Now I checked with 10.1 ans it seems net.add_addr_allfibs=3D0 doesn'= t >>>> work >>>> any more: >>>> netstat -f inet -nr >>>> Routing tables >>>> >>>> Internet: >>>> Destination Gateway Flags Netif Expire >>>> default 172.21.32.1 UGS egn >>>> 127.0.0.1 link#2 UH lo0 >>>> 172.21.32.0/19 link#1 U egn >>>> 172.21.35.1 link#1 UHS lo0 >>>> >>>> netstat -F 1 -f inet -nr >>>> Routing tables (fib: 1) >>>> >>>> Internet: >>>> Destination Gateway Flags Netif Expire >>>> 127.0.0.1 link#2 UH lo0 >>>> 172.21.32.0/19 link#1 U egn >>>> >>>> 'sysctl net.add_addr_allfibs' >>>> net.add_addr_allfibs: 0 >>> Are you sure net.add_addr_allfibs was applied before interface addres= s >>> added? >> Sorry, I messed it up. Forgot that on my production systems (where I >> tested), / is read-only with /etc as union-mount. >> Adding net.add_addr_allfibs=3D0 to the correct sysctl.conf made the in= et >> routing table stay empty. >> >> But unfortunately not the inet6 routing table :-( >> So I still need to delete iface routes for my jail setups, hence need = to >> revert r248895. > Hm. If the problem happens with inet6 routes only, why do you need to > revert r248895 ?=20 For consistency. Either I populate own iface-routes for both, inet and inet6, or I clean both. The latter is what my script has been doing for some time (I think I wrote it when I tested 9.1-RC), so for me it's much less effort to make my script working by reverting r248895 instead of adding another one which cares about inet (v4) only (for the moment). Probably net.add_addr_allfibs will also influence inet6 routing as well in the future, then I'll redo my rc.jails_fibprepare. Thanks, -Harry --------------enig81B20D416B47109DFDD66864 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAlQ7nnEACgkQLDqVQ9VXb8h86gCgr59GmiQsbjteXxN5zlvKL6cU CZsAoKEz0GhkZNIR5a5iqi1Q88+QwFPy =8eck -----END PGP SIGNATURE----- --------------enig81B20D416B47109DFDD66864-- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 15:23:58 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7602D109; Mon, 13 Oct 2014 15:23:58 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B06D518C; Mon, 13 Oct 2014 15:23:57 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id n3so7736713wiv.15 for ; Mon, 13 Oct 2014 08:23:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=US0Th29h9nJaK8mgK2lxpU+314KEeLr15OP9LiBzylE=; b=FOkT+adxgLpmP1at8zFh2S+mU/ugHTsfXBexvKQZ/COhoF/uaXOPnTtwTS/oA9Wlnf yTf3DiVm30v2zuydUS+vhxa8s4Fu1ESbumQW1Qj91pxLaSfH+GI8eoNKi8f31wi7lgEb oyt55TEZz9bYQg9BwyA6KSWqQVb6re8vQQTfQLJWlld6wdY251GS6NUNdnd+GoJKmPwt cxBHvbjpeDqmPqESlNzyJpUklTE45XyWa76xeQkRt1ic3Dc8NCSTyT4Dd8cD1WCO8Dyf ekcX2bs7U7DiSA7iPBNyQ36jBvCntCWlMf3K9xcUhWRlU34+TGTLNYFD4HnNzYvOwh1Y dmsQ== MIME-Version: 1.0 X-Received: by 10.180.72.45 with SMTP id a13mr1498776wiv.50.1413213835679; Mon, 13 Oct 2014 08:23:55 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.220.227 with HTTP; Mon, 13 Oct 2014 08:23:55 -0700 (PDT) In-Reply-To: <543B9873.3040605@omnilan.de> References: <53569ABA.60007@omnilan.de> <535771F3.4070007@freebsd.org> <543B8ED5.6040206@omnilan.de> <543B9075.2000102@FreeBSD.org> <543B9873.3040605@omnilan.de> Date: Mon, 13 Oct 2014 09:23:55 -0600 X-Google-Sender-Auth: 5fxoXsJaLCIZPcoeGnhdg96ydaI Message-ID: Subject: Re: Deleting IPv4 iface-routes from extra FIBs From: Alan Somers To: Harald Schmalzbauer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-net@freebsd.org" , "Alexander V. Chernikov" , FreeBSD X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 15:23:58 -0000 On Mon, Oct 13, 2014 at 3:16 AM, Harald Schmalzbauer wrote: > Bez=C3=BCglich Alexander V. Chernikov's Nachricht vom 13.10.2014 10:42 > (localtime): >> On 13.10.2014 12:35, Harald Schmalzbauer wrote: >>> Bez=C3=BCglich Julian Elischer's Nachricht vom 23.04.2014 09:55 >>> (localtime): > ... >>>> yes, we made two behaviours. >>>> Add interface routes to all active FIBS or only add them to the first >>>> fib and let the user populate other fibs as needed. >>>> It appears you want the second behaviour, so I suggest you use that >>>> option and set up all your routes manually. >>> Hello, >>> >>> last time I had the iface-route problem, I just reverted r248895 (for >>> 9.3). There was inconsitent behaviour with v6 iface routes and >>> net.add_addr_allfibs=3D0. >>> Now I checked with 10.1 ans it seems net.add_addr_allfibs=3D0 doesn't w= ork >>> any more: >>> netstat -f inet -nr >>> Routing tables >>> >>> Internet: >>> Destination Gateway Flags Netif Expire >>> default 172.21.32.1 UGS egn >>> 127.0.0.1 link#2 UH lo0 >>> 172.21.32.0/19 link#1 U egn >>> 172.21.35.1 link#1 UHS lo0 >>> >>> netstat -F 1 -f inet -nr >>> Routing tables (fib: 1) >>> >>> Internet: >>> Destination Gateway Flags Netif Expire >>> 127.0.0.1 link#2 UH lo0 >>> 172.21.32.0/19 link#1 U egn >>> >>> 'sysctl net.add_addr_allfibs' >>> net.add_addr_allfibs: 0 >> Are you sure net.add_addr_allfibs was applied before interface address >> added? > > Sorry, I messed it up. Forgot that on my production systems (where I > tested), / is read-only with /etc as union-mount. > Adding net.add_addr_allfibs=3D0 to the correct sysctl.conf made the inet > routing table stay empty. > > But unfortunately not the inet6 routing table :-( > So I still need to delete iface routes for my jail setups, hence need to > revert r248895. What do your ipv6 routing tables look like when the sysctl is set correctly and 248895 is in place? > > Strage thing is that 'rcorder' shows nothing iface related before > mountcritlocal, where I resource /etc/rc.d, so the > 'net.add_addr_allfibs' in my union-mounted sysctl.conf should work!?! > But that's my homemade problem ;-) /> > > For those having similar problems, here's how I currently solve my jail > setups: > > jail.conf: > > jail { > allow.set_hostname; > ... > exec.fib =3D 1; > exec.prestart =3D "/bin/sh /.JAIL$name/etc/rc.jails_fibprepare -f > 1 -i inop"; > interface =3D inop; > ... > > =E2=80=93=E2=80=93=E2=80=93 > rc.jails_fibprepare : > > #!/bin/sh > # format FIB for JAIL usage (remove all but own interface routes) > # Does only work if on FreeBSD-9.2 if r248895 was reverted, since > deleting iface routes is prohibited by default. > # TODO: extend jail (8) and jail.conf for routing parameters and delete > this ugly hack! > # TODO: Do it the other way, not deleleting, but adding if "sysctl > net.add_addr_allfibs=3D0". > # Last edited: 20140605.0 > > > _help(){ > echo "Usage: rc.jails_fibprepare -f FIBNUM -i IFACENAME [-4 > defaultrouterIPv4] [-6 defaultrouterIPv6] [-h]" > if [ "X$1" !=3D "X" ]; then > if [ "$1" =3D "-h" ]; then > echo "Prepare routing tabel of specified FIB for jail usage." > echo "This removes all iface routes not belonging to own interface" > echo "and sets default route(s) if specified or automatically, if" > echo "iface used is the same where fib 0 has set the default gatewa= y." > echo " -f: FIBNUM is the number of the fib whose routing > table will be altered." > echo " -i: IFACENAME is the name of the interface we have > our IP on." > echo " -4: IP (v4) of the defaultrouter." > echo " -6: IP (v6) of the defaultrouter." > echo " -h: This help" > echo > else > echo "ERROR:" > echo " $1" > echo > exit 1 > fi > else > echo "Type \"rc.jails_fibprepare -h\" for more help." > exit 1 > fi > exit 0 > } > > _find_unwanted_destinations(){ > # first, generate complete destination lists (separate for v4+v6) > dest4list=3D`setfib ${fibnum} netstat -f inet -nr | grep -E > '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | > cut -s -d ' ' -f 1` > dest6list=3D`setfib ${fibnum} netstat -f inet6 -nr | grep -E > '^[[:print:]]+(%[[:alnum:].]+|[[:digit:]])[[:blank:]]+U[[:print:]]+$' | > cut -s -d ' ' -f 1` > # Create lists with wanted destinations (separate for v4+v6) > for ifn in ${ifnames}; do > link=3D`setfib ${fibnum} netstat -I ${ifn} | sed -n -E > 's/^[[:print:]]+<[lL](ink#[[:digit:]]{1,2})>[[:print:]]+$/l\1/p'` > dest4wanted=3D"`setfib ${fibnum} netstat -f inet -nr | grep -E > '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' ' > -f 1` ${dest4wanted:-}" > dest6wanted=3D"`setfib ${fibnum} netstat -f inet6 -nr | grep -E > '^[^[:blank:]]+[[:blank:]]+'"${link}"'[[:blank:]]+.*$' | cut -s -d ' ' > -f 1` ${dest6wanted:-}" > done > # remove wanted destinations from v4 list > for dest in ${dest4wanted}; do > dest4list=3D"`echo ${dest4list} | sed -E 's,'"${dest}"' *,,'`" > done > # remove wanted destinations from v6 list > for dest in ${dest6wanted}; do > dest6list=3D"`echo ${dest6list} | sed -E 's,'"${dest}"' *,,'`" > done > } > > _clean_fib(){ > _find_unwanted_destinations || return 1 > # extract default gateway IPv4 if it's on one of our interfaces and > none is set already > for ifn in ${ifnames}; do > if [ "X${dv4gw}" =3D "X" ]; then > dv4gw=3D"`netstat -f inet -nr | sed -n -E > 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin= t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" > fi > done > # extract default gateway IPv6 if it's on one of our interfaces and > none is set already > for ifn in ${ifnames}; do > if [ "X${dv6gw}" =3D "X" ]; then > dv6gw=3D"`netstat -f inet6 -nr | sed -n -E > 's/^default[[:print:]]+[[:blank:]]([^[:blank:]]+[.:][^[:blank:]]+)[[:prin= t:]]+[^[:blank:]]+[[:blank:]]+'"${ifn}"'$/\1/p'`" > fi > done > # remove v4 destinations > for dest in ${dest4list}; do > route -q delete -net -inet ${dest} -fib ${fibnum} || return 1 > done > # remove v6 destinations > for dest in ${dest6list}; do > route -q delete -net -inet6 ${dest} -fib ${fibnum} || return 1 > done > # Set v4 defaultrouter > if [ "X${dv4gw}" !=3D "X" ]; then > route -q add -net -inet default ${dv4gw} -fib ${fibnum} || return 1 > fi > # Set v6 defaultrouter > if [ "X${dv6gw}" !=3D "X" ]; then > route -q add -net -inet6 default ${dv6gw} -fib ${fibnum} || return 1 > fi > } > > if [ $# -gt 8 ]; then > _help "Too many arguments!" > else > if [ $# -lt 4 ]; then > _help "At least \"-f FIBUM\" and \"-i IFACENAME\" is required!" > else > if ! expr $# % 2 >/dev/null; then > while [ $# -gt 0 ]; do > case "$1" in > -f) if ! setfib ${2} true; then > _help "FIBNUM too high!" > else > fibnum=3D$2 > fi > ;; > -i) if ! ifconfig ${2} >/dev/null 2>&1; then > _help "No such interface: \"$2\"" > else > ifnames=3D"$2 ${ifnames:-}" > fi > ;; > -4) dv4gw=3D"$2";; > -6) dv6gw=3D"$2";; > -h|*) _help "$1" > esac > shift 2 > done > _clean_fib && exit 0 > else > _help "Wrong number of arguments ($#), only even numbers can be > valid!" > fi > fi > fi > exit 1 > > =E2=80=93=E2=80=93=E2=80=93 > r248895-revert patch against 10.1: > > --- src/sys/net/if.c 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/if.c 2014-10-13 10:47:51.000000000 +0200 > @@ -1371,8 +1371,7 @@ > return (0); > > err =3D rtrequest_fib(RTM_DELETE, rt_key(rt), rt->rt_gateway, > - rt_mask(rt), > - rt->rt_flags|RTF_RNH_LOCKED|RTF_PINNED, > + rt_mask(rt), rt->rt_flags|RTF_RNH_LOCKED, > (struct rtentry **) NULL, rt->rt_fibnum); > if (err) { > log(LOG_WARNING, "if_rtdel: error %d\n", err); > --- src/sys/net/route.c 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/route.c 2014-10-13 10:47:51.000000000 +0200 > @@ -1210,14 +1210,6 @@ > error =3D 0; > } > #endif > - if ((flags & RTF_PINNED) =3D=3D 0) { > - /* Check if target route can be deleted */ > - rt =3D (struct rtentry *)rnh->rnh_lookup(dst, > - netmask, rnh); > - if ((rt !=3D NULL) && (rt->rt_flags & RTF_PINNED)) > - senderr(EADDRINUSE); > - } > - > /* > * Remove the item from the tree and return it. > * Complain if it is not there and do no more processing. > @@ -1521,7 +1513,6 @@ > int didwork =3D 0; > int a_failure =3D 0; > static struct sockaddr_dl null_sdl =3D {sizeof(null_sdl), AF_LINK}; > - struct radix_node_head *rnh; > > if (flags & RTF_HOST) { > dst =3D ifa->ifa_dstaddr; > @@ -1580,6 +1571,7 @@ > */ > for ( fibnum =3D startfib; fibnum <=3D endfib; fibnum++) { > if (cmd =3D=3D RTM_DELETE) { > + struct radix_node_head *rnh; > struct radix_node *rn; > /* > * Look up an rtentry that is in the routing tree and > @@ -1626,8 +1618,7 @@ > */ > bzero((caddr_t)&info, sizeof(info)); > info.rti_ifa =3D ifa; > - info.rti_flags =3D flags | > - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; > + info.rti_flags =3D flags | (ifa->ifa_flags & ~IFA_RTSELF); > info.rti_info[RTAX_DST] =3D dst; > /* > * doing this for compatibility reasons > @@ -1639,33 +1630,6 @@ > info.rti_info[RTAX_GATEWAY] =3D ifa->ifa_addr; > info.rti_info[RTAX_NETMASK] =3D netmask; > error =3D rtrequest1_fib(cmd, &info, &rt, fibnum); > - > - if ((error =3D=3D EEXIST) && (cmd =3D=3D RTM_ADD)) { > - /* > - * Interface route addition failed. > - * Atomically delete current prefix generating > - * RTM_DELETE message, and retry adding > - * interface prefix. > - */ > - rnh =3D rt_tables_get_rnh(fibnum, dst->sa_family); > - RADIX_NODE_HEAD_LOCK(rnh); > - > - /* Delete old prefix */ > - info.rti_ifa =3D NULL; > - info.rti_flags =3D RTF_RNH_LOCKED; > - > - error =3D rtrequest1_fib(RTM_DELETE, &info, NULL, fibnum); > - if (error =3D=3D 0) { > - info.rti_ifa =3D ifa; > - info.rti_flags =3D flags | RTF_RNH_LOCKED | > - (ifa->ifa_flags & ~IFA_RTSELF) | RTF_PINNED; > - error =3D rtrequest1_fib(cmd, &info, &rt, fibnum); > - } > - > - RADIX_NODE_HEAD_UNLOCK(rnh); > - } > - > - > if (error =3D=3D 0 && rt !=3D NULL) { > /* > * notify any listening routing agents of the change > --- src/sys/net/route.h 2014-10-06 12:56:27.000000000 +0200 > +++ src/sys/net/route.h 2014-10-13 10:43:59.000000000 +0200 > @@ -148,7 +148,7 @@ > /* 0x20000 unused, was RTF_WASCLONED */ > #define RTF_PROTO3 0x40000 /* protocol specific routing flag *= / > /* 0x80000 unused */ > -#define RTF_PINNED 0x100000 /* route is immutable */ > +#define RTF_PINNED 0x100000 /* future use (route is immutable, > startintg with r248895) */ > #define RTF_LOCAL 0x200000 /* route represents a local address= */ > #define RTF_BROADCAST 0x400000 /* route represents a bcast > address */ > #define RTF_MULTICAST 0x800000 /* route represents a mcast > address */ > > From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 16:52:49 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from hub.FreeBSD.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 460834F3; Mon, 13 Oct 2014 16:52:48 +0000 (UTC) Date: Mon, 13 Oct 2014 16:52:44 +0000 From: Glen Barber To: freebsd-stable@FreeBSD.org Subject: FreeBSD 10.1-RC2 Now Available Message-ID: <20141013165244.GA61249@hub.FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; x-action=pgp-signed X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event User-Agent: Mutt/1.5.23 (2014-03-12) Cc: FreeBSD Release Engineering Team X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 16:52:49 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 The second RC build of the 10.1-RELEASE release cycle is now available on the FTP servers for the amd64, armv6, i386, ia64, powerpc, powerpc64 and sparc64 architectures. The image checksums follow at the end of this email. Installer images and memory stick images are available here: ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/ If you notice problems you can report them through the Bugzilla PR system or on the -stable mailing list. If you would like to use SVN to do a source based update of an existing system, use the "releng/10.1" branch. A list of changes since 10.0-RELEASE are available here: https://www.freebsd.org/releases/10.1R/relnotes.html Important note to ZFS users on the i386 architecture: A regression has been discovered that affects multi-disk (mirror, raidz-1, raidz-2, etc.) installations that may cause a kernel panic on boot. If using a multi-disk ZFS setup, adding 'options KSTACK_PAGES=4' is suspected to resolve the problem. Please *do* *not* upgrade your system with freebsd-update(8) if using a multi-disk ZFS setup, since this will override the kernel configuration with the GENERIC kernel. Note to consumers of the i386 DVD installer: As result of a package build failure, the xorg-related ports (including gnome2) are not available on the 10.1-RC2 i386 dvd. This will not be an issue for the 10.1-RC3 dvd, nor the 10.1-RELEASE dvd. Some of the changes between 10.1-RC1 and 10.1-RC2 include: o Fix XHCI driver for devices which have more than 15 physical root HUB ports. o Fix old iSCSI initiator to work with new CAM locking. o Fix page length reported for Block Limits VPD page. o Add QCOW v1 & v2 support to mkimg(1). Pre-installed virtual machine images for 10.1-RC2 are also available for amd64 and i386 architectures. The images are located here: ftp://ftp.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/10.1-RC2/ The disk images are available in QCOW2, VHD, VMDK, and raw disk image formats. The image download size is approximately 135 MB, which decompress to a 20GB sparse image. The partition layout is: . 512k - freebsd-boot GPT partition type (bootfs GPT label) . 1GB - freebsd-swap GPT partition type (swapfs GPT label) . ~17GB - freebsd-ufs GPT partition type (rootfs GPT label) Note to consumers of the dvd1.iso image: The packages included on the dvd will not be recognized by bsdconfig(8), and the cause is being investigated. The packages, however, can be installed manually. To install packages from the dvd1.iso installer, create and mount the /dist directory: # mkdir -p /dist # mount -t cd9660 /dev/cd0 /dist Next, install pkg(8) from the DVD: # env REPOS_DIR=/dist/packages/repos \ pkg bootstrap At this point, pkg-install(8) can be used to install additional packages from the DVD. Please note, the REPOS_DIR environment variable should be used each time using the DVD as the package repository, otherwise conflicts with packages from the upstream mirrors may occur when they are fetched. For example, to install Gnome and Xorg, run: # env REPOS_DIR=/dist/packages/repos \ pkg install xorg-server xorg gnome2 [...] The freebsd-update(8) utility supports binary upgrades of amd64 and i386 systems running earlier FreeBSD releases. Systems running earlier FreeBSD releases can upgrade as follows: # freebsd-update upgrade -r 10.1-RC2 During this process, freebsd-update(8) may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly. # freebsd-update install The system must be rebooted with the newly installed kernel before continuing. # shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: # freebsd-update install It is recommended to rebuild and install all applications if possible, especially if upgrading from an earlier FreeBSD release, for example, FreeBSD 8.x. Alternatively, the user can install misc/compat9x and other compatibility libraries, afterwards the system must be rebooted into the new userland: # shutdown -r now Finally, after rebooting, freebsd-update needs to be run again to remove stale files: # freebsd-update install == ISO CHECKSUMS == o 10.1-RC2 amd64 GENERIC: SHA256 (FreeBSD-10.1-RC2-amd64-bootonly.iso) = 29946a1adda034d8f4b2b96ef93128aed10c88361ac657af02246cf25ce71011 SHA256 (FreeBSD-10.1-RC2-amd64-bootonly.iso.xz) = cf8939caadb923b728653a4609751d60f37b064e8094c356b31d4688e46ea157 SHA256 (FreeBSD-10.1-RC2-amd64-disc1.iso) = 17c9ca37e8886c7185bec8e079e24c3673ed6fa390076adfe83be87c016d07c1 SHA256 (FreeBSD-10.1-RC2-amd64-disc1.iso.xz) = 621d14e3a647efc80ae68e0dadda549a39171bdd3f13822cd03ac28bf02fb3c1 SHA256 (FreeBSD-10.1-RC2-amd64-dvd1.iso) = 65aaebf8bdad144ee6c32cf87572f281f93f8c26b3e12176be60b142ab1f5925 SHA256 (FreeBSD-10.1-RC2-amd64-dvd1.iso.xz) = be305d361549f30a26e31a3a8fcd2e73365626873709dd481debd384d83c8fc4 SHA256 (FreeBSD-10.1-RC2-amd64-memstick.img) = 7d8f4082ad3af69900966d85a276226f6b32ca521f54a92e9cceb70034f97900 SHA256 (FreeBSD-10.1-RC2-amd64-memstick.img.xz) = 1c8c1f95157c65d64081c2388c7b5b506fa7582c979f25fa1fd1dbee281109dc SHA256 (FreeBSD-10.1-RC2-amd64-mini-memstick.img) = e424456d8ec67fdc59122f0f99ba8e9dfe107acaf47d10aa8efed812186e7e14 SHA256 (FreeBSD-10.1-RC2-amd64-mini-memstick.img.xz) = 31cb5d666310867bc7ff667ca88d930fa8e19d90623ddc4d7e82f8699f8f27b7 SHA256 (FreeBSD-10.1-RC2-amd64-uefi-bootonly.iso) = 4f9209249ab6651a78dff0de1253c21dfa9d0333a519e4f07d83d498e4594a7b SHA256 (FreeBSD-10.1-RC2-amd64-uefi-bootonly.iso.xz) = aafbc8d01c8988d98087eafbaced85918a17ed50933c661bec31ab1a151ed7dd SHA256 (FreeBSD-10.1-RC2-amd64-uefi-disc1.iso) = 3c9aee2ca224ac938d874c3e15ef909f081be17d45f582e2ef3502d4fbce5899 SHA256 (FreeBSD-10.1-RC2-amd64-uefi-disc1.iso.xz) = 53e7a9cbd4b4c70c53df755e6b84d85538556ed619a5b752065f3a48bbc602ba SHA256 (FreeBSD-10.1-RC2-amd64-uefi-dvd1.iso) = 8af08712d8642dea37e3026780560e05e093dd5a69f793baf176cfbfeda55db3 SHA256 (FreeBSD-10.1-RC2-amd64-uefi-dvd1.iso.xz) = e481d437ef2357a77b253d3572e6f87b3a7fd5a3745eba99fdb6c7350ad8a0fe SHA256 (FreeBSD-10.1-RC2-amd64-uefi-memstick.img) = 99eddee719348ba195f60eb1e66018613483e7cb088b306ae0e1c5e68957ec4e SHA256 (FreeBSD-10.1-RC2-amd64-uefi-memstick.img.xz) = 0695d3d1c668ff72163489fe8d3d0c71e24c6abe6f792e2295524f3b1679f0bf SHA256 (FreeBSD-10.1-RC2-amd64-uefi-mini-memstick.img) = a00cbd4257fc75d1d8ea9a4e59b49d6c70ddebb605672faa6c78d21ac15584eb SHA256 (FreeBSD-10.1-RC2-amd64-uefi-mini-memstick.img.xz) = 0b1d47b8997a78fc82436dc1e5a5553a6ac91ce76f11d4e348cd10c0ebfb5ef5 MD5 (FreeBSD-10.1-RC2-amd64-bootonly.iso) = 260c3870a23507b8c677a4e84f6f316f MD5 (FreeBSD-10.1-RC2-amd64-bootonly.iso.xz) = 343b167301da9f0d540af9cda8d92341 MD5 (FreeBSD-10.1-RC2-amd64-disc1.iso) = 68143c12325deb54e013c5524a2ff599 MD5 (FreeBSD-10.1-RC2-amd64-disc1.iso.xz) = a3f4bcc704da81cb6eac7a9394e5d153 MD5 (FreeBSD-10.1-RC2-amd64-dvd1.iso) = 1fdf05bbbaa101de79c991cc10fd27c0 MD5 (FreeBSD-10.1-RC2-amd64-dvd1.iso.xz) = 6e867a77c412b747a7fe04d3eb446125 MD5 (FreeBSD-10.1-RC2-amd64-memstick.img) = 0e4742f5d6f8e6072ceb56b06196ba81 MD5 (FreeBSD-10.1-RC2-amd64-memstick.img.xz) = 62d59a2e73c74fcdd86e330396b0e8bf MD5 (FreeBSD-10.1-RC2-amd64-mini-memstick.img) = de6806708536ae07218b01821b4dbd90 MD5 (FreeBSD-10.1-RC2-amd64-mini-memstick.img.xz) = 0f3cc2c415aae504bfe9b8c2e77f5859 MD5 (FreeBSD-10.1-RC2-amd64-uefi-bootonly.iso) = 7d871a86e62eb54507217ff99bc9746a MD5 (FreeBSD-10.1-RC2-amd64-uefi-bootonly.iso.xz) = dbbcc0434ce8e362b5317889920b7c74 MD5 (FreeBSD-10.1-RC2-amd64-uefi-disc1.iso) = fefd9fa1083727cdff30e00117fff123 MD5 (FreeBSD-10.1-RC2-amd64-uefi-disc1.iso.xz) = 2ec25d0f0e246904ca57d700973ce54a MD5 (FreeBSD-10.1-RC2-amd64-uefi-dvd1.iso) = 73e8127cd99e166520100982a902b3fc MD5 (FreeBSD-10.1-RC2-amd64-uefi-dvd1.iso.xz) = 27a24354aa9bab47a4a408cfc44d3ac5 MD5 (FreeBSD-10.1-RC2-amd64-uefi-memstick.img) = 00513126431151315ea81bf47e8b4cf2 MD5 (FreeBSD-10.1-RC2-amd64-uefi-memstick.img.xz) = faa5ffb1846cac74abefc19cd7bf7bd2 MD5 (FreeBSD-10.1-RC2-amd64-uefi-mini-memstick.img) = 1d5f8f8a257ee91f74a44fc5046980a7 MD5 (FreeBSD-10.1-RC2-amd64-uefi-mini-memstick.img.xz) = 8b2ae4b7aa8896b9d77b940610058ed2 o 10.1-RC2 i386 GENERIC: SHA256 (FreeBSD-10.1-RC2-i386-bootonly.iso) = 1d7820f05c88a7d76577432930ae065defbd667de4469407f48c57f02e5d9ae9 SHA256 (FreeBSD-10.1-RC2-i386-bootonly.iso.xz) = 35f812b99660425289d65d2b8e36b36b8560286bb9f1129b2b409e2d5b243823 SHA256 (FreeBSD-10.1-RC2-i386-disc1.iso) = 6598fae7fc5565a974a9a3548c91c2e86ad818cbc69d46afd2cb99d0ad1a61c7 SHA256 (FreeBSD-10.1-RC2-i386-disc1.iso.xz) = 115063ea096160ebe900279cec4f4b7ba29f31191604c49865b68db3da6bc59a SHA256 (FreeBSD-10.1-RC2-i386-dvd1.iso) = c25a4132bfca2e411d2fcad28dede237e75838dc8b7eb130b2eae90c1ee233bd SHA256 (FreeBSD-10.1-RC2-i386-dvd1.iso.xz) = 9da1b024910018f15972260adca5af1c8ba021c25e3172f52693768759483ccd SHA256 (FreeBSD-10.1-RC2-i386-memstick.img) = fc9f4bc55a24edc030c628952ed40955a889763236ab89cb765ac88c12ef4db6 SHA256 (FreeBSD-10.1-RC2-i386-memstick.img.xz) = 7fde1d11573e51b55be0518e9ee47aa342f7760575cf5b559dddc86eb84e57c7 SHA256 (FreeBSD-10.1-RC2-i386-mini-memstick.img) = a5eac33c9410324bd43d46fcb4e29f8e82835cecb2f39f9736e81a7e853b0545 SHA256 (FreeBSD-10.1-RC2-i386-mini-memstick.img.xz) = bb3c5e3b88568178330fd23969f4c31a33c1192f53d1fa4616f768fec9d58aeb MD5 (FreeBSD-10.1-RC2-i386-bootonly.iso) = 1b979ae2171fabdffc92ef36553d79a3 MD5 (FreeBSD-10.1-RC2-i386-bootonly.iso.xz) = 20c375bcc2b116e492021daa540bfa55 MD5 (FreeBSD-10.1-RC2-i386-disc1.iso) = 534f490343670e7fc035b840fe85b634 MD5 (FreeBSD-10.1-RC2-i386-disc1.iso.xz) = 66b2a8c2e9c9a1c6bd1df31140bd1e24 MD5 (FreeBSD-10.1-RC2-i386-dvd1.iso) = 1df8dcfd05d22252011006dd6399556a MD5 (FreeBSD-10.1-RC2-i386-dvd1.iso.xz) = 4968536ee233da1bedcbe2bb85bd198b MD5 (FreeBSD-10.1-RC2-i386-memstick.img) = dc5d7c5ec315815781d26bc1b18c40df MD5 (FreeBSD-10.1-RC2-i386-memstick.img.xz) = 909ebc74f12e9fedaef3fd555da9c8b8 MD5 (FreeBSD-10.1-RC2-i386-mini-memstick.img) = 5bac6d4fc1ec41923ed5825544eb7d03 MD5 (FreeBSD-10.1-RC2-i386-mini-memstick.img.xz) = c6ef1acd682121396faf9c34a5c98b94 o 10.1-RC2 ia64 GENERIC: SHA256 (FreeBSD-10.1-RC2-ia64-bootonly.iso) = 4983658f094d61cea9753d9234b81b828cb8e57342a571ce96d1398f0d9c0374 SHA256 (FreeBSD-10.1-RC2-ia64-bootonly.iso.xz) = 55c220be224138dad136273f8f6784d18321401ce05f18889d392bbca5c1bd6c SHA256 (FreeBSD-10.1-RC2-ia64-disc1.iso) = 1341988aa41dd884f413da3e7b7c8a31ff495cc515d1047960a09e2f3e27acea SHA256 (FreeBSD-10.1-RC2-ia64-disc1.iso.xz) = b50537af2bfc27cee12877754adfb6f07869cd8b6bccb6cc9da39312317ce572 SHA256 (FreeBSD-10.1-RC2-ia64-memstick.img) = 5db739c7fc8a671f5cd6a506657f712a4bf5b2cabfb84fda166fa7796adb5ee2 SHA256 (FreeBSD-10.1-RC2-ia64-memstick.img.xz) = 3581b3b4d66d105722681c029650816b8955c221bd699332b71ac019c5c150c2 SHA256 (FreeBSD-10.1-RC2-ia64-mini-memstick.img) = c43bb08f441fa619d043cc7d8eee93b2df684944b06395c88a7d127f4d5317cc SHA256 (FreeBSD-10.1-RC2-ia64-mini-memstick.img.xz) = d6594a83f5552fa5cddeb5b3d35d44403cdc0656242bbc9e3427f0e24b4765fb MD5 (FreeBSD-10.1-RC2-ia64-bootonly.iso) = deb642cba1a62685ec2397586f9375b6 MD5 (FreeBSD-10.1-RC2-ia64-bootonly.iso.xz) = e9e4641670c17d9a90dad56c537d7be2 MD5 (FreeBSD-10.1-RC2-ia64-disc1.iso) = 651e8aa21de30fa355f6408b51467b48 MD5 (FreeBSD-10.1-RC2-ia64-disc1.iso.xz) = c41a0e1a8781aaffb172e274ae31b199 MD5 (FreeBSD-10.1-RC2-ia64-memstick.img) = 97e6fe50ad5693748aff8f594256703a MD5 (FreeBSD-10.1-RC2-ia64-memstick.img.xz) = 1f7b3349fb202260f130fcd18dfe620f MD5 (FreeBSD-10.1-RC2-ia64-mini-memstick.img) = 12db9bb798ff028e56390b5b6b3f90db MD5 (FreeBSD-10.1-RC2-ia64-mini-memstick.img.xz) = 2c27478b2ecbf0850ebefa5aec889f63 o 10.1-RC2 powerpc GENERIC: SHA256 (FreeBSD-10.1-RC2-powerpc-bootonly.iso) = df9187065c973c5d230d705586094fedadafbf3b040330a72d134495d8669374 SHA256 (FreeBSD-10.1-RC2-powerpc-bootonly.iso.xz) = f2b25db77ad2e517f25a307cec6eef664032eb6ed05de8fcde5d8488d1006f0d SHA256 (FreeBSD-10.1-RC2-powerpc-disc1.iso) = f30241116122509e9c2620ad05173abd9a6626b9df37edc91bb13bcbb122c712 SHA256 (FreeBSD-10.1-RC2-powerpc-disc1.iso.xz) = a50ec0d1dc3ecd7d746823a755e87c35e442c34e93698578ee5fefe9b6cffa95 SHA256 (FreeBSD-10.1-RC2-powerpc-memstick.img) = be69aa124898dac14be1d364efc8826505f869d06d464e78beb37d4be4346505 SHA256 (FreeBSD-10.1-RC2-powerpc-memstick.img.xz) = c8543dedf57ea7d03e697673acb1e33609202032d37d3a7b46c73e7649e5f75b SHA256 (FreeBSD-10.1-RC2-powerpc-mini-memstick.img) = ca282f2031315359ed41cf5c22c62b85c0a6b0c07e065b8b08c5ea4702e80e54 SHA256 (FreeBSD-10.1-RC2-powerpc-mini-memstick.img.xz) = aaee6fa01c4a25bf3f0aa7cef9e6be55fc858a7944e7139f947b469cdb8f4fd9 MD5 (FreeBSD-10.1-RC2-powerpc-bootonly.iso) = a18ccbad899c62862c27033493536167 MD5 (FreeBSD-10.1-RC2-powerpc-bootonly.iso.xz) = 99c1df1164f7cec13b6cf2a8c6fc48ba MD5 (FreeBSD-10.1-RC2-powerpc-disc1.iso) = ec8520a6698919a7c3cd453213209eef MD5 (FreeBSD-10.1-RC2-powerpc-disc1.iso.xz) = 4cae1f8ce27978a0a1057680ef6a2e1e MD5 (FreeBSD-10.1-RC2-powerpc-memstick.img) = ac672fbaccb6e78efa7e0c309c700fd1 MD5 (FreeBSD-10.1-RC2-powerpc-memstick.img.xz) = 40923438febc489659e47ecedff52565 MD5 (FreeBSD-10.1-RC2-powerpc-mini-memstick.img) = 43f02f55704ee01511fad8971882e75b MD5 (FreeBSD-10.1-RC2-powerpc-mini-memstick.img.xz) = fb2f4c7b58ed69c2f076b9183bb1393e o 10.1-RC2 powerpc64 GENERIC64: SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-bootonly.iso) = 4ba8ca0164d41e5e443efeff9257395c6de132070a7b04925660508d4e64bc9d SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-bootonly.iso.xz) = 64fff18e3446425816c3e5c28806d99d6e8f320a80d581053ddec2224016917a SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-disc1.iso) = 7638f5e6347c532b18af7f62089bb9dbfc414e9140ca276f39e2a9a11c73d0bf SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-disc1.iso.xz) = e5f7a269ad71b19b4fa7971ebf6d4390f9e5110408063695f7a8f32d1c3deece SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-memstick.img) = b73fdce3c76ca7891c24be8cfc010e685adce642cc7ee2c4fe7192ed841a7475 SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-memstick.img.xz) = c236514b0333a33b4e1844282e6d1d494cad4be485e50ee7b7c2af0de2e735b3 SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-mini-memstick.img) = 44665d14bedabde3e901817b55bd2719b57aa2171b268d5ef33dd0dda6853f08 SHA256 (FreeBSD-10.1-RC2-powerpc-powerpc64-mini-memstick.img.xz) = d001573e42f886e71211e9c0e014e9711663668bf822e6de2c0301aa04750c7b MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-bootonly.iso) = d6db65d412997aee0ba1c8cfea37a6f7 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-bootonly.iso.xz) = eb8a11eab979876818528c9a460cf624 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-disc1.iso) = a5c36b59babac087321f7c6fd22e2c51 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-disc1.iso.xz) = a634f1cfd29c113c78abf4f1d9d505df MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-memstick.img) = 9fcb24fce57c39faedbe8cb8b5d66973 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-memstick.img.xz) = 40c1fa3caf6748062e26c4d39e5aacb9 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-mini-memstick.img) = 4efb562c4290546c4f615ed85e25fb38 MD5 (FreeBSD-10.1-RC2-powerpc-powerpc64-mini-memstick.img.xz) = 192e4b28dada9ee0924875d26fceab90 o 10.1-RC2 sparc64 GENERIC: SHA256 (FreeBSD-10.1-RC2-sparc64-bootonly.iso) = 35419964e9e1f9c1eac6b5eb6aad3b1f065f03885acf8c9f527a8f3b19d9e195 SHA256 (FreeBSD-10.1-RC2-sparc64-bootonly.iso.xz) = 265f6a7b13b02d05f82e518bbf17ff944af3d8e86017f6d8d7aafa486217b476 SHA256 (FreeBSD-10.1-RC2-sparc64-disc1.iso) = c3c8816c011cb869767c6847d0dcb91599c05443e1179a26993ded44f2c59ebf SHA256 (FreeBSD-10.1-RC2-sparc64-disc1.iso.xz) = 7d2ddbde8bd34e4de19d608dad2181c17157c536351b6f8963fbbf460baf2b69 MD5 (FreeBSD-10.1-RC2-sparc64-bootonly.iso) = 58077bcaf525c3a50992b6f0a4dea704 MD5 (FreeBSD-10.1-RC2-sparc64-bootonly.iso.xz) = 3c938c99f592d9473fc6cd096501f474 MD5 (FreeBSD-10.1-RC2-sparc64-disc1.iso) = d1e9338344ea39d24a13951b8058b18b MD5 (FreeBSD-10.1-RC2-sparc64-disc1.iso.xz) = c6dd1614e228d793027594795eac07a3 o 10.1-RC2 armv6 BEAGLEBONE: SHA256 (FreeBSD-10.1-RC2-arm-armv6-BEAGLEBONE.img.bz2) = d97f752b7e00318b73a436015c3b6ef81cc8f71365beaf5498036236faed2ff6 MD5 (FreeBSD-10.1-RC2-arm-armv6-BEAGLEBONE.img.bz2) = cff5db6508c2c262affda5edd91c9f62 o 10.1-RC2 armv6 RPI-B: SHA256 (FreeBSD-10.1-RC2-arm-armv6-RPI-B.img.bz2) = 81d5a17bb7a056f9e33ce50fd01f429f23469e48e2406a036e34f95af3163f02 MD5 (FreeBSD-10.1-RC2-arm-armv6-RPI-B.img.bz2) = 558083568da6deb6e0ba0dedc4daa116 o 10.1-RC2 armv6 PANDABOARD: SHA256 (FreeBSD-10.1-RC2-arm-armv6-PANDABOARD.img.bz2) = e7b6b6561edde9145f8ae1173b88ab8cb6c054e8e403a117c41e9cc2eddc9da5 MD5 (FreeBSD-10.1-RC2-arm-armv6-PANDABOARD.img.bz2) = a26ab9c70cb0e02582e4ce7cd952ca55 o 10.1-RC2 armv6 ZEDBOARD: SHA256 (FreeBSD-10.1-RC2-arm-armv6-ZEDBOARD.img.bz2) = 504bfdfacbb6d1b79516b36df08fa3656f49233d65e8f0314af3e112f4de4d67 MD5 (FreeBSD-10.1-RC2-arm-armv6-ZEDBOARD.img.bz2) = 20a07b13aaff145fbffa5e27c9caabfe == VM IMAGE CHECKSUMS == o 10.1-RC2 amd64: SHA256 (FreeBSD-10.1-RC2-amd64.qcow2.xz) = 758ad632bcaf2fd020dcc7cb280bee5c176fc912120184eca1349ece2aa1bf47 SHA256 (FreeBSD-10.1-RC2-amd64.raw.xz) = 27d60c9539f51c71ff4072e296456d41ca563d6d92ca6e175ccf3babb0695fca SHA256 (FreeBSD-10.1-RC2-amd64.vhd.xz) = 92b31588441f127df99708272b2b427588c2cff9f1cdb58d7626df04d1c97a37 SHA256 (FreeBSD-10.1-RC2-amd64.vmdk.xz) = 146079eb9623d6edcb4db5118a1a08aa7c001b0ab22202be6d2a255473c61c3d MD5 (FreeBSD-10.1-RC2-amd64.qcow2.xz) = eeda5be18e38083485e029f5e196052d MD5 (FreeBSD-10.1-RC2-amd64.raw.xz) = 2320dcccde4ca941f287a496b2b07b7a MD5 (FreeBSD-10.1-RC2-amd64.vhd.xz) = 8e83bd0ab014905d621e7a7d148fa4dd MD5 (FreeBSD-10.1-RC2-amd64.vmdk.xz) = 052d5cb4735f69b3fc9737f682e22507 o 10.1-RC2 i386: SHA256 (FreeBSD-10.1-RC2-i386.qcow2.xz) = 96a7ddeeeedf2f80c611137555e59f441dfdf1827c05391359be7d342342a108 SHA256 (FreeBSD-10.1-RC2-i386.raw.xz) = b0e7e7537cec76265eb9e74dcfb4d91b841253af327c5fb1434151628bf10331 SHA256 (FreeBSD-10.1-RC2-i386.vhd.xz) = 3f7c2066c186d576f2f7484949e2fbf10e1d6d4d7672f709e55a3c425e55d6f9 SHA256 (FreeBSD-10.1-RC2-i386.vmdk.xz) = 9cc11597676b999d7d0d851233afc9e7c9282ab60ec03aeb35472d74f5e54a0d MD5 (FreeBSD-10.1-RC2-i386.qcow2.xz) = e1d63b4ba22f15f161c4b42b15fa2a86 MD5 (FreeBSD-10.1-RC2-i386.raw.xz) = b2e35a2b0871dc9a4862106abec5359b MD5 (FreeBSD-10.1-RC2-i386.vhd.xz) = b1b041d2a257b107d8d32c5f2e930e16 MD5 (FreeBSD-10.1-RC2-i386.vmdk.xz) = 2bf01d3064f9d6a7741d6f2e52200203 Glen Love FreeBSD? Support this and future releases with a donation to the FreeBSD Foundation! https://www.freebsdfoundation.org/donate/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJUPANcAAoJEAMUWKVHj+KT6hEP/jWFWb270Fc/A9bcQNMsbd2C uAZ1oePB/N92dK7OUWRF6BAa/94VHhG8VZo9yRV5tZ3LiVzPqdg/rHFnW4qYW434 tOf8UJ930mytNZuD9SovIJlID31qa0L48AszcTWT1JtsNph3fVHA3kvFZSdSMkoH ZUYgbk886NJzdIWQtRzyORZvwqEEmpFNPNb33Jzb9cPmzS1GWnANMC2ol5LsTKMq prGlK1JP07/cqF0OpGmMW/bxPbhLcQMgBqCwj8SOd7XXKOpQenkaef0MiZN4GemR tF+mYrlQfkN6PRN0g0ep1wayNtBDJvDttu4FnXS0CeC1vvU1GWONGzSqhbKvkNfw VZR7MrsTgflPHdshRLFZ/nmrzfVNB73SWE2pg/H5r3weQfafd7rWgmuOZX1CYZv4 wFPzosfC/FhkET+VZpDk4lhuZNp9w0t7FiFbZ2YXLjqbO1crCIqu1lmp+daj7m2C +12iGWjYPOFjlG38LEYR+PVGw0T7LPixSe6NoL/Wf9jv0383LRAirPnGL91Ojt7d CM12EXNavm7nJ1H3uiUzwstqMBDiaBqFnOoD54rZPIYjYjo4ld9UwY7V5tXFdPe7 pkY0dFvUHTGzzojU3Iu90l90s7QIrGnsJXJydFkIDPHKos5U/Bmkf0M6ePcrhaSe IbL6zeH4NmK1LVxa11+y =bagC -----END PGP SIGNATURE----- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 18:40:12 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC8CF7D1; Mon, 13 Oct 2014 18:40:12 +0000 (UTC) Received: from mail-yh0-x22d.google.com (mail-yh0-x22d.google.com [IPv6:2607:f8b0:4002:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CDE0D6E; Mon, 13 Oct 2014 18:40:12 +0000 (UTC) Received: by mail-yh0-f45.google.com with SMTP id b6so3886715yha.32 for ; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=GdkDgud9ZwTDGxKB4JcbBLYVup8Tc1bc/vmglFgvaPQ=; b=v5kmjvXon+D/5MwtcYJEB9SNU1KjAZzTQYAHkNmseDvyUpcNC4FrPpEd0d2QUHbgYi LhgTBQvf30lBJVM/Un1bFKUGcr4G5yspEYIZLRH4tjVGO2WsNCpgDqjpMIG5DY/NM4nO y/J5FFk4mK6Suel9TjXHelawJyEBgE7DOF3SxIpDjbP4vsqjkD343ZMvcnCc2nB8A+pu Y87QUFc45YrOZIdM0vMFcHAM0Bn3bwkA+3P5FjBgW7QpHN5MkuXKc4uDMjl2zxfLoArX JBFgJ6cfqa2SG5WxXauBzNOi0e2lk2LNjxqlDOtWWHq8hEE+lvwCVHj+J5uKiXGBiN1e GZTQ== MIME-Version: 1.0 X-Received: by 10.236.96.197 with SMTP id r45mr536807yhf.52.1413225611417; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 11:40:11 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 11:40:11 -0700 X-Google-Sender-Auth: Fs7HlWehuGXSlb7kaqKeWaNiw2E Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:40:12 -0000 >> Yer I would have got the zio details but typically its "optimised out" b= y >> the >> compiler, so will need some effort to track that down unfortunately :( > > > Well, let me know if you can. Re-creating a new 10.x VM is taking a while= as > it's taking me forever to checkout the sources. > > Things like that need to somehow continue to be accessible. So there is an outstanding child, but it isn't clear what state it's in because the child pointer in the zio_link isn't valid. It's as if the memory got re-used for something else. I'm dumping my findings below on the off-chance that it helps you push this along. (kgdb) thread 459 [Switching to thread 459 (Thread 101524)]#0 sched_switch (td=3D0xfffff80063111000, newtd=3D, flags=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/sched_ule.c:1945 1945 cpuid =3D PCPU_GET(cpuid); Current language: auto; currently minimal (kgdb) bt #0 sched_switch (td=3D0xfffff80063111000, newtd=3D, flags=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/sched_ule.c:1945 #1 0xffffffff807aa199 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_synch.c:494 #2 0xffffffff807e6e82 in sleepq_switch (wchan=3D, pri=3D) at /usr/home/kmacy/devel/svn/10/sys/kern/subr_sleepqueue.c:538 #3 0xffffffff807e6ce3 in sleepq_wait (wchan=3D0xfffff8004ddf4a50, pri=3D0) at /usr/home/kmacy/devel/svn/10/sys/kern/subr_sleepqueue.c:617 #4 0xffffffff80750d7a in _cv_wait (cvp=3D0xfffff8004ddf4a50, lock=3D0xfffff8004ddf4a30) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_condvar.c:139 #5 0xffffffff817d145b in zio_wait (zio=3D) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/zio.c:1442 #6 0xffffffff81779d3c in dsl_pool_sync (dp=3D0xfffff8004d364800, txg=3D117= 33518) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/dsl_pool.c:531 #7 0xffffffff8179d800 in spa_sync (spa=3D0xfffffe000372f000, txg=3D1173351= 8) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/spa.c:6604 #8 0xffffffff817a7e9d in txg_sync_thread (arg=3D0xfffff8004d364800) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/txg.c:518 #9 0xffffffff8076ed34 in fork_exit (callout=3D0xffffffff817a7c50 , arg=3D0xfffff8004d364800, frame=3D0xfffffe012043fac0) at /usr/home/kmacy/devel/svn/10/sys/kern/kern_fork.c:996 #10 0xffffffff80b96b3e in fork_trampoline () at /usr/home/kmacy/devel/svn/10/sys/amd64/amd64/exception.S:606 #11 0x0000000000000000 in ?? () (kgdb) f 5 #5 0xffffffff817d145b in zio_wait (zio=3D) at /usr/home/kmacy/devel/svn/10/sys/modules/zfs/../../cddl/contrib/open= solaris/uts/common/fs/zfs/zio.c:1442 1442 cv_wait(&zio->io_cv, &zio->io_lock); (kgdb) disassemble zio_wait Dump of assembler code for function zio_wait: 0xffffffff817d13c0 : push %rbp 0xffffffff817d13c1 : mov %rsp,%rbp 0xffffffff817d13c4 : push %r15 0xffffffff817d13c6 : push %r14 0xffffffff817d13c8 : push %r12 0xffffffff817d13ca : push %rbx 0xffffffff817d13cb : mov %rdi,%r14 0xffffffff817d13ce : cmpl $0x1,0x254(%r14) 0xffffffff817d13d6 : je 0xffffffff817d13f0 0xffffffff817d13d8 : mov $0xffffffff81883de9,%rdi 0xffffffff817d13df : mov $0xffffffff81883b20,%rsi 0xffffffff817d13e6 : mov $0x599,%edx 0xffffffff817d13eb : callq 0xffffffff81a19200 0xffffffff817d13f0 : cmpq $0x0,0x2f0(%r14) 0xffffffff817d13f8 : je 0xffffffff817d1412 0xffffffff817d13fa : mov $0xffffffff8188410c,%rdi 0xffffffff817d1401 : mov $0xffffffff81883b20,%rsi 0xffffffff817d1408 : mov $0x59a,%edx 0xffffffff817d140d : callq 0xffffffff81a19200 0xffffffff817d1412 : mov %gs:0x0,%rax 0xffffffff817d141b : mov %rax,0x2f8(%r14) 0xffffffff817d1422 : mov %r14,%rdi 0xffffffff817d1425 : callq 0xffffffff817d24c0 0xffffffff817d142a : lea 0x300(%r14),%r15 0xffffffff817d1431 : xor %esi,%esi 0xffffffff817d1433 : mov $0xffffffff81883b20,%rdx 0xffffffff817d143a : mov $0x5a0,%ecx 0xffffffff817d143f : mov %r15,%rdi 0xffffffff817d1442 : callq 0xffffffff807a8270 <_sx_xloc= k> 0xffffffff817d1447 : lea 0x320(%r14),%rbx 0xffffffff817d144e : jmp 0xffffffff817d145b 0xffffffff817d1450 : mov %rbx,%rdi 0xffffffff817d1453 : mov %r15,%rsi 0xffffffff817d1456 : callq 0xffffffff80750ba0 <_cv_wait= > 0xffffffff817d145b : cmpq $0x0,0x2f0(%r14) 0xffffffff817d1463 : jne 0xffffffff817d1450 0xffffffff817d1465 : mov $0xffffffff81883b20,%rsi 0xffffffff817d146c : mov $0x5a3,%edx 0xffffffff817d1471 : mov %r15,%rdi 0xffffffff817d1474 : callq 0xffffffff807a8630 <_sx_xunl= ock> 0xffffffff817d1479 : mov 0x268(%r14),%r12d 0xffffffff817d1480 : lea 0xf0(%r14),%rdi 0xffffffff817d1487 : callq 0xffffffff8172d480 0xffffffff817d148c : lea 0x110(%r14),%rdi 0xffffffff817d1493 : callq 0xffffffff8172d480 0xffffffff817d1498 : mov %r15,%rdi 0xffffffff817d149b : callq 0xffffffff807a8010 0xffffffff817d14a0 : mov %rbx,%rdi 0xffffffff817d14a3 : callq 0xffffffff80750b50 0xffffffff817d14a8 : mov 0xffffffff818af6e0,%rdi 0xffffffff817d14b0 : mov %r14,%rsi 0xffffffff817d14b3 : callq 0xffffffff81a19400 0xffffffff817d14b8 : mov %r12d,%eax 0xffffffff817d14bb : pop %rbx 0xffffffff817d14bc : pop %r12 0xffffffff817d14be : pop %r14 0xffffffff817d14c0 : pop %r15 0xffffffff817d14c2 : pop %rbp 0xffffffff817d14c3 : retq End of assembler dump. 0xffffffff817d1422 : mov %r14,%rdi 0xffffffff817d1425 : callq 0xffffffff817d24c0 0xffffffff817d142a : lea 0x300(%r14),%r15 0xffffffff817d1431 : xor %esi,%esi 0xffffffff817d1433 : mov $0xffffffff81883b20,%rdx 0xffffffff817d143a : mov $0x5a0,%ecx 0xffffffff817d143f : mov %r15,%rdi 0xffffffff817d1442 : callq 0xffffffff807a8270 <_sx_xloc= k> 0xffffffff817d1447 : lea 0x320(%r14),%rbx 0xffffffff817d144e : jmp 0xffffffff817d145b 0xffffffff817d1450 : mov %rbx,%rdi 0xffffffff817d1453 : mov %r15,%rsi 0xffffffff817d1456 : callq 0xffffffff80750ba0 <_cv_wait= > (kgdb) p *(zio_t *)$r14 $1 =3D { io_bookmark =3D { zb_objset =3D 0, zb_object =3D 0, zb_level =3D 0, zb_blkid =3D 0 }, io_prop =3D { zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_verify =3D 0, zp_nopwrite =3D 0 }, io_type =3D ZIO_TYPE_NULL, io_child_type =3D ZIO_CHILD_LOGICAL, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_NOW, io_reexecute =3D 2 '\002', io_state =3D "\001\001", io_txg =3D 0, io_spa =3D 0xfffffe000372f000, io_bp =3D 0x0, io_bp_override =3D 0x0, io_bp_copy =3D { blk_dva =3D {{ dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D {0, 0, 0, 0} } }, io_parent_list =3D { list_size =3D 48, list_offset =3D 16, list_head =3D { list_next =3D 0xfffff800b435c850, list_prev =3D 0xfffff800b435c850 } }, io_child_list =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff80003585770, list_prev =3D 0xfffff80003585770 } }, io_walk_link =3D 0x0, io_logical =3D 0x0, io_transform_stack =3D 0x0, io_ready =3D 0, io_physdone =3D 0, io_done =3D 0, io_private =3D 0x0, io_prev_space_delta =3D 0, io_bp_orig =3D { blk_dva =3D {{ dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }, { dva_word =3D {0, 0} }}, blk_prop =3D 0, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D {0, 0, 0, 0} } }, io_data =3D 0x0, io_orig_data =3D 0x0, io_size =3D 0, io_orig_size =3D 0, io_vd =3D 0x0, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_offset =3D 0, io_timestamp =3D 0, io_queue_node =3D { avl_child =3D {0x0, 0x0}, avl_pcb =3D 0 }, io_flags =3D 0, io_stage =3D ZIO_STAGE_DONE, io_pipeline =3D 2162688, io_orig_flags =3D 0, io_orig_stage =3D ZIO_STAGE_OPEN, io_orig_pipeline =3D 2162688, io_error =3D 0, io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0}, {0, 0}}, io_child_count =3D 1, io_phys_children =3D 0, io_parent_count =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff8006314a000, io_waiter =3D 0xfffff80063111000, io_lock =3D { lock_object =3D { lo_name =3D 0xffffffff8188655c "zio->io_lock", lo_flags =3D 41091072, lo_data =3D 0, lo_witness =3D 0xfffffe00006eca80 }, sx_lock =3D 1 }, io_cv =3D { cv_description =3D 0xffffffff8188656a "zio->io_cv", cv_waiters =3D 1 }, io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D { tqent_task =3D { ta_link =3D { stqe_next =3D 0x0 }, ta_pending =3D 0, ta_priority =3D 0, ta_func =3D 0, ta_context =3D 0x0 }, tqent_func =3D 0, tqent_arg =3D 0x0 }, io_trim_node =3D { avl_child =3D {0x0, 0x0}, avl_pcb =3D 0 }, io_trim_link =3D { list_next =3D 0x0, list_prev =3D 0x0 } } All of its children have called zio_notify_parent on this zio and are error free: io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0}, {0, 0}}, (kgdb) p ((zio_t *)$r14)->io_stall $5 =3D (uint64_t *) 0x0 One child has itself not reached the end of zio_done: io_child_count =3D 1, (kgdb) p ((zio_t *)$r14)->io_child_list $2 =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff80003585770, list_prev =3D 0xfffff80003585770 } (kgdb) p *(zio_link_t *)0xfffff80003585770 $15 =3D { zl_parent =3D 0xfffff8004ddf4850, zl_child =3D 0xfffff8004ddf4850, zl_parent_node =3D { list_next =3D 0xfffff8004dd62398, list_prev =3D 0xfffff8004d4d2730 }, zl_child_node =3D { list_next =3D 0x0, list_prev =3D 0x0 } } I don't understand why the parent and child are the same, of course that could be part of the problem. That address isn't a valid zio and there and the child node entry doesn't point to anything. It has a thread executing it (which itself is probably in cv_wait): io_executor =3D 0xfffff8006314a000, (kgdb) p *((struct thread *)0xfffff8006314a000) $11 =3D { td_lock =3D 0xffffffff81224b30, td_proc =3D 0xffffffff814438d8, td_plist =3D { tqe_next =3D 0xfffff8006312b920, tqe_prev =3D 0xfffff8006312a4a0 }, td_runq =3D { tqe_next =3D 0x0, tqe_prev =3D 0xffffffff811ef378 }, td_slpq =3D { tqe_next =3D 0x0, tqe_prev =3D 0xfffff80020e204c0 }, td_lockq =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 }, td_hash =3D { le_next =3D 0x0, le_prev =3D 0xfffffe0000879fc8 }, td_cpuset =3D 0xfffff800631d2318, td_sel =3D 0x0, td_sleepqueue =3D 0x0, td_turnstile =3D 0xfffff80063128480, td_rlqe =3D 0x0, td_umtxq =3D 0xfffff8004d7a6380, td_tid =3D 100345, td_sigqueue =3D { sq_signals =3D { __bits =3D {0, 0, 0, 0} }, sq_kill =3D { __bits =3D {0, 0, 0, 0} }, sq_list =3D { tqh_first =3D 0x0, tqh_last =3D 0xfffff8006314a0b8 }, sq_proc =3D 0xffffffff814438d8, sq_flags =3D 1 }, td_lend_user_pri =3D 255 '=C3=BF', td_flags =3D 4, td_inhibitors =3D 2, td_pflags =3D 2097152, td_dupfd =3D 0, td_sqqueue =3D 0, td_wchan =3D 0xfffff800b48a2900, td_wmesg =3D 0xffffffff80d64d7e "-", td_lastcpu =3D 0 '\0', td_oncpu =3D 255 '=C3=BF', td_owepreempt =3D 0 '\0', td_tsqueue =3D 0 '\0', td_locks =3D 0, td_rw_rlocks =3D 0, td_lk_slocks =3D 0, td_stopsched =3D 0, td_blocked =3D 0x0, td_lockname =3D 0x0, td_contested =3D { lh_first =3D 0x0 }, td_sleeplocks =3D 0xffffffff81369470, ---Type to continue, or q to quit--- td_intr_nesting_level =3D 0, td_pinned =3D 0, td_ucred =3D 0xfffff800027f1e00, td_estcpu =3D 0, td_slptick =3D 226829, td_blktick =3D 0, td_swvoltick =3D 226829, td_cow =3D 0, td_ru =3D { ru_utime =3D { tv_sec =3D 0, tv_usec =3D 0 }, ru_stime =3D { tv_sec =3D 0, tv_usec =3D 0 }, ru_maxrss =3D 0, ru_ixrss =3D 0, ru_idrss =3D 0, ru_isrss =3D 0, ru_minflt =3D 0, ru_majflt =3D 0, ru_nswap =3D 0, ru_inblock =3D 0, ru_oublock =3D 0, ru_msgsnd =3D 0, ru_msgrcv =3D 0, ru_nsignals =3D 0, ru_nvcsw =3D 6, ru_nivcsw =3D 0 }, td_rux =3D { rux_runtime =3D 0, rux_uticks =3D 0, rux_sticks =3D 0, rux_iticks =3D 0, rux_uu =3D 0, rux_su =3D 0, rux_tu =3D 0 }, td_incruntime =3D 410296, td_runtime =3D 410296, td_pticks =3D 0, td_sticks =3D 0, td_iticks =3D 0, td_uticks =3D 0, td_intrval =3D 0, td_oldsigmask =3D { __bits =3D {0, 0, 0, 0} }, td_generation =3D 6, td_sigstk =3D { ss_sp =3D 0x0, ss_size =3D 0, ss_flags =3D 0 }, td_xsig =3D 0, td_profil_addr =3D 0, td_profil_ticks =3D 0, td_name =3D "zio_write_intr_7\000\000\000", td_fpop =3D 0x0, td_dbgflags =3D 0, td_dbgksi =3D { ksi_link =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 }, ksi_info =3D { si_signo =3D 0, si_errno =3D 0, si_code =3D 0, si_pid =3D 0, si_uid =3D 0, si_status =3D 0, si_addr =3D 0x0, si_value =3D { sival_int =3D 0, sival_ptr =3D 0x0, sigval_int =3D 0, sigval_ptr =3D 0x0 }, _reason =3D { _fault =3D { _trapno =3D 0 }, _timer =3D { _timerid =3D 0, _overrun =3D 0 }, _mesgq =3D { _mqd =3D 0 }, _poll =3D { _band =3D 0 }, __spare__ =3D { __spare1__ =3D 0, __spare2__ =3D {0, 0, 0, 0, 0, 0, 0} } } }, ksi_flags =3D 0, ksi_sigq =3D 0x0 }, td_ng_outbound =3D 0, td_osd =3D { osd_nslots =3D 0, osd_slots =3D 0x0, osd_next =3D { le_next =3D 0x0, le_prev =3D 0x0 } }, td_map_def_user =3D 0x0, td_dbg_forked =3D 0, td_vp_reserv =3D 0, td_no_sleeping =3D 0, td_dom_rr_idx =3D 0, td_sigmask =3D { __bits =3D {0, 0, 0, 0} }, td_rqindex =3D 21 '\025', td_base_pri =3D 84 'T', td_priority =3D 84 'T', td_pri_class =3D 3 '\003', td_user_pri =3D 172 '=C2=AC', td_base_user_pri =3D 172 '=C2=AC', td_pcb =3D 0xfffffe012060eb80, td_state =3D TDS_INHIBITED, td_retval =3D {0, 0}, td_slpcallout =3D { ---Type to continue, or q to quit--- c_links =3D { le =3D { le_next =3D 0x0, le_prev =3D 0x0 }, sle =3D { sle_next =3D 0x0 }, tqe =3D { tqe_next =3D 0x0, tqe_prev =3D 0x0 } }, c_time =3D 0, c_precision =3D 0, c_arg =3D 0x0, c_func =3D 0, c_lock =3D 0x0, c_flags =3D 16, c_cpu =3D 0 }, td_frame =3D 0xfffffe012060eac0, td_kstack_obj =3D 0xfffff8006314cd00, td_kstack =3D 18446741879524470784, td_kstack_pages =3D 4, td_critnest =3D 1, td_md =3D { md_spinlock_count =3D 1, md_saved_flags =3D 582, md_spurflt_addr =3D 0 }, td_sched =3D 0xfffff8006314a468, td_ar =3D 0x0, td_lprof =3D {{ lh_first =3D 0x0 }, { lh_first =3D 0x0 }}, td_dtrace =3D 0xfffff800b437ce00, td_errno =3D 0, td_vnet =3D 0x0, td_vnet_lpush =3D 0x0, td_intr_frame =3D 0x0, td_rfppwait_p =3D 0x0, td_ma =3D 0x0, td_ma_cnt =3D 0 } Interesting bits about the executing thread: td_name =3D "zio_write_intr_7\000\000\000", td_critnest =3D 1, td_md =3D { md_spinlock_count =3D 1, md_saved_flags =3D 582, md_spurflt_addr =3D 0 }, Unfortunately executor is never cleared, so this is just the last thread that happened to execute this i/o. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 18:43:47 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 480C4987; Mon, 13 Oct 2014 18:43:47 +0000 (UTC) Received: from mail-yh0-x232.google.com (mail-yh0-x232.google.com [IPv6:2607:f8b0:4002:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE488E2F; Mon, 13 Oct 2014 18:43:46 +0000 (UTC) Received: by mail-yh0-f50.google.com with SMTP id a41so3905591yho.9 for ; Mon, 13 Oct 2014 11:43:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=3MfuhrtClGMRcJI4ReBhfBwre3FoIcn3vzxn2vjuBIg=; b=OpNi31hwVi3Pyb7OHyc+289I2vPOXI6MGIs9HkT5f8PgAml8lTLDtW17hazfa1svOC +nzhvCBdMeq+PO5QVIcuUat0UphiwS9WVOVuzcRG36zf+/D9LFmoDlIYUcfWjurUwnAL C5e+arEmCjfHWZYc99AOFu34QiZNcjiYby+ilU7nHxWy68D5kKXF9J7F6KP4PXaroAJl VvzAp2P2hrUxBEfJAKXLJ6IlBl4eTrXvQzV26dAMxMMqw5ppgLBgK8EFARg9Z2Q+OyVc 6XiF0+LzTeELdYXKwmnYe518Gspl/uOkYyJTH5iaJDFQ/FE1k25gRuulCSwJOHW6UsJx ACZQ== MIME-Version: 1.0 X-Received: by 10.236.103.170 with SMTP id f30mr659352yhg.4.1413225825965; Mon, 13 Oct 2014 11:43:45 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 11:43:45 -0700 (PDT) In-Reply-To: <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 11:43:45 -0700 X-Google-Sender-Auth: Sv6PFUGBZE66F3NhqfSBLjHi2ys Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:43:47 -0000 >> A recent quick read of the code would lead me to believe that > > Yer I would have got the zio details but typically its "optimised out" by > the > compiler, so will need some effort to track that down unfortunately :( FYI it isn't actually optimized out. The debug info accounting just isn't very good about tracking the values in registers. You'll see in my last mail that just looking at the assembler makes it pretty obvious that zio was in %r14. HTH. Cheers. -K From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 18:48:40 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D46FFC8B for ; Mon, 13 Oct 2014 18:48:40 +0000 (UTC) Received: from mario.brtsvcs.net (mario.brtsvcs.net [IPv6:2607:fc50:0:a400::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA8B2E70 for ; Mon, 13 Oct 2014 18:48:40 +0000 (UTC) Received: from chombo.houseloki.net (unknown [IPv6:2601:7:400:640:21c:c0ff:fe7f:96ee]) by mario.brtsvcs.net (Postfix) with ESMTPSA id 6B3212C1622; Mon, 13 Oct 2014 11:48:33 -0700 (PDT) Received: from [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29] (unknown [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29]) by chombo.houseloki.net (Postfix) with ESMTPSA id 0715AEF5; Mon, 13 Oct 2014 11:48:31 -0700 (PDT) Message-ID: <543C1E7D.1000301@bluerosetech.com> Date: Mon, 13 Oct 2014 11:48:29 -0700 From: Darren Pilgrim Reply-To: freebsd-stable@freebsd.org User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: karl@denninger.net Subject: Re: getting to 4K disk blocks in ZFS References: <540FF3C4.6010305@ish.com.au> <54100258.2000505@freebsd.org> <5410F0B4.9040808@ish.com.au> <54114029.3060507@FreeBSD.org> <20140911072351.GA50786@anubis.morrow.me.uk> In-Reply-To: <20140911072351.GA50786@anubis.morrow.me.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:48:40 -0000 On 9/11/2014 12:23 AM, Ben Morrow wrote: > Is there any way (short of building a new pool) to get a reasonable idea > of how much extra space a given pool would use if converted? Take the average file size modulo 4096, round it up to a multiple of 512, subtract that from 4096, then multiply by the number of files in the data set. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 18:48:40 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2D36C8A for ; Mon, 13 Oct 2014 18:48:40 +0000 (UTC) Received: from mario.brtsvcs.net (mario.brtsvcs.net [IPv6:2607:fc50:0:a400::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 78EEAE6F for ; Mon, 13 Oct 2014 18:48:40 +0000 (UTC) Received: from chombo.houseloki.net (unknown [IPv6:2601:7:400:640:21c:c0ff:fe7f:96ee]) by mario.brtsvcs.net (Postfix) with ESMTPSA id 937942C1621; Mon, 13 Oct 2014 11:48:32 -0700 (PDT) Received: from [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29] (unknown [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29]) by chombo.houseloki.net (Postfix) with ESMTPSA id 9143AEF4; Mon, 13 Oct 2014 11:48:30 -0700 (PDT) Message-ID: <543C1E7B.4090204@bluerosetech.com> Date: Mon, 13 Oct 2014 11:48:27 -0700 From: Darren Pilgrim Reply-To: freebsd-stable User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Steven Hartland Subject: Re: getting to 4K disk blocks in ZFS References: <540FF3C4.6010305@ish.com.au> <54100258.2000505@freebsd.org> <5410F0B4.9040808@ish.com.au> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 18:48:40 -0000 On 9/10/2014 6:22 PM, Steven Hartland wrote: > From: "Aristedes Maniatis" >> Should the FreeBSD project change this minimum in the next release? >> There seems to be no downside and a huge amount of pain for people >> who stumble along with the defaults not knowing what a mess they are >> creating to solve later. > > The downside is wasted space which can be significant and hence when > I last suggested just this it was unfortunately rejected. I always found that justification rather weak. If the default is 512b and you use 4k disks, you kill performance, create a massive headache of a correction task, and prevent ZFS from doing its data integrity assurance. If the default is 4k and (for the limited time they're still common) you use true 512b disks, you can waste space. Sure, but how much space? I nerd-sniped myself, and calculated it against my own systems: On my own Maildir (literally everything except spam since 1999): Average size of files modulo 4096: 3137 Total files: 2651888 ashift=12 would add 512 B slack per file--about 2.5% of the set's 54 GB on-disk size. Maildirs are historically one of the worst cases for slack. The worst case of an extra 3584 B slack per file would be 17.6%. A 25% slack allowance is typically what I use for planning IMAP storage requirements, so we're not even out of bounds. On the Samba server at work, which holds roughly two decades of office documents and other "human" files: Average size of files modulo 4096: 1260 Total files: 351892 Here, ashift=12 would add 2.5 kB slack per file--about 0.26% of the set's 350 GB on-disk size. The worst case would be 0.36% slack. In other words, it likely doesn't matter except in specific cases where you're already worrying about effective disk usage. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 19:02:21 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 522B5405; Mon, 13 Oct 2014 19:02:21 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B198EB6; Mon, 13 Oct 2014 19:02:20 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 3D0C020E708EB; Mon, 13 Oct 2014 19:02:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 157FE20E708E8; Mon, 13 Oct 2014 19:02:15 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 20:02:12 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:02:21 -0000 ----- Original Message ----- From: "K. Macy" > On Mon, Oct 13, 2014 at 1:06 AM, Steven Hartland > wrote: > >> ----- Original Message ----- From: "K. Macy" >> >> A recent quick read of the code would lead me to believe that zio_wait not >>> returning there means that the zio never reached the zio_done stage. >>> Parent >>> zios seem to yield in a couple of stages in the pipeline if they have >>> incomplete children. They determine this by calling zio_wait_for_children >>> with zio child types and their corresponding wait type. In so doing they >>> set the io_stall to the count of the number of waiters of the first >>> non-zero check. This parent I/O will be resumed by the last child zio of >>> that type and wait state in zio_notify_parent. I'm sure you know all this >>> - >>> but I wrote it to preface asking for the following fields of the zio being >>> waited on in dsl_pool_sync_mos: io_stall (i.e, which field in io_children >>> is pointed to) *io_stall, io_children[*][*], io_child_list (at a first >>> glance just the addresses). The other alternative is that it reexecuting >>> has gotten in to a bad place in the state machine so io_reexecute. >>> >> >> Yer I would have got the zio details but typically its "optimised out" by >> the >> compiler, so will need some effort to track that down unfortunately :( >> > > Well, let me know if you can. Re-creating a new 10.x VM is taking a while > as it's taking me forever to checkout the sources. > > Things like that need to somehow continue to be accessible. I believe there's some pool corruption here somewhere as every once in a while I trip and ASSERT panic: panic: solaris assert: size >= SPA_MINBLOCKSIZE || range_tree_space(msp->ms_tree) == 0, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, line: 1636 #3 0xffffffff80607ed3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff8179321d in assfail (a=, f=, l=) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 #5 0xffffffff81501ecf in metaslab_passivate (msp=0xfffff800091e5800, size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1636 #6 0xffffffff81501ca7 in metaslab_group_alloc (mg=0xfffff80044ef7400, psize=512, asize=512, txg=11733518, min_distance=536281088, dva=, d=-1) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2206 #7 0xffffffff81500c9d in metaslab_alloc_dva (spa=0xfffffe00022bb000, mc=0xfffff800045d0c00, psize=512, dva=0xfffffe000e409640, d=0, hintdva=, flags=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2381 #8 0xffffffff81500709 in metaslab_alloc (spa=0xfffffe00022bb000, mc=0xfffff800045d0c00, psize=512, bp=0xfffffe000e409640, ndvas=3, txg=11733518, hintbp=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:2586 #9 0xffffffff8154ca8a in zio_dva_allocate (zio=0xfffff80044a57398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2444 #10 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a57398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #11 0xffffffff8154d25f in zio_ready (zio=0xfffff800487d2730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3031 #12 0xffffffff81548d54 in zio_execute (zio=0xfffff800487d2730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #13 0xffffffff80651410 in taskqueue_run_locked (queue=0xfffff80054291000) at /usr/src/sys/kern/subr_taskqueue.c:342 #14 0xffffffff80651dcb in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:563 (kgdb) frame 5 #5 0xffffffff81501ecf in metaslab_passivate (msp=0xfffff800091e5800, size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1636 1636 ASSERT(size >= SPA_MINBLOCKSIZE || range_tree_space(msp->ms_tree) == 0); (kgdb) print size $5 = 0 (kgdb) print msp->ms_tree $6 = (range_tree_t *) 0xfffff800091e5400 (kgdb) print *msp->ms_tree $7 = { rt_root = { avl_root = 0xfffff80048fab840, avl_compar = 0xffffffff81502850 , avl_offset = 0, avl_numnodes = 1, avl_size = 64 }, rt_space = 5632, rt_ops = 0xffffffff8160b110, rt_arg = 0xfffff800091e5800, rt_histogram = {0 , 1, 0 }, rt_lock = 0xfffff800091e5800 } (kgdb) print *msp $8 = { ms_lock = { lock_object = { lo_name = 0xffffffff815eb310 "msp->ms_lock", lo_flags = 40960000, lo_data = 0, lo_witness = 0x0 }, sx_lock = 18446735279027798016 }, ms_load_cv = { cv_description = 0xffffffff815eb31e "msp->ms_load_cv", cv_waiters = 0 }, ms_sm = 0xfffff8000924bb80, ms_ops = 0xffffffff8160b100, ms_id = 119, ms_start = 3992977408, ms_size = 33554432, ms_fragmentation = 90, ms_alloctree = {0xfffff800091e5000, 0xfffff800091ec000, 0xfffff800091e3c00, 0xfffff800091e3400}, ms_freetree = {0xfffff800091e4c00, 0xfffff800091e4000, 0xfffff800091e4400, 0xfffff800091e3000}, ms_defertree = {0xfffff800091e3800, 0xfffff800091df400}, ms_tree = 0xfffff800091e5400, ms_condensing = 0, ms_condense_wanted = 0, ms_loaded = 1, ms_loading = 0, ms_deferspace = 0, ms_weight = 13835058055282163712, ms_access_txg = 11733526, ms_size_tree = { avl_root = 0xfffff80048fab858, avl_compar = 0xffffffff81502360 , avl_offset = 24, avl_numnodes = 1, avl_size = 64 }, ms_lbas = {0, 0, 0, 0, 0, 0, 0, 0, 0, 4026524672, 4026530816, 0 }, ms_group = 0xfffff80044ef7400, ms_group_node = { avl_child = {0x0, 0x0}, avl_pcb = 18446735278773074705 }, ms_txg_node = { tn_next = {0x0, 0xfffff800091d7b28, 0xfffff800091e2328, 0x0}, tn_member = "\000\001\001" } } Also when it stalls I've seen this in the zfs debug:- 13 39363 :zfs-dprintf dnode.c - dnode_free_range:1655: ds=mos obj=31 blkid=0 nblks=1125899906842624 txg=3078494 13 39363 :zfs-dprintf dbuf.c - dbuf_free_range:816: ds=mos obj=31 start=0 end=0 13 39363 :zfs-dprintf dbuf.c - dbuf_dirty:1133: ds=mos obj=40 lvl=0 blkid=-1 size=140 13 39363 :zfs-dprintf dnode.c - dnode_setdirty:1286: ds=mos obj=40 txg=3078494 nblks looks suspisiously large and I'm not sure blkid should be -1? For reference to see this I'm using: sysctl debug.zfs_flags=64 dtrace -n 'zfs-dprintf {printf("%s - %s:%d: %s", stringof(arg0), stringof(arg1), arg2, stringof(arg3))}' With all that said the following looks like it might indicate the issue: zdb -e -m sys1boot Metaslabs: vdev 0 metaslabs 127 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 417 free 0 metaslab 1 offset 2000000 spacemap 24 free 0 metaslab 2 offset 4000000 spacemap 64 free 0 metaslab 3 offset 6000000 spacemap 1094 free 0 metaslab 4 offset 8000000 spacemap 1091 free 0 metaslab 5 offset a000000 spacemap 1093 free 0 metaslab 6 offset c000000 spacemap 1095 free 0 metaslab 7 offset e000000 spacemap 1096 free 0 metaslab 8 offset 10000000 spacemap 1098 free 0 metaslab 9 offset 12000000 spacemap 1097 free 0 metaslab 10 offset 14000000 spacemap 1099 free 0 metaslab 11 offset 16000000 spacemap 1102 free 0 metaslab 12 offset 18000000 spacemap 113 free 0 metaslab 13 offset 1a000000 spacemap 110 free 0 metaslab 14 offset 1c000000 spacemap 115 free 0 metaslab 15 offset 1e000000 spacemap 1103 free 0 metaslab 16 offset 20000000 spacemap 416 free 0 metaslab 17 offset 22000000 spacemap 81 free 0 metaslab 18 offset 24000000 spacemap 57 free 0 metaslab 19 offset 26000000 spacemap 102 free 0 metaslab 20 offset 28000000 spacemap 100 free 0 metaslab 21 offset 2a000000 spacemap 104 free 0 metaslab 22 offset 2c000000 spacemap 125 free 0 metaslab 23 offset 2e000000 spacemap 105 free 0 metaslab 24 offset 30000000 spacemap 108 free 0 metaslab 25 offset 32000000 spacemap 126 free 0 metaslab 26 offset 34000000 spacemap 248 free 0 metaslab 27 offset 36000000 spacemap 55 free 0 metaslab 28 offset 38000000 spacemap 82 free 0 metaslab 29 offset 3a000000 spacemap 149 free 0 metaslab 30 offset 3c000000 spacemap 152 free 0 metaslab 31 offset 3e000000 spacemap 155 free 0 metaslab 32 offset 40000000 spacemap 156 free 0 metaslab 33 offset 42000000 spacemap 72 free 0 metaslab 34 offset 44000000 spacemap 96 free 0 metaslab 35 offset 46000000 spacemap 159 free 0 metaslab 36 offset 48000000 spacemap 158 free 0 metaslab 37 offset 4a000000 spacemap 160 free 0 metaslab 38 offset 4c000000 spacemap 1271 free 0 metaslab 39 offset 4e000000 spacemap 161 free 0 metaslab 40 offset 50000000 spacemap 1273 free 0 metaslab 41 offset 52000000 spacemap 1277 free 0 metaslab 42 offset 54000000 spacemap 35 free 0 metaslab 43 offset 56000000 spacemap 25 free 0 metaslab 44 offset 58000000 spacemap 32 free 0 metaslab 45 offset 5a000000 spacemap 150 free 0 metaslab 46 offset 5c000000 spacemap 151 free 0 metaslab 47 offset 5e000000 spacemap 163 free 0 metaslab 48 offset 60000000 spacemap 415 free 0 metaslab 49 offset 62000000 spacemap 98 free 0 metaslab 50 offset 64000000 spacemap 109 free 0 metaslab 51 offset 66000000 spacemap 171 free 0 metaslab 52 offset 68000000 spacemap 80 free 0 metaslab 53 offset 6a000000 spacemap 134 free 0 metaslab 54 offset 6c000000 spacemap 135 free 0 metaslab 55 offset 6e000000 spacemap 154 free 0 metaslab 56 offset 70000000 spacemap 140 free 0 metaslab 57 offset 72000000 spacemap 141 free 0 metaslab 58 offset 74000000 spacemap 1272 free 0 metaslab 59 offset 76000000 spacemap 138 free 0 metaslab 60 offset 78000000 spacemap 139 free 0 metaslab 61 offset 7a000000 spacemap 18 free 0 metaslab 62 offset 7c000000 spacemap 148 free 0 metaslab 63 offset 7e000000 spacemap 1270 free 0 metaslab 64 offset 80000000 spacemap 114 free 0 metaslab 65 offset 82000000 spacemap 112 free 0 metaslab 66 offset 84000000 spacemap 116 free 0 metaslab 67 offset 86000000 spacemap 164 free 0 metaslab 68 offset 88000000 spacemap 243 free 0 metaslab 69 offset 8a000000 spacemap 128 free 0 metaslab 70 offset 8c000000 spacemap 1101 free 0 metaslab 71 offset 8e000000 spacemap 153 free 0 metaslab 72 offset 90000000 spacemap 120 free 0 metaslab 73 offset 92000000 spacemap 62 free 0 metaslab 74 offset 94000000 spacemap 122 free 0 metaslab 75 offset 96000000 spacemap 131 free 0 metaslab 76 offset 98000000 spacemap 129 free 0 metaslab 77 offset 9a000000 spacemap 157 free 0 metaslab 78 offset 9c000000 spacemap 133 free 0 metaslab 79 offset 9e000000 spacemap 137 free 0 metaslab 80 offset a0000000 spacemap 165 free 0 metaslab 81 offset a2000000 spacemap 168 free 0 metaslab 82 offset a4000000 spacemap 170 free 0 metaslab 83 offset a6000000 spacemap 127 free 0 metaslab 84 offset a8000000 spacemap 180 free 0 metaslab 85 offset aa000000 spacemap 162 free 0 metaslab 86 offset ac000000 spacemap 1100 free 0 metaslab 87 offset ae000000 spacemap 107 free 0 metaslab 88 offset b0000000 spacemap 119 free 0 metaslab 89 offset b2000000 spacemap 61 free 0 metaslab 90 offset b4000000 spacemap 60 free 0 metaslab 91 offset b6000000 spacemap 123 free 0 metaslab 92 offset b8000000 spacemap 130 free 0 metaslab 93 offset ba000000 spacemap 59 free 0 metaslab 94 offset bc000000 spacemap 167 free 0 metaslab 95 offset be000000 spacemap 136 free 0 metaslab 96 offset c0000000 spacemap 144 free 0 metaslab 97 offset c2000000 spacemap 166 free 0 metaslab 98 offset c4000000 spacemap 169 free 0 metaslab 99 offset c6000000 spacemap 58 free 0 metaslab 100 offset c8000000 spacemap 56 free 0 metaslab 101 offset ca000000 spacemap 54 free 0 metaslab 102 offset cc000000 spacemap 53 free 0 metaslab 103 offset ce000000 spacemap 52 free 0 metaslab 104 offset d0000000 spacemap 106 free 0 metaslab 105 offset d2000000 spacemap 51 free 0 metaslab 106 offset d4000000 spacemap 50 free 0 metaslab 107 offset d6000000 spacemap 121 free 0 metaslab 108 offset d8000000 spacemap 124 free 0 metaslab 109 offset da000000 spacemap 49 free 0 metaslab 110 offset dc000000 spacemap 142 free 0 metaslab 111 offset de000000 spacemap 132 free 0 metaslab 112 offset e0000000 spacemap 48 free 0 metaslab 113 offset e2000000 spacemap 47 free 0 metaslab 114 offset e4000000 spacemap 46 free 0 metaslab 115 offset e6000000 spacemap 45 free 0 metaslab 116 offset e8000000 spacemap 143 free 0 metaslab 117 offset ea000000 spacemap 44 free 0 metaslab 118 offset ec000000 spacemap 43 free 177K metaslab 119 offset ee000000 spacemap 42 free 13.0K metaslab 120 offset f0000000 spacemap 41 free 0 metaslab 121 offset f2000000 spacemap 179 free 0 metaslab 122 offset f4000000 spacemap 40 free 0 metaslab 123 offset f6000000 spacemap 39 free 0 metaslab 124 offset f8000000 spacemap 38 free 0 metaslab 125 offset fa000000 spacemap 37 free 0 metaslab 126 offset fc000000 spacemap 36 free 0 Only two metaslabs with any free? zfs list -r sys1boot NAME USED AVAIL REFER MOUNTPOINT sys1boot 1.76G 2.08G 11K /sys1boot sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - So basically the zpool says its out of space even though zfs says its under half full? Given this I'm guessing that on import when it goes to write an update its hitting a brick wall and can't, with the zio being flagged as can't fail it seems to get stuck. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 19:14:00 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55534B95; Mon, 13 Oct 2014 19:14:00 +0000 (UTC) Received: from mail-yh0-x22f.google.com (mail-yh0-x22f.google.com [IPv6:2607:f8b0:4002:c01::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 067D11F1; Mon, 13 Oct 2014 19:13:59 +0000 (UTC) Received: by mail-yh0-f47.google.com with SMTP id c41so3897462yho.34 for ; Mon, 13 Oct 2014 12:13:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Ea7Yw4RhAHySmYJCtQSRSqE3Bd1Y69auwHcMR4b5ME8=; b=O9l5alfrZBEuTq/aT0d3XKiYz7mpM00ajBByGZyluc7Lo+W0z3IjnxVaGozJmNl2hO WEjLYGNByL9CNzaTMVlFBfeF8VTRC+rfhBhw9VtMKuTbzTiJ2ue9aBu8enBheWN15fVY G4Y6+8/LQjSh/y8fJjiecAbGM58H81g0IEOWxkLxdHvAOhMHE1tOB47k6Fe6uONqC75G 2CZpFkykHIGGO+31/4uWZzMGCvY4ikCiau7GP/wvCdmXmLyJzw1RgqQa42K+7ADRJmEi Kk3h+wVdZ7wA95kqppWLEJMhWAIYT9XzzNg65Oej5FTLRIxjZyvWKOdcKzHTawlmBmu9 wIGQ== MIME-Version: 1.0 X-Received: by 10.236.25.1 with SMTP id y1mr814329yhy.62.1413227639056; Mon, 13 Oct 2014 12:13:59 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 12:13:58 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Date: Mon, 13 Oct 2014 12:13:58 -0700 X-Google-Sender-Auth: F6WId1xapj9yD1oj_DAfO2MWkvQ Message-ID: Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:14:00 -0000 >>> Yer I would have got the zio details but typically its "optimised out" by >>> the >>> compiler, so will need some effort to track that down unfortunately :( >>> >> >> Well, let me know if you can. Re-creating a new 10.x VM is taking a while >> as it's taking me forever to checkout the sources. >> >> Things like that need to somehow continue to be accessible. > > > I believe there's some pool corruption here somewhere as every once in a > while > I trip and ASSERT panic: > panic: solaris assert: size >= SPA_MINBLOCKSIZE || > range_tree_space(msp->ms_tree) == 0, file: > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c, > line: 1636 > <... snip> You are correct. (kgdb) p ((zio_t *)$r14)->io_reexecute $32 = 2 '\002' (kgdb) p ((zio_t *)$r14)->io_flags $33 = 0 (kgdb) p ((zio_t *)$r14)->io_spa->spa_suspended $34 = 1 '\001' This means zio_suspend has been called from zio_done: else if (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND) { /* * We'd fail again if we reexecuted now, so suspend * until conditions improve (e.g. device comes online). */ zio_suspend(spa, zio); } If failure mode were panic we would have panicked when attempting the import: void zio_suspend(spa_t *spa, zio_t *zio) { if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_PANIC) fm_panic("Pool '%s' has encountered an uncorrectable I/O " "failure and the failure mode property for this pool " "is set to panic.", spa_name(spa)); From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 19:14:36 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4E09CF2; Mon, 13 Oct 2014 19:14:36 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 92DBD208; Mon, 13 Oct 2014 19:14:36 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id CD8AB20E708EE; Mon, 13 Oct 2014 19:14:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 264B920E708EB; Mon, 13 Oct 2014 19:14:34 +0000 (UTC) Message-ID: <15EB5C3A83504A949CBDA96420361ADB@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 20:14:30 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:14:36 -0000 ----- Original Message ----- From: "K. Macy" To: "Steven Hartland" Cc: "Mark Martinec" ; "freebsd-fs@FreeBSD.org" ; "FreeBSD Stable" Sent: Monday, October 13, 2014 7:43 PM Subject: Re: zfs pool import hangs on [tx->tx_sync_done_cv] >>> A recent quick read of the code would lead me to believe that >> >> Yer I would have got the zio details but typically its "optimised out" by >> the >> compiler, so will need some effort to track that down unfortunately :( > > FYI it isn't actually optimized out. The debug info accounting just > isn't very good about tracking the values in registers. You'll see in > my last mail that just looking at the assembler makes it pretty > obvious that zio was in %r14. HTH. > > Cheers. > > > -K > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 19:19:11 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 92137D7 for ; Mon, 13 Oct 2014 19:19:11 +0000 (UTC) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6554426C for ; Mon, 13 Oct 2014 19:19:11 +0000 (UTC) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id D27FD37B587; Mon, 13 Oct 2014 14:19:03 -0500 (CDT) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3jGqSR2D4Mz1Q2; Mon, 13 Oct 2014 14:19:03 -0500 (CDT) Date: Mon, 13 Oct 2014 14:19:03 -0500 From: "Matthew D. Fuller" To: freebsd-stable Subject: Re: getting to 4K disk blocks in ZFS Message-ID: <20141013191903.GR2161@over-yonder.net> References: <540FF3C4.6010305@ish.com.au> <54100258.2000505@freebsd.org> <5410F0B4.9040808@ish.com.au> <543C1E7B.4090204@bluerosetech.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <543C1E7B.4090204@bluerosetech.com> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.4 at thyme.infocus-llc.com X-Virus-Status: Clean Cc: Steven Hartland X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:19:11 -0000 On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of Darren Pilgrim, and lo! it spake thus: > > If the default is 4k and (for the limited time they're still common) > you use true 512b disks, you can waste space. Sure, but how much > space? The median file in /usr/ports is 408 bytes. Over 90% of the files are under 2k, which means the wastage for them is over 100% (before counting what gain compression might get). A little offhand mathery says it's about 78% extra overhead on the whole. And that includes the almost hundred megs (over 22% of the total size of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files (another 20%). If you pull those out, the overhead is 130%. (To be sure, relatively few people have ports trees eating most of their space, but still; it's pretty pathological. I for one did decide some years back to always force 4k on any new FSen to make future life simpler, accepting the bloat, but it's there.) -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 19:22:37 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 69AEF284 for ; Mon, 13 Oct 2014 19:22:37 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 2CEEE365 for ; Mon, 13 Oct 2014 19:22:36 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 6CA9520E708EE; Mon, 13 Oct 2014 19:22:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 7DF5B20E708EC; Mon, 13 Oct 2014 19:22:34 +0000 (UTC) Message-ID: <43D94A22FBD2477FBBDEFF16C0088DDA@multiplay.co.uk> From: "Steven Hartland" To: "Matthew D. Fuller" , "freebsd-stable" References: <540FF3C4.6010305@ish.com.au> <54100258.2000505@freebsd.org> <5410F0B4.9040808@ish.com.au> <543C1E7B.4090204@bluerosetech.com> <20141013191903.GR2161@over-yonder.net> Subject: Re: getting to 4K disk blocks in ZFS Date: Mon, 13 Oct 2014 20:22:31 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 19:22:37 -0000 ----- Original Message ----- From: "Matthew D. Fuller" > On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of > Darren Pilgrim, and lo! it spake thus: >> >> If the default is 4k and (for the limited time they're still common) >> you use true 512b disks, you can waste space. Sure, but how much >> space? > > The median file in /usr/ports is 408 bytes. Over 90% of the files are > under 2k, which means the wastage for them is over 100% (before > counting what gain compression might get). A little offhand mathery > says it's about 78% extra overhead on the whole. > > And that includes the almost hundred megs (over 22% of the total size > of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files > (another 20%). If you pull those out, the overhead is 130%. > > > (To be sure, relatively few people have ports trees eating most of > their space, but still; it's pretty pathological. I for one did > decide some years back to always force 4k on any new FSen to make > future life simpler, accepting the bloat, but it's there.) And thats before you add the overhead if your running RAIDZ... A good read on this is http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 20:10:19 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70875F3B; Mon, 13 Oct 2014 20:10:19 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id D984598C; Mon, 13 Oct 2014 20:10:18 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 95A7720E708F1; Mon, 13 Oct 2014 20:10:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: *** X-Spam-Status: No, score=3.1 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE, STOX_REPLY_TYPE_WITHOUT_QUOTES autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 0E33D20E708EF; Mon, 13 Oct 2014 20:10:14 +0000 (UTC) Message-ID: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> Subject: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 21:10:11 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Mark Martinec , "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 20:10:19 -0000 ----- Original Message ----- From: "K. Macy" > You are correct. > > (kgdb) p ((zio_t *)$r14)->io_reexecute > $32 = 2 '\002' > (kgdb) p ((zio_t *)$r14)->io_flags > $33 = 0 > (kgdb) p ((zio_t *)$r14)->io_spa->spa_suspended > $34 = 1 '\001' > > This means zio_suspend has been called from zio_done: > else if (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND) { > /* > * We'd fail again if we reexecuted now, so suspend > * until conditions improve (e.g. device comes online). > */ > zio_suspend(spa, zio); > } > > If failure mode were panic we would have panicked when attempting the import: > void > zio_suspend(spa_t *spa, zio_t *zio) > { > if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_PANIC) > fm_panic("Pool '%s' has encountered an uncorrectable I/O " > "failure and the failure mode property for this pool " > "is set to panic.", spa_name(spa)); Yep and forcing that panic I got the following stack: #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff80607977 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff80607e85 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80607ed3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff81548dfa in zio_suspend (spa=, zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1527 #5 0xffffffff8154ec66 in zio_done (zio=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3264 #6 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a0dac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #7 0xffffffff8154ebfc in zio_done (zio=0xfffff8004884b398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #8 0xffffffff81548d54 in zio_execute (zio=0xfffff8004884b398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #9 0xffffffff8154ebfc in zio_done (zio=0xfffff80044c0a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #10 0xffffffff81548d54 in zio_execute (zio=0xfffff80044c0a000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #11 0xffffffff8154ebfc in zio_done (zio=0xfffff80044a2fac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #12 0xffffffff81548d54 in zio_execute (zio=0xfffff80044a2fac8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #13 0xffffffff8154ebfc in zio_done (zio=0xfffff80044853398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3258 #14 0xffffffff81548d54 in zio_execute (zio=0xfffff80044853398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #15 0xffffffff8154ea2a in zio_done (zio=0xfffff8004877e398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3313 #16 0xffffffff81548d54 in zio_execute (zio=0xfffff8004877e398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #17 0xffffffff8154ea2a in zio_done (zio=0xfffff80044cb0730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3313 #18 0xffffffff81548d54 in zio_execute (zio=0xfffff80044cb0730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1407 #19 0xffffffff80651410 in taskqueue_run_locked (queue=0xfffff800488cf400) at /usr/src/sys/kern/subr_taskqueue.c:342 #20 0xffffffff80651dcb in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:563 Along with: (kgdb) print (*(zio_t *)0xfffff80044853398)->io_error $20 = 28 (kgdb) print (*(zio_t *)0xfffff80044a2fac8)->io_error $21 = 28 grep 28 /usr/include/sys/errno.h #define ENOSPC 28 /* No space left on device */ So the issue is simply the pool is out of space to perform the import as that process, when not readonly, requires space to write to the pool. The problem with that is that during this process it has the pool lock so any subsequent zpool actions are dead in the water as they will block waiting on that lock. Something to discuss with the openzfs guys, but I would say the import should fail with a no space error. So Mark the mystery is solved, when you upgraded you ran the pool so low on space that it now can't be imported RW as that requires a write. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 20:47:27 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25741631 for ; Mon, 13 Oct 2014 20:47:27 +0000 (UTC) Received: from gw.catspoiler.org (cl-1657.chi-02.us.sixxs.net [IPv6:2001:4978:f:678::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C98D3D1A for ; Mon, 13 Oct 2014 20:47:26 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id s9DKlGxD030176; Mon, 13 Oct 2014 13:47:20 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201410132047.s9DKlGxD030176@gw.catspoiler.org> Date: Mon, 13 Oct 2014 13:47:16 -0700 (PDT) From: Don Lewis Subject: Re: getting to 4K disk blocks in ZFS To: killing@multiplay.co.uk In-Reply-To: <43D94A22FBD2477FBBDEFF16C0088DDA@multiplay.co.uk> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-stable@FreeBSD.org, fullermd@over-yonder.net X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 20:47:27 -0000 On 13 Oct, Steven Hartland wrote: > ----- Original Message ----- > From: "Matthew D. Fuller" > > >> On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of >> Darren Pilgrim, and lo! it spake thus: >>> >>> If the default is 4k and (for the limited time they're still common) >>> you use true 512b disks, you can waste space. Sure, but how much >>> space? >> >> The median file in /usr/ports is 408 bytes. Over 90% of the files are >> under 2k, which means the wastage for them is over 100% (before >> counting what gain compression might get). A little offhand mathery >> says it's about 78% extra overhead on the whole. >> >> And that includes the almost hundred megs (over 22% of the total size >> of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files >> (another 20%). If you pull those out, the overhead is 130%. >> >> >> (To be sure, relatively few people have ports trees eating most of >> their space, but still; it's pretty pathological. I for one did >> decide some years back to always force 4k on any new FSen to make >> future life simpler, accepting the bloat, but it's there.) > > And thats before you add the overhead if your running RAIDZ... > > A good read on this is > http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ This is a timely subject. I'm planning on moving my Cyrus imap mail spool from a 4K/1K UFS filesystem to a three drive raidz1. It looks like the UFS fragmentation overhead is about 2.4%. ZFS ashift=12 increases that to about 17%. Combine that with raidz and now the overhead is about 40%. Ouch! From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 20:55:43 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03DAB7A1; Mon, 13 Oct 2014 20:55:43 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 80D49DE6; Mon, 13 Oct 2014 20:55:42 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGsbw4pMNzNr; Mon, 13 Oct 2014 22:55:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:from:from :date:date:message-id:received:received:received; s=jakla4; t= 1413233736; x=1415825737; bh=6vVXfSie0JrMZ/XRKkdzqrFhYP73Kvzoe9z 5MW9q7cg=; b=FcmCvr5bkoqpBHw1IS279ppSl6ayjJqUiaDl0rgvZ8J+QuEkfZ2 rU94jkMQko9tvYD853yVtoBLmt8CR2d6mk25JiOH/DHNLMvFvV4NlwDkzVKxse3H eG1/ck+EEqLs6cNuMSgTvzMP7QDcF2KKP+gkhh5AMUagPC0LEDxhp3bo= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id nPgztBolmqQc; Mon, 13 Oct 2014 22:55:36 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Mon, 13 Oct 2014 22:55:36 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGsbq6N3lzSJ; Mon, 13 Oct 2014 22:55:35 +0200 (CEST) Message-ID: <543C3C47.4010208@ijs.si> Date: Mon, 13 Oct 2014 22:55:35 +0200 From: mark User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> In-Reply-To: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 20:55:43 -0000 On 10/13/2014 22:10, Steven Hartland wrote: > So the issue is simply the pool is out of space to perform the import > as that process, when not readonly, requires space to write to the pool. > > The problem with that is that during this process it has the pool lock so > any subsequent zpool actions are dead in the water as they will block > waiting on that lock. > > Something to discuss with the openzfs guys, but I would say the import > should fail with a no space error. > > So Mark the mystery is solved, when you upgraded you ran the pool so low > on space that it now can't be imported RW as that requires a write. > > Regards > Steve Thank you both for analysis and effort! I can't rule out the possibility that my main system pool on a SSD was low on space at some point in time, but the three 4 GiB cloned pools (sys1boot and its brothers) were all created as a zfs send / receive copies of the main / (root) file system and I haven't noticed anything unusual during syncing. This syncing was done manually (using zxfer) and independently from the upgrade on the system - on a steady/quiet system, when the source file system definitely had sufficient free space. The source file system now shows 1.2 GiB of usage shown by df: shiny/ROOT 61758388 1271620 60486768 2% / Seems unlikely that the 1.2 GiB has grown to 4 GiB space on a cloned filesystem. Will try to import the main two pools after re-creating a sane boot pool... Mark From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 21:25:24 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B0AAE1F8; Mon, 13 Oct 2014 21:25:24 +0000 (UTC) Received: from orthanc.ca (orthanc.ca [IPv6:2607:f2f8:abf8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "orthanc.ca", Issuer "orthanc.ca CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 772C6142; Mon, 13 Oct 2014 21:25:24 +0000 (UTC) Received: from [192.168.42.6] (d66-183-211-183.bchsia.telus.net [66.183.211.183]) (authenticated bits=0) by orthanc.ca (8.14.7/8.14.7) with ESMTP id s9DLPJUd041285 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Mon, 13 Oct 2014 14:25:22 -0700 (PDT) (envelope-from lyndon@orthanc.ca) Content-Type: multipart/signed; boundary="Apple-Mail=_F8830B7C-F8B5-4799-BA91-F14FEB3DD492"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: getting to 4K disk blocks in ZFS From: Lyndon Nerenberg In-Reply-To: <201410132047.s9DKlGxD030176@gw.catspoiler.org> Date: Mon, 13 Oct 2014 14:25:19 -0700 Message-Id: References: <201410132047.s9DKlGxD030176@gw.catspoiler.org> To: Don Lewis X-Mailer: Apple Mail (2.1878.6) X-Spam-Status: No, score=0.4 required=5.0 tests=RDNS_DYNAMIC autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on orthanc.ca Cc: freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 21:25:24 -0000 --Apple-Mail=_F8830B7C-F8B5-4799-BA91-F14FEB3DD492 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Oct 13, 2014, at 1:47 PM, Don Lewis wrote: > Combine that with raidz and now the > overhead is about 40%. Ouch! But ~33 of that 40% is generic RAID overhead, and has nothing to do with = the 4K block size issue. (I.e., you would have 33% hit even if you were = running UFS on a three disk RAID5.) On any real-world system where you're running ZFS, it's unlikely the 4K = block overhead is really going to be an issue. And the underlying disk = hardware is moving to 4K physical sectors, anyway. Sooner or later = you're just going to have to suck it up. --lyndon --Apple-Mail=_F8830B7C-F8B5-4799-BA91-F14FEB3DD492 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQIcBAEBAgAGBQJUPEM/AAoJEG8PnXiV/JnU/fwP/i5a9ns87syp7G8aGMPVu7tk UMQilNakksxbIsVxmLk/qKgnktAaEX6e871KPSSAgB34EZkV5UCpdARMtCD1Vdmu iJVCdwqODtz2pgWYI7T1PYP2swnl2gpQvWt1SToMEITP0Tyqeqh184Cmj9fMoUI+ WwuIZ3emlF4VXBPCJxYDyR7BsbBEM8MMYzlwBxSGTTZzJoqoPwOMYrmx4fW3ALXe SnV7buUTSIMUYfIHfIlMDiOrfFS7/n8xy0cKFSyHCGW/uzTDVdfMEWzHZPc/ALQD 7+rQZRXTIHya/0xkksLnBTPZQ5o9/+PGZ5k0UIgGQCPFGXLSiRSHgsz/FGlg467i w+38g1933+iMTFoawmU4QV+YvrwWnddqcZbcqI6zM9kcCOy47LUJ7+oKildg421u AXOmVMEtJ6v27pK0xcBUqPGeP/GGzXvMZF4GV251iL7W7LcAqC8jLBVaRvT/uRYn U5h4IM4FDBQYsb3dW6N6mkjTXK3eo2hc7iwZnZCrA6qIxL5DqmbBxmrXC/wtuR/8 UFA7pS4qP+NKoTWXq7vxZhpm4KqZSBVCSoSQpNRUjhM/r++VUSRNsQYb01DSKj1F FOFCea+Opwgj1Na0/IWInecIeH9IcUQYyjD2y3l1aedhlvsmaFwkm1tF2J2FIu70 1jT9aCRZSkfcV9is5/nD =TiSk -----END PGP SIGNATURE----- --Apple-Mail=_F8830B7C-F8B5-4799-BA91-F14FEB3DD492-- From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 21:56:52 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0EFFA78B; Mon, 13 Oct 2014 21:56:52 +0000 (UTC) Received: from mail-in7.apple.com (mail-out7.apple.com [17.151.62.29]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D7B66679; Mon, 13 Oct 2014 21:56:51 +0000 (UTC) Received: from mail-out.apple.com (mail-out.apple.com [17.151.62.51]) (using TLS with cipher RC4-MD5 (128/128 bits)) (Client did not present a certificate) by mail-in7.apple.com (Apple Secure Mail Relay) with SMTP id 7C.2D.31401.3AA4C345; Mon, 13 Oct 2014 14:56:51 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from relay5.apple.com ([17.128.113.88]) by local.mail-out.apple.com (Oracle Communications Messaging Server 7.0.5.30.0 64bit (built Oct 22 2013)) with ESMTP id <0NDE00MAKKYRJD51@local.mail-out.apple.com>; Mon, 13 Oct 2014 14:56:51 -0700 (PDT) X-AuditID: 11973e16-f793b6d000007aa9-68-543c4aa35376 Received: from [17.149.228.210] (Unknown_Domain [17.149.228.210]) (using TLS with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by relay5.apple.com (Apple SCV relay) with SMTP id 4C.54.31858.4AA4C345; Mon, 13 Oct 2014 14:56:52 -0700 (PDT) Subject: Re: getting to 4K disk blocks in ZFS From: Charles Swiger In-reply-to: Date: Mon, 13 Oct 2014 14:56:49 -0700 Message-id: <77AA5757-5DC1-415B-899E-30545BF91516@mac.com> References: <201410132047.s9DKlGxD030176@gw.catspoiler.org> To: Lyndon Nerenberg X-Mailer: Apple Mail (2.1878.6) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMLMWRmVeSWpSXmKPExsUiON3OWHexl02IQUeLlMXhZiGLk03NrA5M HjM+zWcJYIzisklJzcksSy3St0vgymjumMlSMJ+tYtbcbqYGxi7WLkZODgkBE4nV99awQNhi EhfurWfrYuTiEBKYxSQxcc9BdpAEr4CgxI/J94CKODiYBeQlDp6XBQkzC2hJfH/UygJR38Qk 0fN4AhPM0Aub+xghEv1MEhv3rQJLCAvoSjy7/YcRZBCbgJrEhIk8IGFOATuJCSs+ge1iEVCV +DP5GgvEAgeJjpnbWUHKeQWsJC79MQAJCwnkS/xp+gM2UURAQ2L69X9Qa+UlPnw4zg6yVkLg N6vE1YVv2ScwCs9C8sIshBdmIXlhASPzKkah3MTMHN3MPHO9xIKCnFS95PzcTYyQUBbbwfhw ldUhRgEORiUe3k4n6xAh1sSy4srcQ4zSHCxK4rwflGxChATSE0tSs1NTC1KL4otKc1KLDzEy cXBKNTC6vniWN03Es/pE1+npORalbaInRKxTc0S9wwNa33pc3LJwisvW9e58N9ebmcjOzGJ6 t63uVrEew3WWyF0y/3g9xO/w8Hmd1zL+6q5idqWLI9jJ5c7V9x+YdV48XdFXzcPhnRa+wHXH G8fsCZYGOfvFjtbt0Z/PXh70X/D3HU3pHR9/3Did5qPEUpyRaKjFXFScCAAGd4oiRgIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrGLMWRmVeSWpSXmKPExsUiOPXJJd0lXjYhBh9e8Focbhay2DL7CIvF yaZmVgdmjxmf5rN43H31mSmAKYrLJiU1J7MstUjfLoEro+9PL0vBNraKKef72BsYp7B2MXJy SAiYSFzY3McIYYtJXLi3nq2LkYtDSKCfSWLW8hY2kASzgJbEjX8vmUBsXgEDiSW7NjGD2MIC uhLPbv8BaubgYBNQk5gwkQckzClgJ7H8RSvYfBYBVYk/k6+xQIxxkuhZeARqpLbEsoWvmSFG Wkk8/tTODmILCeRKTJ14GSwuIqAhMf36PyaI2+QlPnw4zj6BkX8WkotmIbloFpKxCxiZVzEK FKXmJFaa6iUWFOSk6iXn525iBIVeQ2HEDsb/y6wOMQpwMCrx8Fr8sQoRYk0sK67MPcQowcGs JMJbyWUTIsSbklhZlVqUH19UmpNafIhRmoNFSZx3owJQSiA9sSQ1OzW1ILUIJsvEwSnVwCjG ovl7//xd0x7NOpayy5DFqtGHK3TD7c2NiW8ORP7l/f3p+zeWb8l/o65PlzrwU8jpp7CQVbKI Cv8bMwf+esfgiqhHavdtqwSzsibfKFrxSeD4krN/N0SlN+/huMppwbT0+NQGe860s5PXHFGp vTfjjxDTVrG8L1zSzhe3l24NSv2cHS30y1uJpTgj0VCLuag4EQDoI0ejOQIAAA== Cc: Don Lewis , freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 21:56:52 -0000 Hi-- On Oct 13, 2014, at 2:25 PM, Lyndon Nerenberg wrote: [ ... ] > On any real-world system where you're running ZFS, it's unlikely the 4K block overhead is really going to be an issue. And the underlying disk hardware is moving to 4K physical sectors, anyway. Sooner or later you're just going to have to suck it up. Or SSDs, which currently have anywhere from 2KB to 16KB "sectors". I suspect that MIX -- http://en.wikipedia.org/wiki/MIX_%28Email%29 -- will gain in popularity. Big messages are kept one per file, just as Maildir does, but MIX also does a pretty good job of conserving inodes (or equivalent) and minimizing wasted space from intrinsic fragmentation due to filesystem blocksize by aggregating small messages together. Regards, -- -Chuck From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 21:56:56 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07E5680A; Mon, 13 Oct 2014 21:56:56 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B67AB67A; Mon, 13 Oct 2014 21:56:55 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 4284820E708FA; Mon, 13 Oct 2014 21:56:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id D2AB920E708F8; Mon, 13 Oct 2014 21:56:51 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "mark" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Mon, 13 Oct 2014 22:56:48 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 21:56:56 -0000 ----- Original Message ----- From: "mark" To: ; Cc: "FreeBSD Stable" Sent: Monday, October 13, 2014 9:55 PM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > On 10/13/2014 22:10, Steven Hartland wrote: >> So the issue is simply the pool is out of space to perform the import >> as that process, when not readonly, requires space to write to the pool. >> >> The problem with that is that during this process it has the pool lock so >> any subsequent zpool actions are dead in the water as they will block >> waiting on that lock. >> >> Something to discuss with the openzfs guys, but I would say the import >> should fail with a no space error. >> >> So Mark the mystery is solved, when you upgraded you ran the pool so low >> on space that it now can't be imported RW as that requires a write. >> >> Regards >> Steve > > Thank you both for analysis and effort! > > I can't rule out the possibility that my main system pool > on a SSD was low on space at some point in time, but the > three 4 GiB cloned pools (sys1boot and its brothers) were all > created as a zfs send / receive copies of the main / (root) > file system and I haven't noticed anything unusual during > syncing. This syncing was done manually (using zxfer) and > independently from the upgrade on the system - on a steady/quiet > system, when the source file system definitely had sufficient > free space. > > The source file system now shows 1.2 GiB of usage shown > by df: > shiny/ROOT 61758388 1271620 60486768 2% / > Seems unlikely that the 1.2 GiB has grown to 4 GiB space > on a cloned filesystem. > > Will try to import the main two pools after re-creating > a sane boot pool... Yer zfs list only shows around 2-3GB used too but zpool list shows the pool is out of space. Cant rule out an accounting issue though. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 22:51:04 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BB1FCAA for ; Mon, 13 Oct 2014 22:51:04 +0000 (UTC) Received: from gw.catspoiler.org (cl-1657.chi-02.us.sixxs.net [IPv6:2001:4978:f:678::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D3F05BC2 for ; Mon, 13 Oct 2014 22:51:03 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id s9DMotuq030403; Mon, 13 Oct 2014 15:50:59 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201410132250.s9DMotuq030403@gw.catspoiler.org> Date: Mon, 13 Oct 2014 15:50:55 -0700 (PDT) From: Don Lewis Subject: Re: getting to 4K disk blocks in ZFS To: lyndon@orthanc.ca In-Reply-To: MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 22:51:04 -0000 On 13 Oct, Lyndon Nerenberg wrote: > > On Oct 13, 2014, at 1:47 PM, Don Lewis wrote: > >> Combine that with raidz and now the >> overhead is about 40%. Ouch! > > But ~33 of that 40% is generic RAID overhead, and has nothing to do > with the 4K block size issue. (I.e., you would have 33% hit even if > you were running UFS on a three disk RAID5.) Actually, I meant this as just the fragmentation overhead. I was incorrectly thinking that there was one parity block per stripe, and with a three disk raidz1, I'd get two data sectors and one parity sector per stripe, and that even a tiny file would get two data sectors and one parity sector. The 40% wasted space was calcualted based on an effective 8K sector size. Parity would then add another 50% to the space. It turns out that this isn't the way it works, at least according to my inderstanding from the previously posted link. A tiny file only gets allocated a single data sector, but also gets its own dedicated parity sector. This helps reduce the waste from fragmentation, but increases the parity overhead. With ashift=12, if you filled the filessytem with 4K files, half the space would be consumed by parity blocks instead of the expected 1/3rd. If you reduce the file size to 512b, you still can't fit in any more files because each file would still require a 4K data sector and a 4K parity sector, so only 6.25% of the space consumed would hold useful data. It looks like my mail spool would grow by a factor of about 1.9 because of fragmentation and parity overhead. Actually, not as bad as I feared it could be, but not much more efficient than a mirror. > On any real-world system where you're running ZFS, it's unlikely the > 4K block overhead is really going to be an issue. And the underlying > disk hardware is moving to 4K physical sectors, anyway. Sooner or > later you're just going to have to suck it up. If you're storing enough small files to make it worthwhile to optimize for that case, you might be better off with older technology 512b sector drives than newer 4K sector drives that have to be 8x larger to store the same amount of data. Drive capacities aren't growing nearly as fast anymore. How long does it take drive capacity to go up by a factor of 8x? From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 23:02:25 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0CC0F3B for ; Mon, 13 Oct 2014 23:02:25 +0000 (UTC) Received: from gw.catspoiler.org (cl-1657.chi-02.us.sixxs.net [IPv6:2001:4978:f:678::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83A01D30 for ; Mon, 13 Oct 2014 23:02:25 +0000 (UTC) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.13.3/8.13.3) with ESMTP id s9DN2F91030438; Mon, 13 Oct 2014 16:02:19 -0700 (PDT) (envelope-from truckman@FreeBSD.org) Message-Id: <201410132302.s9DN2F91030438@gw.catspoiler.org> Date: Mon, 13 Oct 2014 16:02:15 -0700 (PDT) From: Don Lewis Subject: Re: getting to 4K disk blocks in ZFS To: cswiger@mac.com In-Reply-To: <77AA5757-5DC1-415B-899E-30545BF91516@mac.com> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Cc: freebsd-stable@FreeBSD.org, lyndon@orthanc.ca X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 23:02:25 -0000 On 13 Oct, Charles Swiger wrote: > Hi-- > > On Oct 13, 2014, at 2:25 PM, Lyndon Nerenberg > wrote: > [ ... ] >> On any real-world system where you're running ZFS, it's unlikely the >> 4K block overhead is really going to be an issue. And the underlying >> disk hardware is moving to 4K physical sectors, anyway. Sooner or >> later you're just going to have to suck it up. > > Or SSDs, which currently have anywhere from 2KB to 16KB "sectors". Which is even worse because you're more likely to care about wasted space because of the much higher cost per byte. > I suspect that MIX -- http://en.wikipedia.org/wiki/MIX_%28Email%29 -- > will gain in popularity. Big messages are kept one per file, just as > Maildir does, but MIX also does a pretty good job of conserving inodes > (or equivalent) and minimizing wasted space from intrinsic > fragmentation due to filesystem blocksize by aggregating small > messages together. Interesting, but it would be nice to have a more generic solution that could be used to solve the equivalent problem with /usr/ports and similar sorts of things. For instance, it looks like /usr/src expands by quite a bit on an ashift=12 raidz1, though not quite as much as my mail spool. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 23:17:07 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FC79274; Mon, 13 Oct 2014 23:17:07 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 24903E44; Mon, 13 Oct 2014 23:17:06 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id E8A1C20E70905; Mon, 13 Oct 2014 23:17:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 1231E20E70903; Mon, 13 Oct 2014 23:17:03 +0000 (UTC) Message-ID: <49F465E3114F4244955B87BC7E90D093@multiplay.co.uk> From: "Steven Hartland" To: "Don Lewis" , References: <201410132302.s9DN2F91030438@gw.catspoiler.org> Subject: Re: getting to 4K disk blocks in ZFS Date: Tue, 14 Oct 2014 00:17:00 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: lyndon@orthanc.ca, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 23:17:07 -0000 ----- Original Message ----- From: "Don Lewis" > On 13 Oct, Charles Swiger wrote: >> Hi-- >> >> On Oct 13, 2014, at 2:25 PM, Lyndon Nerenberg >> wrote: >> [ ... ] >>> On any real-world system where you're running ZFS, it's unlikely the >>> 4K block overhead is really going to be an issue. And the underlying >>> disk hardware is moving to 4K physical sectors, anyway. Sooner or >>> later you're just going to have to suck it up. >> >> Or SSDs, which currently have anywhere from 2KB to 16KB "sectors". > > Which is even worse because you're more likely to care about wasted > space because of the much higher cost per byte. > >> I suspect that MIX -- http://en.wikipedia.org/wiki/MIX_%28Email%29 -- >> will gain in popularity. Big messages are kept one per file, just as >> Maildir does, but MIX also does a pretty good job of conserving inodes >> (or equivalent) and minimizing wasted space from intrinsic >> fragmentation due to filesystem blocksize by aggregating small >> messages together. > > Interesting, but it would be nice to have a more generic solution that > could be used to solve the equivalent problem with /usr/ports and > similar sorts of things. For instance, it looks like /usr/src expands > by quite a bit on an ashift=12 raidz1, though not quite as much as my > mail spool. Dont worry about ports just create a volume set it to lz4 and forget about it. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 23:48:08 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 76D22B19 for ; Mon, 13 Oct 2014 23:48:08 +0000 (UTC) Received: from luigi.brtsvcs.net (luigi.brtsvcs.net [IPv6:2607:fc50:1000:1f00::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5198C3F9 for ; Mon, 13 Oct 2014 23:48:08 +0000 (UTC) Received: from chombo.houseloki.net (unknown [IPv6:2601:7:400:640:21c:c0ff:fe7f:96ee]) by luigi.brtsvcs.net (Postfix) with ESMTPSA id 7D9F62D4F9B; Mon, 13 Oct 2014 16:48:00 -0700 (PDT) Received: from [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29] (unknown [IPv6:2601:7:2580:674:baca:3aff:fe83:bd29]) by chombo.houseloki.net (Postfix) with ESMTPSA id 14AFEF37; Mon, 13 Oct 2014 16:47:58 -0700 (PDT) Message-ID: <543C64AA.1000307@bluerosetech.com> Date: Mon, 13 Oct 2014 16:47:54 -0700 From: Darren Pilgrim Reply-To: freebsd-stable User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: "Matthew D. Fuller" Subject: Re: getting to 4K disk blocks in ZFS References: <540FF3C4.6010305@ish.com.au> <54100258.2000505@freebsd.org> <5410F0B4.9040808@ish.com.au> <543C1E7B.4090204@bluerosetech.com> <20141013191903.GR2161@over-yonder.net> In-Reply-To: <20141013191903.GR2161@over-yonder.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 23:48:08 -0000 On 10/13/2014 12:19 PM, Matthew D. Fuller wrote: > On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of > Darren Pilgrim, and lo! it spake thus: >> >> If the default is 4k and (for the limited time they're still common) >> you use true 512b disks, you can waste space. Sure, but how much >> space? > > The median file in /usr/ports is 408 bytes. Over 90% of the files are > under 2k, which means the wastage for them is over 100% (before > counting what gain compression might get). A little offhand mathery > says it's about 78% extra overhead on the whole. > > And that includes the almost hundred megs (over 22% of the total size > of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files > (another 20%). If you pull those out, the overhead is 130%. The worst case is about 140%. But that's 140% of 388 MB, so we go from tiny to somewhat less tiny. Huge slack is expected in small, populous datasets. I'd much rather have the integrity assurance and performance gains, instead of saving a few hundred MB out of a multi-terabyte pool. From owner-freebsd-stable@FreeBSD.ORG Mon Oct 13 23:51:29 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2F4F0D2A; Mon, 13 Oct 2014 23:51:29 +0000 (UTC) Received: from mail-in2.apple.com (mail-out2.apple.com [17.151.62.25]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 064355F2; Mon, 13 Oct 2014 23:51:28 +0000 (UTC) Received: from mail-out.apple.com (mail-out.apple.com [17.151.62.51]) (using TLS with cipher RC4-MD5 (128/128 bits)) (Client did not present a certificate) by mail-in2.apple.com (Apple Secure Mail Relay) with SMTP id 6F.D7.26497.A756C345; Mon, 13 Oct 2014 16:51:22 -0700 (PDT) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from relay4.apple.com ([17.128.113.87]) by local.mail-out.apple.com (Oracle Communications Messaging Server 7.0.5.30.0 64bit (built Oct 22 2013)) with ESMTP id <0NDE00CQ8Q9MI841@local.mail-out.apple.com>; Mon, 13 Oct 2014 16:51:22 -0700 (PDT) X-AuditID: 11973e11-f79f76d000006781-57-543c657a60d4 Received: from [17.149.228.210] (Unknown_Domain [17.149.228.210]) (using TLS with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by relay4.apple.com (Apple SCV relay) with SMTP id D7.A8.03493.3856C345; Mon, 13 Oct 2014 16:51:31 -0700 (PDT) Subject: Re: getting to 4K disk blocks in ZFS From: Charles Swiger In-reply-to: <201410132302.s9DN2F91030438@gw.catspoiler.org> Date: Mon, 13 Oct 2014 16:51:21 -0700 Message-id: <2272C292-3FAE-49CB-968A-E31A606EC77C@mac.com> References: <201410132302.s9DN2F91030438@gw.catspoiler.org> To: Don Lewis X-Mailer: Apple Mail (2.1878.6) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrILMWRmVeSWpSXmKPExsUiON3OWLcq1SbE4NNHZYvDzUIWJ5uaWR2Y PGZ8ms8SwBjFZZOSmpNZllqkb5fAlfHh/nSWghMCFb1v77A0MM7i7WLk5JAQMJFY1H2EEcIW k7hwbz1bFyMXh5DALCaJ9VcnMYEkeAUEJX5MvsfSxcjBwSwgL3HwvCxImFlAS+L7o1awsJBA E5PEU0YQE2TknMdGEFP6mSQ2XH7CClIuLKAr8ez2H7AaNgE1iQkTeUDCnAI2EutObWcGsVkE VCWuzHrADjHdUOL/lS0sEAdYSdyb9QnMFhKwlmj6OhnMFhFQkZjY85cF4np5iQ8fjrOD7JUQ +M4q0b1iBuMERuFZSB6YhfDALCQPLGBkXsUolJuYmaObmWekl1hQkJOql5yfu4kREsaCOxiP r7I6xCjAwajEwysRbhMixJpYVlyZe4hRmoNFSZy3MwooJJCeWJKanZpakFoUX1Sak1p8iJGJ g1OqgdHjYmq83K//kdNeLTA22HZS5aGMtvCynQJyycdz/+8/d0FA6pMduzZ7klrRplg/4dBL Rkz9c3IvFlgfOdxSl3hk5v4vJcWxd8+G7mN6uKxtt3LKU/Mrsjydj5WC+bKaja94f6/4dPdk xD3fsmDj+svW8rsvq5S/nb76s1Ytm5XVCbnDc/i8m5VYijMSDbWYi4oTAWcCf9dEAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrKLMWRmVeSWpSXmKPExsUiOPXJJd3mVJsQgzk/DS0ONwtZbJl9hMXi ZFMzqwOzx4xP81k87r76zBTAFMVlk5Kak1mWWqRvl8CV8eH+dJaCEwIVvW/vsDQwzuLtYuTg kBAwkZjz2KiLkRPIFJO4cG89WxcjF4eQQD+TxLlVlxhBEswCWhI3/r1kArF5BQwkluzaxAxi CwvoSjy7/YcRZA6bgJrEhIk8IGFOARuJM2+2gZWzCKhKXJn1gB1ijLHE8RXzoGx5ie1v5zBD jLSS6Py8HWyVkIC1RNPXySwgtoiAisTEnr8sELfJS3z4cJx9AiP/LCQXzUJy0SwkYxcwMq9i FChKzUmsNNFLLCjISdVLzs/dxAgKvIbC8B2M/5ZZHWIU4GBU4uEtiLQJEWJNLCuuzD3EKMHB rCTC+zYEKMSbklhZlVqUH19UmpNafIhRmoNFSZyXIRooJZCeWJKanZpakFoEk2Xi4JRqYGQ6 9cSp36hLrikvKP5A99tOzpPzMl4zxNz6YmS35hUbm/L3jaa9yXdCo9mi5uSZlm/Vfz1buONJ 6xWpfWkTuc7/vHyA9d7jCFfpdYan6niXOPwNZOeYs85/+Y9GWYZKUe+duSlGn1L+NunVSCkv MrUQzrm2vLGU74YHT7b93XX5xr02Vfc6lViKMxINtZiLihMBXbykRjgCAAA= Cc: freebsd-stable@FreeBSD.org, lyndon@orthanc.ca X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Oct 2014 23:51:29 -0000 On Oct 13, 2014, at 4:02 PM, Don Lewis wrote: > On 13 Oct, Charles Swiger wrote: >> Hi-- >> >> On Oct 13, 2014, at 2:25 PM, Lyndon Nerenberg >> wrote: >> [ ... ] >>> On any real-world system where you're running ZFS, it's unlikely the >>> 4K block overhead is really going to be an issue. And the underlying >>> disk hardware is moving to 4K physical sectors, anyway. Sooner or >>> later you're just going to have to suck it up. >> >> Or SSDs, which currently have anywhere from 2KB to 16KB "sectors". > > Which is even worse because you're more likely to care about wasted > space because of the much higher cost per byte. Yep. Mail spools see a surprising amount of disk activity, and while SSDs do wonderfully for the read side and seek times, lots of small writes isn't something they handle very well. >> I suspect that MIX -- http://en.wikipedia.org/wiki/MIX_%28Email%29 -- >> will gain in popularity. Big messages are kept one per file, just as >> Maildir does, but MIX also does a pretty good job of conserving inodes >> (or equivalent) and minimizing wasted space from intrinsic >> fragmentation due to filesystem blocksize by aggregating small >> messages together. > > Interesting, but it would be nice to have a more generic solution that > could be used to solve the equivalent problem with /usr/ports and > similar sorts of things. For instance, it looks like /usr/src expands > by quite a bit on an ashift=12 raidz1, though not quite as much as my > mail spool. For readonly data, it's easy enough to keep the tree in a tarball, iso, or similar and let libarchive / bsdtar / mount -t cd9660 deal with it. As soon as you start trying to update pieces of that content, though, it turns immediately back into the same hard problem. Any storage medium that imposes a minimum physical sector size is going to demand that the filesystem on top of it honor that or suffer expensive read-modify-write cycles when the logical sector size is smaller than the actual physical sector size. Regards, -- -Chuck From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 00:00:04 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BC569EBC; Tue, 14 Oct 2014 00:00:04 +0000 (UTC) Received: from isis.morrow.me.uk (isis.morrow.me.uk [204.109.63.142]) by mx1.freebsd.org (Postfix) with ESMTP id 973E66C6; Tue, 14 Oct 2014 00:00:04 +0000 (UTC) Received: from anubis.morrow.me.uk (unknown [93.89.81.46]) (Authenticated sender: mauzo) by isis.morrow.me.uk (Postfix) with ESMTPSA id 0C89645087; Mon, 13 Oct 2014 23:59:56 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.8.3 isis.morrow.me.uk 0C89645087 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=morrow.me.uk; s=dkim201101; t=1413244797; bh=3kdSKvOiavbBHJXX4Lt48JdMMoeL01rk+eq0i9+qMsE=; h=Date:From:To:Subject:References:In-Reply-To; b=bE4zuGevKufptt8y+4Lznat81ns+nsm+32BayxQrLDmLVkDiijOlPn7moA8eOS1sh 01cGYwtz2r4AH3u8T2Vv0WfBd+F88NMhseMf3GMCrJ9gaMpXtA3JbnPfnKHZoC2BbI UKdeBWEGzrka9C+kSUG76p1itZCFGp8pNa4/Gx/0= X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.98.4 at isis.morrow.me.uk Received: by anubis.morrow.me.uk (Postfix, from userid 5001) id 7AFCF15701; Tue, 14 Oct 2014 00:59:53 +0100 (BST) Date: Tue, 14 Oct 2014 00:59:53 +0100 From: Ben Morrow To: truckman@FreeBSD.org, freebsd-stable@freebsd.org Subject: Re: getting to 4K disk blocks in ZFS Message-ID: <20141013235951.GA43024@anubis.morrow.me.uk> Mail-Followup-To: truckman@FreeBSD.org, freebsd-stable@freebsd.org References: <77AA5757-5DC1-415B-899E-30545BF91516@mac.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201410132302.s9DN2F91030438@gw.catspoiler.org> X-Newsgroups: gmane.os.freebsd.stable User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 00:00:04 -0000 Quoth Don Lewis : > On 13 Oct, Charles Swiger wrote: > > On Oct 13, 2014, at 2:25 PM, Lyndon Nerenberg > > wrote: > > [ ... ] > >> On any real-world system where you're running ZFS, it's unlikely the > >> 4K block overhead is really going to be an issue. And the underlying > >> disk hardware is moving to 4K physical sectors, anyway. Sooner or > >> later you're just going to have to suck it up. [...] > > > I suspect that MIX -- http://en.wikipedia.org/wiki/MIX_%28Email%29 -- > > will gain in popularity. Big messages are kept one per file, just as > > Maildir does, but MIX also does a pretty good job of conserving inodes > > (or equivalent) and minimizing wasted space from intrinsic > > fragmentation due to filesystem blocksize by aggregating small > > messages together. > > Interesting, but it would be nice to have a more generic solution that > could be used to solve the equivalent problem with /usr/ports and > similar sorts of things. For instance, it looks like /usr/src expands > by quite a bit on an ashift=12 raidz1, though not quite as much as my > mail spool. Put a UFS on a zvol? You get the raidz/snapshots of the zpool but since UFS uses fragments you should waste less space with small files. In principle ZFS could use fragments too, though the copy-on-write logic would end up looking exactly like SSD wear-levelling logic, and might be slow enough to be a problem. I don't know if anyone is working on this. (Does anyone know if zvols take notice of BIO_DELETE and return the space to the pool? In the raid case this would be a significant advantage for an fs with a lot of churn, like a frequently-updated /usr/src.) Ben From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 00:08:16 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08B3FCF; Tue, 14 Oct 2014 00:08:16 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id C0F257E8; Tue, 14 Oct 2014 00:08:15 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 3D53420E7090C; Tue, 14 Oct 2014 00:08:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 30AC720E70908; Tue, 14 Oct 2014 00:08:11 +0000 (UTC) Message-ID: <3E07EBCFC6914F9D8BF582DDA08F360D@multiplay.co.uk> From: "Steven Hartland" To: "Ben Morrow" , , References: <77AA5757-5DC1-415B-899E-30545BF91516@mac.com> <20141013235951.GA43024@anubis.morrow.me.uk> Subject: Re: getting to 4K disk blocks in ZFS Date: Tue, 14 Oct 2014 01:08:08 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 00:08:16 -0000 ----- Original Message ----- From: "Ben Morrow" > Put a UFS on a zvol? You get the raidz/snapshots of the zpool but since > UFS uses fragments you should waste less space with small files. So your forcing to do exactly what it did when using 512b sectors just with an extra layer in the middle for no reason. Doesn't sound like a good idea. > In principle ZFS could use fragments too, though the copy-on-write logic > would end up looking exactly like SSD wear-levelling logic, and might be > slow enough to be a problem. I don't know if anyone is working on this. Most SSD's advertise this so you could achieve this if you wanted to but then your just back where you started from. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 01:15:59 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C5426B0; Tue, 14 Oct 2014 01:15:59 +0000 (UTC) Received: from mail-yk0-x235.google.com (mail-yk0-x235.google.com [IPv6:2607:f8b0:4002:c07::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A26BD6C; Tue, 14 Oct 2014 01:15:59 +0000 (UTC) Received: by mail-yk0-f181.google.com with SMTP id q200so3925603ykb.40 for ; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=oqg72uyd7A6ysA/Jre/9gGgkOX8P5fjPtNjpQ0dRirI=; b=nccjuB8TWlrxdg7X2JIQ3UMQ0IQ1yaD2gZpw0YO/VNcaBV2paBJ5qUYR0FCatyk3z9 J6wrMVKPcG//A8JNXZRlIhwLMv2aXx/7oHVPypXgyBMiPqmaXE8totx0Ya7XKyseFaTp 1LF7GELvD/H3rhge39KBRgvw1iHDa3/D9r/amC/DoBgx104NhjCqcPEMRqnbUap3JMyc 8Mq66qjBZORZNbFqumLtr9BAzy7M+kBb6/j8Sdqr4BytlHd6Zp4z3Z7ekX0JadvNBm3d EgqXN7K8ngy12zgyJ+4BVFFsvPXmJp/Y9nmvvwLBIOxtFz3yw5DgOdjM4t7Cp97Z2LYn EyDQ== MIME-Version: 1.0 X-Received: by 10.236.199.46 with SMTP id w34mr3092953yhn.20.1413249358423; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 18:15:58 -0700 (PDT) In-Reply-To: References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Date: Mon, 13 Oct 2014 18:15:58 -0700 X-Google-Sender-Auth: MmfIWfMdJ_ON90WeEuTiHDaEu-g Message-ID: Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Steven Hartland Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable , mark X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:15:59 -0000 >> Thank you both for analysis and effort! >> >> I can't rule out the possibility that my main system pool >> on a SSD was low on space at some point in time, but the >> three 4 GiB cloned pools (sys1boot and its brothers) were all >> created as a zfs send / receive copies of the main / (root) >> file system and I haven't noticed anything unusual during >> syncing. This syncing was done manually (using zxfer) and >> independently from the upgrade on the system - on a steady/quiet >> system, when the source file system definitely had sufficient >> free space. >> >> The source file system now shows 1.2 GiB of usage shown >> by df: >> shiny/ROOT 61758388 1271620 60486768 2% / >> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >> on a cloned filesystem. >> >> Will try to import the main two pools after re-creating >> a sane boot pool... > > > Yer zfs list only shows around 2-3GB used too but zpool list > shows the pool is out of space. Cant rule out an accounting > issue though. > What is using the extra space in the pool? Is there an unmounted dataset or snapshot? Do you know how to easily tell? Unlike txg and zio processing I don't have the luxury of having just read that part of the codebase. Thanks. From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 01:24:26 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0BD7F8C0; Tue, 14 Oct 2014 01:24:26 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id AAA4FE51; Tue, 14 Oct 2014 01:24:25 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jGzZ00r7gz1Pj; Tue, 14 Oct 2014 03:24:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413249860; x=1415841861; bh=E00J03d8mnC6AmiSMuV81G4q/ 4Li+rNK8OluLpXLO8w=; b=NYM7VzCHHu12F0S1O94X/tU+PB5tZoNSVlbkupIdH FfNZBxWgqt0NqzpoMsH1Ps7UXCznZTrFfVDzkO+stidS9kKnHEdeImdjbVOqCpFE CLU2PXdXGtMXyGyqMOVORL9OYeLBwwjC5Kz6Pt/uW25zis+SOx08J2ivWEB+DW9c Po= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id 95NqP2gMnm1A; Tue, 14 Oct 2014 03:24:20 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 03:24:20 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jGzYv6YGTzqr; Tue, 14 Oct 2014 03:24:19 +0200 (CEST) Message-ID: <543C7B43.5020301@ijs.si> Date: Tue, 14 Oct 2014 03:24:19 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org, freebsd-fs@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:24:26 -0000 On 10/14/2014 03:15, K. Macy wrote: > What is using the extra space in the pool? Is there an unmounted > dataset or snapshot? Do you know how to easily tell? Unlike txg and > zio processing I don't have the luxury of having just read that part > of the codebase. Most likely the snapshots (regular periodic snapshots). Changes after upgrading an OS can maybe take an additional 50% of space (just guessing). Btw, ashift=12. Still can't see how that would amount to 4 GiB, but it's possible. Mark From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 01:32:29 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 15AA3AD8; Tue, 14 Oct 2014 01:32:29 +0000 (UTC) Received: from mail-yk0-x234.google.com (mail-yk0-x234.google.com [IPv6:2607:f8b0:4002:c07::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6BDFF1D; Tue, 14 Oct 2014 01:32:28 +0000 (UTC) Received: by mail-yk0-f180.google.com with SMTP id 142so3981126ykq.11 for ; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=MM2GVLyT6Pehvu1oKKeJ4UpP2MiIgQ6VqLc9yOfdDts=; b=z1bNNuOfrK0vIXEmo/JeWKq/CxTQAv/lk+NIVz41KHBOdBn9ylniupC+lvBNhd/JxD fXe/XZQEfqHUtG3BT3RsDDtROc1AZ3R36qfRBrAskb2x/Qb3d2ZGdLvPxUqrwrOZCzx4 gIElWQxuHvH/zD1zFgYSGmKXW0MCEWDH5afeAr0l0UB84lFW+3Kda2HW534klvJ5Yb2k S63BeW+MkkWEDgIKV+inkJcus+L6XyeXcjxZW+WR/eOrOGO9feCNgn+Ud8RviNl5/b3M zDUmoaJ70WFYnkc33IShp5xaBu+hTnxbiY5dzY84zdEK0CbUFXFPbf2jisPLSJk7z63M rc2A== MIME-Version: 1.0 X-Received: by 10.236.231.161 with SMTP id l31mr2441674yhq.104.1413250347868; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Mon, 13 Oct 2014 18:32:27 -0700 (PDT) In-Reply-To: <543C7B43.5020301@ijs.si> References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <543C7B43.5020301@ijs.si> Date: Mon, 13 Oct 2014 18:32:27 -0700 X-Google-Sender-Auth: VfHbR0ZPCoKAbKWaADfhCwJ9OWQ Message-ID: Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] From: "K. Macy" To: Mark Martinec Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 01:32:29 -0000 On Mon, Oct 13, 2014 at 6:24 PM, Mark Martinec wrote: > On 10/14/2014 03:15, K. Macy wrote: >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > > Most likely the snapshots (regular periodic snapshots). > Changes after upgrading an OS can maybe take an additional 50% > of space (just guessing). Btw, ashift=12. > Still can't see how that would amount to 4 GiB, but it's possible. > Disconcerting. Is this something that others are likely to hit? Should accounting for writes fail with ENOSPC a bit earlier so that we never reach a state like this? I.e. non-metadata writes will fail at a lower threshold than data or if that is already the case, reduce the threshold further. -K From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 08:13:36 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5510CAF8 for ; Tue, 14 Oct 2014 08:13:36 +0000 (UTC) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 16BAC8E2 for ; Tue, 14 Oct 2014 08:13:35 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1XdxDq-0006ow-Sj for freebsd-stable@freebsd.org; Tue, 14 Oct 2014 10:13:27 +0200 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: freebsd-stable@freebsd.org Subject: Re: Server insists on wrong hostname References: <5438F7A4.4070908@burnus.net> Date: Tue, 14 Oct 2014 10:12:45 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <5438F7A4.4070908@burnus.net> User-Agent: Opera Mail/12.17 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: -- X-Spam-Score: -2.9 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED, BAYES_00 autolearn=disabled version=3.3.2 X-Scan-Signature: 74b82c602e180a6a7275945acf229479 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:13:36 -0000 On Sat, 11 Oct 2014 11:25:56 +0200, Christian Alge wrote: > Hi, > > I am operating a FreeBSD 10.0-STABLE server and once upon a time I > decided to change its hostname. I edited it in /etc/rc.conf, /etc/hosts > and /etc/defaults/rc.conf. I also set it by invoking `sudo hostname`, > but every time I restart it, it aquires the old name again. I fail to > see, where that setting comes from. Can any of you help me out with > that? Thanks in advance. There is no need to edit /etc/default/rc.conf. The setting in rc.conf overrides it. It can be useful to post your config files. Maybe somebody spots a spelling error for example. Ronald. From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 08:15:07 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 982E7BF4; Tue, 14 Oct 2014 08:15:07 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id E4C7B8F9; Tue, 14 Oct 2014 08:15:06 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 1C80A20E70935; Tue, 14 Oct 2014 08:14:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 42F1120E70934; Tue, 14 Oct 2014 08:14:57 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 09:14:54 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , mark , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:15:07 -0000 ----- Original Message ----- From: "K. Macy" >>> Thank you both for analysis and effort! >>> >>> I can't rule out the possibility that my main system pool >>> on a SSD was low on space at some point in time, but the >>> three 4 GiB cloned pools (sys1boot and its brothers) were all >>> created as a zfs send / receive copies of the main / (root) >>> file system and I haven't noticed anything unusual during >>> syncing. This syncing was done manually (using zxfer) and >>> independently from the upgrade on the system - on a steady/quiet >>> system, when the source file system definitely had sufficient >>> free space. >>> >>> The source file system now shows 1.2 GiB of usage shown >>> by df: >>> shiny/ROOT 61758388 1271620 60486768 2% / >>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >>> on a cloned filesystem. >>> >>> Will try to import the main two pools after re-creating >>> a sane boot pool... >> >> >> Yer zfs list only shows around 2-3GB used too but zpool list >> shows the pool is out of space. Cant rule out an accounting >> issue though. >> > > What is using the extra space in the pool? Is there an unmounted > dataset or snapshot? Do you know how to easily tell? Unlike txg and > zio processing I don't have the luxury of having just read that part > of the codebase. Its not clear but I believe it could just be fragmention even though its ashift=9. I sent the last snapshot to another pool of the same size and it resulted in: NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - I believe FRAG is 0% as the feature wasn't enabled for the lifetime of the pool hence its simply not showing a valid value. zfs list -t all -r sys1boot NAME USED AVAIL REFER MOUNTPOINT sys1boot 1.76G 2.08G 11K /sys1boot sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-08-20_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-21_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-22_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-23_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-24_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-26_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-08-27_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-28_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-08-29_04.00 128K - 1.19G - sys1boot/ROOT@auto-2014-08-31_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-01_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-02_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-03_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-04_04.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-05_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-07_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-08_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-09_04.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-10_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-10_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-11_16.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-11_20.00 84.5K - 1.19G - sys1boot/ROOT@auto-2014-09-12_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-12_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-13_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-14_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-15_20.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_00.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-16_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-16_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-17_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-17_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_16.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_20.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-18_23.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_00.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_01.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_02.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_03.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_04.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_05.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_06.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_07.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_08.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_09.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_10.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_11.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_12.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_13.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_14.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_15.00 1K - 1.19G - sys1boot/ROOT@auto-2014-09-19_16.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_17.00 85.5K - 1.19G - sys1boot/ROOT@auto-2014-09-19_18.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_18.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_19.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_20.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.30 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.40 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_21.50 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.00 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.10 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G - sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G - Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 08:25:12 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D9B1EEF4; Tue, 14 Oct 2014 08:25:12 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 923FC9EE; Tue, 14 Oct 2014 08:25:12 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id DA42B20E70936; Tue, 14 Oct 2014 08:25:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 834ED20E70934; Tue, 14 Oct 2014 08:25:10 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "K. Macy" , "Mark Martinec" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <543C7B43.5020301@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 09:25:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 08:25:13 -0000 ----- Original Message ----- From: "K. Macy" > On Mon, Oct 13, 2014 at 6:24 PM, Mark Martinec > wrote: >> On 10/14/2014 03:15, K. Macy wrote: >>> >>> What is using the extra space in the pool? Is there an unmounted >>> dataset or snapshot? Do you know how to easily tell? Unlike txg and >>> zio processing I don't have the luxury of having just read that part >>> of the codebase. >> >> >> Most likely the snapshots (regular periodic snapshots). >> Changes after upgrading an OS can maybe take an additional 50% >> of space (just guessing). Btw, ashift=12. >> Still can't see how that would amount to 4 GiB, but it's possible. >> > > Disconcerting. Is this something that others are likely to hit? Should > accounting for writes fail with ENOSPC a bit earlier so that we never > reach a state like this? I.e. non-metadata writes will fail at a lower > threshold than data or if that is already the case, reduce the > threshold further. I thought I remembered seeing some recent changes in this area, but I can't find them ATM. Something to raise on the openzfs list. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 09:23:18 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6715A53 for ; Tue, 14 Oct 2014 09:23:18 +0000 (UTC) Received: from mx18-out5.antispamcloud.com (mx18-out5.antispamcloud.com [207.244.64.174]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2DDFEEE8 for ; Tue, 14 Oct 2014 09:23:17 +0000 (UTC) Received: from net06.redwood.com ([91.233.6.246] helo=mail.redwood.com) by mx18.antispamcloud.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1XdxlF-0007Xd-PT for freebsd-stable@freebsd.org; Tue, 14 Oct 2014 10:47:36 +0200 Received: from exceu2.rwd.lan ([fe80::e42a:a959:5640:381a]) by exceu2.rwd.lan ([fe80::e42a:a959:5640:381a%10]) with mapi id 14.01.0438.000; Tue, 14 Oct 2014 10:47:03 +0200 From: Maikel Verheijen To: "freebsd-stable@freebsd.org" Subject: nvd disk on nvme controller not detected at boot-time Thread-Topic: nvd disk on nvme controller not detected at boot-time Thread-Index: AQHP54ttNqAodkLuEEmDqzIhnS+Rzg== Date: Tue, 14 Oct 2014 08:47:03 +0000 Message-ID: Accept-Language: en-US, nl-NL Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.4.140807 x-originating-ip: [10.31.1.9] Content-Type: multipart/mixed; boundary="_004_D062AFA3206A0maikelverheijenredwoodcom_" MIME-Version: 1.0 X-Filter-ID: s0sct1PQhAABKnZB5plbIV6jvtGXzsRLyDxHGPnOSosTzNYXe+HZr4WowsEZlxIc1tZWiYINMnD4 agKXMg0NtLCB1teyBlyHrLQJQcKVj0p6MshLFGmPLwEYVPrYNakSdUv6vb7lzwh4VvH3mLi3t0di 2A4r9us+Fq8MBvFDhJiwBoiMLPNSd4AONee5k3VKjvWCdTUaSHP3aCqCfHs4hmbjO41FyBEqIaDu dcVplPHiP2deNHnCCQnyM4RVY/zyoytlfvVSAzGze0kpVPx/2xlw3lc2fGUzXpAkkHIUoAVWBb39 uS1TjWG2Inx+Ts2Qj8hrpg3mbQyxBZ9hRhprUrGSLuktd0nGVM3eBtn1rnIBEnR5pS0H0ztYMT4H EQRovdF5iICzzviO5xBr9DD4Vr5KRDcS27MOEloxyjZAAnE0bRdiIM9FFlfh4Bu9E5FcL6moPrg7 3tecPvrTIwlxLME9f1vMYmLTJwtuI5QMTJJ2dPA89bStV3EMkuoKt7nZxae5UkOBPRpbEOOkRwK0 bIOoT5f9zNwjlArtXM+EHVJx8dumVr26nCpuktVluILRmJdDwLTy7ggkbtiREBmTEN9TLrF9l3It GfA/WrnALV6l8mEyev4VjNXOhq541pfIB5hQ6nsDvccjqgmDvD9Wh+B3pA4pSkF5s25RT3iDmeE= X-Report-Abuse-To: spam@mx99.antispamcloud.com X-Filter-Fingerprint: IFrWXGses7OKB5S5G8/dJfl+3wvwn7+SNaMrU0HGnonJUWjZ8+qhjyB23tbDuyLOYL8Ff78gYsez 4Rl08xudmXi4esCQ0R1MchVjt7wblGlvhFgW0MjUMRkF5sMCDfftTXNFDzN17hnrWeZYOJvLq0Ic WjZ+XcEjj/7Pkld0zkmvziDInX9WdMov2kn2yXjdwv61T+KDYyYtREgszdyFwv8IxCB3p/oCKvxr eyISh3JGb7OS5oVgiO+kDxZrVPLz3MmEGC2PrUKqLq5WmHK+Nw== X-Originating-IP: 91.233.6.246 X-Spampanel-Domain: redwood.com X-Spampanel-Username: sysadmin Authentication-Results: antispamcloud.com; auth=pass (login) smtp.auth=sysadmin X-Spampanel-Outgoing-Class: unsure X-Spampanel-Outgoing-Evidence: Combined (0.12) X-Recommended-Action: accept X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 09:23:18 -0000 --_004_D062AFA3206A0maikelverheijenredwoodcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi list! We recently purchased a HP DL380e G8 server to serve as our backup server w= ith an Intel P3600 that uses the nvme interface. We added load_nvme=3D"YES"= and load_nvd=3D"YES" to our loader.conf, and they both get loaded, however= the ssd disk device is not detected at boot. When we unload and reload the= nvd module the disk does get detected. Is there a way to see if the load-o= rder is correct? We added verbose_loading=3D"YES" to the loader.conf, but d= mesg doesn't show me the actual loading. One thing I did see related to the nvme in the dmesg (full dmesg attached) = output is this: nvme0: mem 0xfbdf0000-0xfbdf3fff irq 16 at device 0.0= on pci3 =85 nvme0: SET FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000080 cdw11:00000000 nvme0: INTERNAL DEVICE ERROR (00/06) sqid:0 cid:9 cdw0:0 After unloading and reloading the nvd module the disk does get detected: nvd0: NVMe namespace nvd0: 381554MB (781422768 512 byte sectors) Is there anyone that might give me some pointers on how to get the nvd0 dev= ice loaded consistently at boot time? Thanks in advance, Kind regards, Maikel Verheijen --_004_D062AFA3206A0maikelverheijenredwoodcom_ Content-Type: text/plain; name="dmesg_output.txt" Content-Description: dmesg_output.txt Content-Disposition: attachment; filename="dmesg_output.txt"; size=71643; creation-date="Tue, 14 Oct 2014 08:47:03 GMT"; modification-date="Tue, 14 Oct 2014 08:47:03 GMT" Content-ID: Content-Transfer-Encoding: base64 Q29weXJpZ2h0IChjKSAxOTkyLTIwMTQgVGhlIEZyZWVCU0QgUHJvamVjdC4KQ29weXJpZ2h0IChj KSAxOTc5LCAxOTgwLCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAx OTk0CglUaGUgUmVnZW50cyBvZiB0aGUgVW5pdmVyc2l0eSBvZiBDYWxpZm9ybmlhLiBBbGwgcmln aHRzIHJlc2VydmVkLgpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZy ZWVCU0QgRm91bmRhdGlvbi4KRnJlZUJTRCAxMC4wLVJFTEVBU0UtcDkgIzA6IE1vbiBTZXAgMTUg MTQ6MzU6NTIgVVRDIDIwMTQKICAgIHJvb3RAYW1kNjQtYnVpbGRlci5kYWVtb25vbG9neS5uZXQ6 L3Vzci9vYmovdXNyL3NyYy9zeXMvR0VORVJJQyBhbWQ2NApGcmVlQlNEIGNsYW5nIHZlcnNpb24g My4zICh0YWdzL1JFTEVBU0VfMzMvZmluYWwgMTgzNTAyKSAyMDEzMDYxMApDUFU6IEludGVsKFIp IFhlb24oUikgQ1BVIEU1LTI0MjAgdjIgQCAyLjIwR0h6ICgyMTk0Ljc2LU1IeiBLOC1jbGFzcyBD UFUpCiAgT3JpZ2luID0gIkdlbnVpbmVJbnRlbCIgIElkID0gMHgzMDZlNCAgRmFtaWx5ID0gMHg2 ICBNb2RlbCA9IDB4M2UgIFN0ZXBwaW5nID0gNAogIEZlYXR1cmVzPTB4YmZlYmZiZmY8RlBVLFZN RSxERSxQU0UsVFNDLE1TUixQQUUsTUNFLENYOCxBUElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQ QVQsUFNFMzYsQ0xGTFVTSCxEVFMsQUNQSSxNTVgsRlhTUixTU0UsU1NFMixTUyxIVFQsVE0sUEJF PgogIEZlYXR1cmVzMj0weDdmYmVlM2ZmPFNTRTMsUENMTVVMUURRLERURVM2NCxNT04sRFNfQ1BM LFZNWCxTTVgsRVNULFRNMixTU1NFMyxDWDE2LHhUUFIsUERDTSxQQ0lELERDQSxTU0U0LjEsU1NF NC4yLHgyQVBJQyxQT1BDTlQsVFNDRExULEFFU05JLFhTQVZFLE9TWFNBVkUsQVZYLEYxNkMsUkRS QU5EPgogIEFNRCBGZWF0dXJlcz0weDJjMTAwODAwPFNZU0NBTEwsTlgsUGFnZTFHQixSRFRTQ1As TE0+CiAgQU1EIEZlYXR1cmVzMj0weDE8TEFIRj4KICBTdGFuZGFyZCBFeHRlbmRlZCBGZWF0dXJl cz0weDI4MTxHU0ZTQkFTRSxTTUVQLEVOSE1PVlNCPgogIFRTQzogUC1zdGF0ZSBpbnZhcmlhbnQs IHBlcmZvcm1hbmNlIHN0YXRpc3RpY3MKcmVhbCBtZW1vcnkgID0gNjg3MTk0NzY3MzYgKDY1NTM2 IE1CKQphdmFpbCBtZW1vcnkgPSA2NjQ4MTA3NDE3NiAoNjM0MDEgTUIpCkV2ZW50IHRpbWVyICJM QVBJQyIgcXVhbGl0eSA2MDAKQUNQSSBBUElDIFRhYmxlOiA8SFAgICAgIFByb0xpYW50PgpGcmVl QlNEL1NNUDogTXVsdGlwcm9jZXNzb3IgU3lzdGVtIERldGVjdGVkOiAxMiBDUFVzCkZyZWVCU0Qv U01QOiAxIHBhY2thZ2UocykgeCA2IGNvcmUocykgeCAyIFNNVCB0aHJlYWRzCiBjcHUwIChCU1Ap OiBBUElDIElEOiAgMAogY3B1MSAoQVApOiBBUElDIElEOiAgMQogY3B1MiAoQVApOiBBUElDIElE OiAgMgogY3B1MyAoQVApOiBBUElDIElEOiAgMwogY3B1NCAoQVApOiBBUElDIElEOiAgNAogY3B1 NSAoQVApOiBBUElDIElEOiAgNQogY3B1NiAoQVApOiBBUElDIElEOiAgNgogY3B1NyAoQVApOiBB UElDIElEOiAgNwogY3B1OCAoQVApOiBBUElDIElEOiAgOAogY3B1OSAoQVApOiBBUElDIElEOiAg OQogY3B1MTAgKEFQKTogQVBJQyBJRDogMTAKIGNwdTExIChBUCk6IEFQSUMgSUQ6IDExCkFDUEkg QklPUyBXYXJuaW5nIChidWcpOiBJbnZhbGlkIGxlbmd0aCBmb3IgRkFEVC9QbTFhQ29udHJvbEJs b2NrOiAzMiwgdXNpbmcgZGVmYXVsdCAxNiAoMjAxMzA4MjMvdGJmYWR0LTY4MikKQUNQSSBCSU9T IFdhcm5pbmcgKGJ1Zyk6IEludmFsaWQgbGVuZ3RoIGZvciBGQURUL1BtMkNvbnRyb2xCbG9jazog MzIsIHVzaW5nIGRlZmF1bHQgOCAoMjAxMzA4MjMvdGJmYWR0LTY4MikKaW9hcGljMSA8VmVyc2lv biAyLjA+IGlycXMgMjQtNDcgb24gbW90aGVyYm9hcmQKaW9hcGljMCA8VmVyc2lvbiAyLjA+IGly cXMgMC0yMyBvbiBtb3RoZXJib2FyZApyYW5kb206IDxTb2Z0d2FyZSwgWWFycm93PiBpbml0aWFs aXplZAprYmQxIGF0IGtiZG11eDAKYWNwaTA6IDxIUCBQcm9MaWFudD4gb24gbW90aGVyYm9hcmQK YWNwaTA6IFBvd2VyIEJ1dHRvbiAoZml4ZWQpCmNwdTA6IDxBQ1BJIENQVT4gb24gYWNwaTAKY3B1 MTogPEFDUEkgQ1BVPiBvbiBhY3BpMApjcHUyOiA8QUNQSSBDUFU+IG9uIGFjcGkwCmNwdTM6IDxB Q1BJIENQVT4gb24gYWNwaTAKY3B1NDogPEFDUEkgQ1BVPiBvbiBhY3BpMApjcHU1OiA8QUNQSSBD UFU+IG9uIGFjcGkwCmNwdTY6IDxBQ1BJIENQVT4gb24gYWNwaTAKY3B1NzogPEFDUEkgQ1BVPiBv biBhY3BpMApjcHU4OiA8QUNQSSBDUFU+IG9uIGFjcGkwCmNwdTk6IDxBQ1BJIENQVT4gb24gYWNw aTAKY3B1MTA6IDxBQ1BJIENQVT4gb24gYWNwaTAKY3B1MTE6IDxBQ1BJIENQVT4gb24gYWNwaTAK YXR0aW1lcjA6IDxBVCB0aW1lcj4gcG9ydCAweDQwLTB4NDMgaXJxIDAgb24gYWNwaTAKVGltZWNv dW50ZXIgImk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4MiBIeiBxdWFsaXR5IDAKRXZlbnQgdGltZXIg Imk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4MiBIeiBxdWFsaXR5IDEwMApocGV0MDogPEhpZ2ggUHJl Y2lzaW9uIEV2ZW50IFRpbWVyPiBpb21lbSAweGZlZDAwMDAwLTB4ZmVkMDAzZmYgb24gYWNwaTAK VGltZWNvdW50ZXIgIkhQRVQiIGZyZXF1ZW5jeSAxNDMxODE4MCBIeiBxdWFsaXR5IDk1MApFdmVu dCB0aW1lciAiSFBFVCIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzUwCkV2ZW50IHRp bWVyICJIUEVUMSIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVy ICJIUEVUMiIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVyICJI UEVUMyIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVyICJIUEVU NCIgZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVyICJIUEVUNSIg ZnJlcXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVyICJIUEVUNiIgZnJl cXVlbmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCkV2ZW50IHRpbWVyICJIUEVUNyIgZnJlcXVl bmN5IDE0MzE4MTgwIEh6IHF1YWxpdHkgMzQwCmF0cnRjMDogPEFUIHJlYWx0aW1lIGNsb2NrPiBw b3J0IDB4NzAtMHg3MSBvbiBhY3BpMApFdmVudCB0aW1lciAiUlRDIiBmcmVxdWVuY3kgMzI3Njgg SHogcXVhbGl0eSAwClRpbWVjb3VudGVyICJBQ1BJLWZhc3QiIGZyZXF1ZW5jeSAzNTc5NTQ1IEh6 IHF1YWxpdHkgOTAwCmFjcGlfdGltZXIwOiA8MjQtYml0IHRpbWVyIGF0IDMuNTc5NTQ1TUh6PiBw b3J0IDB4OTA4LTB4OTBiIG9uIGFjcGkwCnBjaWIwOiA8QUNQSSBIb3N0LVBDSSBicmlkZ2U+IG9u IGFjcGkwCnBjaTA6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWIwCnBjaWIxOiA8QUNQSSBQQ0ktUENJ IGJyaWRnZT4gYXQgZGV2aWNlIDEuMCBvbiBwY2kwCnBjaTY6IDxBQ1BJIFBDSSBidXM+IG9uIHBj aWIxCnBjaTY6IDxtZW1vcnk+IGF0IGRldmljZSAwLjAgKG5vIGRyaXZlciBhdHRhY2hlZCkKcGNp YjI6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMS4xIG9uIHBjaTAKcGNpNzogPEFD UEkgUENJIGJ1cz4gb24gcGNpYjIKcGNpYjM6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZp Y2UgMy4wIG9uIHBjaTAKcGNpMTA6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWIzCm1maTA6IDxEcmFr ZSBTa2lubnk+IHBvcnQgMHg2MDAwLTB4NjBmZiBtZW0gMHhmYmZmMDAwMC0weGZiZmYzZmZmLDB4 ZmJmODAwMDAtMHhmYmZiZmZmZiBpcnEgNDAgYXQgZGV2aWNlIDAuMCBvbiBwY2kxMAptZmkwOiBV c2luZyBNU0kKbWZpMDogTWVnYXJhaWQgU0FTIGRyaXZlciBWZXIgNC4yMyAKcGNpYjQ6IDxBQ1BJ IFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMy4xIG9uIHBjaTAKcGNpMjU6IDxBQ1BJIFBDSSBi dXM+IG9uIHBjaWI0CnBjaWI1OiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDMuMiBv biBwY2kwCnBjaTEzOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liNQpwY2liNjogPEFDUEkgUENJLVBD SSBicmlkZ2U+IGF0IGRldmljZSAzLjMgb24gcGNpMApwY2kyNjogPEFDUEkgUENJIGJ1cz4gb24g cGNpYjYKcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDQuMCAobm8gZHJpdmVyIGF0 dGFjaGVkKQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgNC4xIChubyBkcml2ZXIg YXR0YWNoZWQpCnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSA0LjIgKG5vIGRyaXZl ciBhdHRhY2hlZCkKcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDQuMyAobm8gZHJp dmVyIGF0dGFjaGVkKQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgNC40IChubyBk cml2ZXIgYXR0YWNoZWQpCnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSA0LjUgKG5v IGRyaXZlciBhdHRhY2hlZCkKcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDQuNiAo bm8gZHJpdmVyIGF0dGFjaGVkKQpwY2kwOiA8YmFzZSBwZXJpcGhlcmFsPiBhdCBkZXZpY2UgNC43 IChubyBkcml2ZXIgYXR0YWNoZWQpCnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSA1 LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkKcGNpMDogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNl IDUuMiAobm8gZHJpdmVyIGF0dGFjaGVkKQpwY2liNzogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0 IGRldmljZSAxNy4wIG9uIHBjaTAKcGNpMjg6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWI3CmVoY2kw OiA8RUhDSSAoZ2VuZXJpYykgVVNCIDIuMCBjb250cm9sbGVyPiBtZW0gMHhmYTQ2MDAwMC0weGZh NDYwM2ZmIGlycSAyMSBhdCBkZXZpY2UgMjYuMCBvbiBwY2kwCnVzYnVzMDogRUhDSSB2ZXJzaW9u IDEuMAp1c2J1czAgb24gZWhjaTAKcGNpYjg6IDxBQ1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZp Y2UgMjguMCBvbiBwY2kwCnBjaTM6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWI4Cm52bWUwOiA8R2Vu ZXJpYyBOVk1lIERldmljZT4gbWVtIDB4ZmJkZjAwMDAtMHhmYmRmM2ZmZiBpcnEgMTYgYXQgZGV2 aWNlIDAuMCBvbiBwY2kzCnBjaWI5OiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDI4 LjQgb24gcGNpMApwY2kyOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liOQppZ2IwOiA8SW50ZWwoUikg UFJPLzEwMDAgTmV0d29yayBDb25uZWN0aW9uIHZlcnNpb24gLSAyLjQuMD4gcG9ydCAweDUwMDAt MHg1MDFmIG1lbSAweGZiYzAwMDAwLTB4ZmJjZmZmZmYsMHhmYmJmMDAwMC0weGZiYmYzZmZmIGly cSAxNiBhdCBkZXZpY2UgMC4wIG9uIHBjaTIKaWdiMDogVXNpbmcgTVNJWCBpbnRlcnJ1cHRzIHdp dGggOSB2ZWN0b3JzCmlnYjA6IEV0aGVybmV0IGFkZHJlc3M6IGZjOjE1OmI0OjI2OjZlOjkwCmln YjA6IEJvdW5kIHF1ZXVlIDAgdG8gY3B1IDAKaWdiMDogQm91bmQgcXVldWUgMSB0byBjcHUgMQpp Z2IwOiBCb3VuZCBxdWV1ZSAyIHRvIGNwdSAyCmlnYjA6IEJvdW5kIHF1ZXVlIDMgdG8gY3B1IDMK aWdiMDogQm91bmQgcXVldWUgNCB0byBjcHUgNAppZ2IwOiBCb3VuZCBxdWV1ZSA1IHRvIGNwdSA1 CmlnYjA6IEJvdW5kIHF1ZXVlIDYgdG8gY3B1IDYKaWdiMDogQm91bmQgcXVldWUgNyB0byBjcHUg NwppZ2IxOiA8SW50ZWwoUikgUFJPLzEwMDAgTmV0d29yayBDb25uZWN0aW9uIHZlcnNpb24gLSAy LjQuMD4gcG9ydCAweDUwMjAtMHg1MDNmIG1lbSAweGZiYTAwMDAwLTB4ZmJhZmZmZmYsMHhmYjlm MDAwMC0weGZiOWYzZmZmIGlycSAxNyBhdCBkZXZpY2UgMC4xIG9uIHBjaTIKaWdiMTogVXNpbmcg TVNJWCBpbnRlcnJ1cHRzIHdpdGggOSB2ZWN0b3JzCmlnYjE6IEV0aGVybmV0IGFkZHJlc3M6IGZj OjE1OmI0OjI2OjZlOjkxCmlnYjE6IEJvdW5kIHF1ZXVlIDAgdG8gY3B1IDgKaWdiMTogQm91bmQg cXVldWUgMSB0byBjcHUgOQppZ2IxOiBCb3VuZCBxdWV1ZSAyIHRvIGNwdSAxMAppZ2IxOiBCb3Vu ZCBxdWV1ZSAzIHRvIGNwdSAxMQppZ2IxOiBCb3VuZCBxdWV1ZSA0IHRvIGNwdSAwCmlnYjE6IEJv dW5kIHF1ZXVlIDUgdG8gY3B1IDEKaWdiMTogQm91bmQgcXVldWUgNiB0byBjcHUgMgppZ2IxOiBC b3VuZCBxdWV1ZSA3IHRvIGNwdSAzCmlnYjI6IDxJbnRlbChSKSBQUk8vMTAwMCBOZXR3b3JrIENv bm5lY3Rpb24gdmVyc2lvbiAtIDIuNC4wPiBwb3J0IDB4NTA0MC0weDUwNWYgbWVtIDB4ZmI4MDAw MDAtMHhmYjhmZmZmZiwweGZiN2YwMDAwLTB4ZmI3ZjNmZmYgaXJxIDE4IGF0IGRldmljZSAwLjIg b24gcGNpMgppZ2IyOiBVc2luZyBNU0lYIGludGVycnVwdHMgd2l0aCA5IHZlY3RvcnMKaWdiMjog RXRoZXJuZXQgYWRkcmVzczogZmM6MTU6YjQ6MjY6NmU6OTIKaWdiMjogQm91bmQgcXVldWUgMCB0 byBjcHUgNAppZ2IyOiBCb3VuZCBxdWV1ZSAxIHRvIGNwdSA1CmlnYjI6IEJvdW5kIHF1ZXVlIDIg dG8gY3B1IDYKaWdiMjogQm91bmQgcXVldWUgMyB0byBjcHUgNwppZ2IyOiBCb3VuZCBxdWV1ZSA0 IHRvIGNwdSA4CmlnYjI6IEJvdW5kIHF1ZXVlIDUgdG8gY3B1IDkKaWdiMjogQm91bmQgcXVldWUg NiB0byBjcHUgMTAKaWdiMjogQm91bmQgcXVldWUgNyB0byBjcHUgMTEKaWdiMzogPEludGVsKFIp IFBSTy8xMDAwIE5ldHdvcmsgQ29ubmVjdGlvbiB2ZXJzaW9uIC0gMi40LjA+IHBvcnQgMHg1MDYw LTB4NTA3ZiBtZW0gMHhmYjYwMDAwMC0weGZiNmZmZmZmLDB4ZmI1ZjAwMDAtMHhmYjVmM2ZmZiBp cnEgMTkgYXQgZGV2aWNlIDAuMyBvbiBwY2kyCmlnYjM6IFVzaW5nIE1TSVggaW50ZXJydXB0cyB3 aXRoIDkgdmVjdG9ycwppZ2IzOiBFdGhlcm5ldCBhZGRyZXNzOiBmYzoxNTpiNDoyNjo2ZTo5Mwpp Z2IzOiBCb3VuZCBxdWV1ZSAwIHRvIGNwdSAwCmlnYjM6IEJvdW5kIHF1ZXVlIDEgdG8gY3B1IDEK aWdiMzogQm91bmQgcXVldWUgMiB0byBjcHUgMgppZ2IzOiBCb3VuZCBxdWV1ZSAzIHRvIGNwdSAz CmlnYjM6IEJvdW5kIHF1ZXVlIDQgdG8gY3B1IDQKaWdiMzogQm91bmQgcXVldWUgNSB0byBjcHUg NQppZ2IzOiBCb3VuZCBxdWV1ZSA2IHRvIGNwdSA2CmlnYjM6IEJvdW5kIHF1ZXVlIDcgdG8gY3B1 IDcKcGNpYjEwOiA8QUNQSSBQQ0ktUENJIGJyaWRnZT4gYXQgZGV2aWNlIDI4Ljcgb24gcGNpMApw Y2kxOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liMTAKcGNpMTogPGJhc2UgcGVyaXBoZXJhbD4gYXQg ZGV2aWNlIDAuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQp2Z2FwY2kwOiA8VkdBLWNvbXBhdGlibGUg ZGlzcGxheT4gbWVtIDB4YmYwMDAwMDAtMHhiZmZmZmZmZiwweGZiNGUwMDAwLTB4ZmI0ZTNmZmYs MHhmYTgwMDAwMC0weGZhZmZmZmZmIGlycSAxNiBhdCBkZXZpY2UgMC4xIG9uIHBjaTEKdmdhcGNp MDogQm9vdCB2aWRlbyBkZXZpY2UKcGNpMTogPGJhc2UgcGVyaXBoZXJhbD4gYXQgZGV2aWNlIDAu MiAobm8gZHJpdmVyIGF0dGFjaGVkKQp1aGNpMDogPFVIQ0kgKGdlbmVyaWMpIFVTQiBjb250cm9s bGVyPiBwb3J0IDB4M2MwMC0weDNjMWYgaXJxIDE2IGF0IGRldmljZSAwLjQgb24gcGNpMQp1c2J1 czEgb24gdWhjaTAKZWhjaTE6IDxFSENJIChnZW5lcmljKSBVU0IgMi4wIGNvbnRyb2xsZXI+IG1l bSAweGZhNDUwMDAwLTB4ZmE0NTAzZmYgaXJxIDIwIGF0IGRldmljZSAyOS4wIG9uIHBjaTAKdXNi dXMyOiBFSENJIHZlcnNpb24gMS4wCnVzYnVzMiBvbiBlaGNpMQpwY2liMTE6IDxQQ0ktUENJIGJy aWRnZT4gYXQgZGV2aWNlIDMwLjAgb24gcGNpMApwY2kyNzogPFBDSSBidXM+IG9uIHBjaWIxMQpp c2FiMDogPFBDSS1JU0EgYnJpZGdlPiBhdCBkZXZpY2UgMzEuMCBvbiBwY2kwCmlzYTA6IDxJU0Eg YnVzPiBvbiBpc2FiMAphaGNpMDogPEludGVsIFBhdHNidXJnIEFIQ0kgU0FUQSBjb250cm9sbGVy PiBwb3J0IDB4NDAwMC0weDQwMDcsMHg0MDA4LTB4NDAwYiwweDQwMTAtMHg0MDE3LDB4NDAxOC0w eDQwMWIsMHg0MDIwLTB4NDAzZiBtZW0gMHhmYTQ0MDAwMC0weGZhNDQwN2ZmIGlycSAxNyBhdCBk ZXZpY2UgMzEuMiBvbiBwY2kwCmFoY2kwOiBBSENJIHYxLjMwIHdpdGggNiA2R2JwcyBwb3J0cywg UG9ydCBNdWx0aXBsaWVyIHN1cHBvcnRlZAphaGNpY2gwOiA8QUhDSSBjaGFubmVsPiBhdCBjaGFu bmVsIDAgb24gYWhjaTAKYWhjaWNoMTogPEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAxIG9uIGFo Y2kwCmFoY2ljaDI6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgMiBvbiBhaGNpMAphaGNpY2gz OiA8QUhDSSBjaGFubmVsPiBhdCBjaGFubmVsIDMgb24gYWhjaTAKYWhjaWNoNDogPEFIQ0kgY2hh bm5lbD4gYXQgY2hhbm5lbCA0IG9uIGFoY2kwCmFoY2ljaDU6IDxBSENJIGNoYW5uZWw+IGF0IGNo YW5uZWwgNSBvbiBhaGNpMAphaGNpZW0wOiA8QUhDSSBlbmNsb3N1cmUgbWFuYWdlbWVudCBicmlk Z2U+IG9uIGFoY2kwCmFjcGlfdHowOiA8VGhlcm1hbCBab25lPiBvbiBhY3BpMAphdGtiZGMwOiA8 S2V5Ym9hcmQgY29udHJvbGxlciAoaTgwNDIpPiBwb3J0IDB4NjAsMHg2NCBpcnEgMSBvbiBhY3Bp MAphdGtiZDA6IDxBVCBLZXlib2FyZD4gaXJxIDEgb24gYXRrYmRjMAprYmQwIGF0IGF0a2JkMAph dGtiZDA6IFtHSUFOVC1MT0NLRURdCnVhcnQwOiA8Tm9uLXN0YW5kYXJkIG5zODI1MCBjbGFzcyBV QVJUIHdpdGggRklGT3M+IHBvcnQgMHgzZjgtMHgzZmYgaXJxIDQgZmxhZ3MgMHgxMCBvbiBhY3Bp MApvcm0wOiA8SVNBIE9wdGlvbiBST00+IGF0IGlvbWVtIDB4YzAwMDAtMHhjN2ZmZiBvbiBpc2Ew CnNjMDogPFN5c3RlbSBjb25zb2xlPiBhdCBmbGFncyAweDEwMCBvbiBpc2EwCnNjMDogVkdBIDwx NiB2aXJ0dWFsIGNvbnNvbGVzLCBmbGFncz0weDMwMD4KdmdhMDogPEdlbmVyaWMgSVNBIFZHQT4g YXQgcG9ydCAweDNjMC0weDNkZiBpb21lbSAweGEwMDAwLTB4YmZmZmYgb24gaXNhMApwcGMwOiBj YW5ub3QgcmVzZXJ2ZSBJL08gcG9ydCByYW5nZQp1YXJ0MTogPE5vbi1zdGFuZGFyZCBuczgyNTAg Y2xhc3MgVUFSVCB3aXRoIEZJRk9zPiBhdCBwb3J0IDB4MmY4LTB4MmZmIGlycSAzIG9uIGlzYTAK ZXN0MDogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1MAplc3Q6 IENQVSBzdXBwb3J0cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1dCBpcyBub3QgcmVjb2duaXplZC4K ZXN0OiBjcHVfdmVuZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFkYzgwMDAwMTkwMApkZXZpY2VfYXR0 YWNoOiBlc3QwIGF0dGFjaCByZXR1cm5lZCA2CnA0dGNjMDogPENQVSBGcmVxdWVuY3kgVGhlcm1h bCBDb250cm9sPiBvbiBjcHUwCmVzdDE6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENv bnRyb2w+IG9uIGNwdTEKZXN0OiBDUFUgc3VwcG9ydHMgRW5oYW5jZWQgU3BlZWRzdGVwLCBidXQg aXMgbm90IHJlY29nbml6ZWQuCmVzdDogY3B1X3ZlbmRvciBHZW51aW5lSW50ZWwsIG1zciAxZGM4 MDAwMDE5MDAKZGV2aWNlX2F0dGFjaDogZXN0MSBhdHRhY2ggcmV0dXJuZWQgNgpwNHRjYzE6IDxD UFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1MQplc3QyOiA8RW5oYW5jZWQgU3Bl ZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHUyCmVzdDogQ1BVIHN1cHBvcnRzIEVuaGFu Y2VkIFNwZWVkc3RlcCwgYnV0IGlzIG5vdCByZWNvZ25pemVkLgplc3Q6IGNwdV92ZW5kb3IgR2Vu dWluZUludGVsLCBtc3IgMWRjODAwMDAxOTAwCmRldmljZV9hdHRhY2g6IGVzdDIgYXR0YWNoIHJl dHVybmVkIDYKcDR0Y2MyOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTIK ZXN0MzogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1Mwplc3Q6 IENQVSBzdXBwb3J0cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1dCBpcyBub3QgcmVjb2duaXplZC4K ZXN0OiBjcHVfdmVuZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFkYzgwMDAwMTkwMApkZXZpY2VfYXR0 YWNoOiBlc3QzIGF0dGFjaCByZXR1cm5lZCA2CnA0dGNjMzogPENQVSBGcmVxdWVuY3kgVGhlcm1h bCBDb250cm9sPiBvbiBjcHUzCmVzdDQ6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENv bnRyb2w+IG9uIGNwdTQKZXN0OiBDUFUgc3VwcG9ydHMgRW5oYW5jZWQgU3BlZWRzdGVwLCBidXQg aXMgbm90IHJlY29nbml6ZWQuCmVzdDogY3B1X3ZlbmRvciBHZW51aW5lSW50ZWwsIG1zciAxZGM4 MDAwMDE5MDAKZGV2aWNlX2F0dGFjaDogZXN0NCBhdHRhY2ggcmV0dXJuZWQgNgpwNHRjYzQ6IDxD UFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1NAplc3Q1OiA8RW5oYW5jZWQgU3Bl ZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHU1CmVzdDogQ1BVIHN1cHBvcnRzIEVuaGFu Y2VkIFNwZWVkc3RlcCwgYnV0IGlzIG5vdCByZWNvZ25pemVkLgplc3Q6IGNwdV92ZW5kb3IgR2Vu dWluZUludGVsLCBtc3IgMWRjODAwMDAxOTAwCmRldmljZV9hdHRhY2g6IGVzdDUgYXR0YWNoIHJl dHVybmVkIDYKcDR0Y2M1OiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTUK ZXN0NjogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1Ngplc3Q6 IENQVSBzdXBwb3J0cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1dCBpcyBub3QgcmVjb2duaXplZC4K ZXN0OiBjcHVfdmVuZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFkYzgwMDAwMTkwMApkZXZpY2VfYXR0 YWNoOiBlc3Q2IGF0dGFjaCByZXR1cm5lZCA2CnA0dGNjNjogPENQVSBGcmVxdWVuY3kgVGhlcm1h bCBDb250cm9sPiBvbiBjcHU2CmVzdDc6IDxFbmhhbmNlZCBTcGVlZFN0ZXAgRnJlcXVlbmN5IENv bnRyb2w+IG9uIGNwdTcKZXN0OiBDUFUgc3VwcG9ydHMgRW5oYW5jZWQgU3BlZWRzdGVwLCBidXQg aXMgbm90IHJlY29nbml6ZWQuCmVzdDogY3B1X3ZlbmRvciBHZW51aW5lSW50ZWwsIG1zciAxZGM4 MDAwMDE5MDAKZGV2aWNlX2F0dGFjaDogZXN0NyBhdHRhY2ggcmV0dXJuZWQgNgpwNHRjYzc6IDxD UFUgRnJlcXVlbmN5IFRoZXJtYWwgQ29udHJvbD4gb24gY3B1Nwplc3Q4OiA8RW5oYW5jZWQgU3Bl ZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHU4CmVzdDogQ1BVIHN1cHBvcnRzIEVuaGFu Y2VkIFNwZWVkc3RlcCwgYnV0IGlzIG5vdCByZWNvZ25pemVkLgplc3Q6IGNwdV92ZW5kb3IgR2Vu dWluZUludGVsLCBtc3IgMWRjODAwMDAxOTAwCmRldmljZV9hdHRhY2g6IGVzdDggYXR0YWNoIHJl dHVybmVkIDYKcDR0Y2M4OiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTgK ZXN0OTogPEVuaGFuY2VkIFNwZWVkU3RlcCBGcmVxdWVuY3kgQ29udHJvbD4gb24gY3B1OQplc3Q6 IENQVSBzdXBwb3J0cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1dCBpcyBub3QgcmVjb2duaXplZC4K ZXN0OiBjcHVfdmVuZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFkYzgwMDAwMTkwMApkZXZpY2VfYXR0 YWNoOiBlc3Q5IGF0dGFjaCByZXR1cm5lZCA2CnA0dGNjOTogPENQVSBGcmVxdWVuY3kgVGhlcm1h bCBDb250cm9sPiBvbiBjcHU5CmVzdDEwOiA8RW5oYW5jZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBD b250cm9sPiBvbiBjcHUxMAplc3Q6IENQVSBzdXBwb3J0cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1 dCBpcyBub3QgcmVjb2duaXplZC4KZXN0OiBjcHVfdmVuZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFk YzgwMDAwMTkwMApkZXZpY2VfYXR0YWNoOiBlc3QxMCBhdHRhY2ggcmV0dXJuZWQgNgpwNHRjYzEw OiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+IG9uIGNwdTEwCmVzdDExOiA8RW5oYW5j ZWQgU3BlZWRTdGVwIEZyZXF1ZW5jeSBDb250cm9sPiBvbiBjcHUxMQplc3Q6IENQVSBzdXBwb3J0 cyBFbmhhbmNlZCBTcGVlZHN0ZXAsIGJ1dCBpcyBub3QgcmVjb2duaXplZC4KZXN0OiBjcHVfdmVu ZG9yIEdlbnVpbmVJbnRlbCwgbXNyIDFkYzgwMDAwMTkwMApkZXZpY2VfYXR0YWNoOiBlc3QxMSBh dHRhY2ggcmV0dXJuZWQgNgpwNHRjYzExOiA8Q1BVIEZyZXF1ZW5jeSBUaGVybWFsIENvbnRyb2w+ IG9uIGNwdTExClpGUyBmaWxlc3lzdGVtIHZlcnNpb246IDUKWkZTIHN0b3JhZ2UgcG9vbCB2ZXJz aW9uOiBmZWF0dXJlcyBzdXBwb3J0ICg1MDAwKQpUaW1lY291bnRlcnMgdGljayBldmVyeSAxLjAw MCBtc2VjCm1maTA6IDY4MCAoNDQ1ODUxMzk2cy8weDAwMjAvaW5mbykgLSBTaHV0ZG93biBjb21t YW5kIHJlY2VpdmVkIGZyb20gaG9zdAptZmkwOiA2ODEgKGJvb3QgKyAzcy8weDAwMjAvaW5mbykg LSBGaXJtd2FyZSBpbml0aWFsaXphdGlvbiBzdGFydGVkIChQQ0kgSUQgMDA3My8xMDAwLzkyNDAv MTAwMCkKbWZpMDogNjgyIChib290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgdmVyc2lv biAyLjEzMC4zOTQtMjU1MAptZmkwOiA2ODMgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBQYWNr YWdlIHZlcnNpb24gMjAuMTIuMS0wMTUwCm1maTA6IDY4NCAoYm9vdCArIDVzLzB4MDAyMC9pbmZv KSAtIEJvYXJkIFJldmlzaW9uIDA0QwptZmkwOiA2ODUgKGJvb3QgKyAyNHMvMHgwMDA0L2luZm8p IC0gRW5jbG9zdXJlIChTRVMpIGRpc2NvdmVyZWQgb24gUEQgMTQoYyBQb3J0IDAgLSAzL3AxKQpt ZmkwOiA2ODYgKGJvb3QgKyAyNHMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIFBEIDE0KGMgUG9y dCAwIC0gMy9wMSkgY29tbXVuaWNhdGlvbiByZXN0b3JlZAptZmkwOiA2ODcgKGJvb3QgKyAyNHMv MHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IEVuY2wgUEQgMTQKbWZpMDogNjg4IChib290ICsgMjRz LzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9ZCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTc5 LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNjg5IChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwOChlMHgxNC9zMSkKbWZpMDogNjkwIChib290ICsgMjRzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiA2OTEgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0 L3MyKQptZmkwOiA2OTIgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5 KGUweDE0L3MyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2MiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDY5MyAoYm9vdCArIDI0 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpCm1maTA6IDY5NCAoYm9v dCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYz LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNjk1IChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwYihlMHgxNC9zNCkKbWZpMDogNjk2IChib290ICsgMjRzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjQsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiA2OTcgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0 L3M1KQptZmkwOiA2OTggKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBj KGUweDE0L3M1KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2NSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDY5OSAoYm9vdCArIDI0 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpCm1maTA6IDcwMCAoYm9v dCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY2 LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzAxIChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwZShlMHgxNC9zNykKbWZpMDogNzAyIChib290ICsgMjRzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjcsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiA3MDMgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0 L3M4KQptZmkwOiA3MDQgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBm KGUweDE0L3M4KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2OCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDcwNSAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpCm1maTA6IDcwNiAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY5 LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzA3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAxMShlMHgxNC9zMTApCm1maTA6IDcwOCAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMTEoZTB4MTQvczEwKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YSwwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDcwOSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4 MTQvczExKQptZmkwOiA3MTAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDEyKGUweDE0L3MxMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBz YXNBZGRyPTUwMDE0MzgwMzA1MGY5NmIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3MTEgKGJvb3Qg KyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikKbWZpMDogNzEy IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTZjLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzEzICg0NDU4NTE1NDlzLzB4MDAyMC9pbmZv KSAtIFRpbWUgZXN0YWJsaXNoZWQgYXMgMDIvMTYvMTQgIDc6Mzk6MDk7ICgxMjEgc2Vjb25kcyBz aW5jZSBwb3dlciBvbikKbWZpMDogNzE0ICg0NDYxNzI2NDdzLzB4MDAwMi9XQVJOKSAtIFJlbW92 ZWQ6IFBEIDA4KGUweDE0L3MxKQptZmkwOiA3MTUgKDQ0NjE3MjY0N3MvMHgwMDAyL2luZm8pIC0g UmVtb3ZlZDogUEQgMDgoZTB4MTQvczEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9y dE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDog NzE2ICg0NDYxNzI2NDdzLzB4MDAwMi9pbmZvKSAtIFN0YXRlIGNoYW5nZSBvbiBQRCAwOChlMHgx NC9zMSkgZnJvbSBKQk9EKDQwKSB0byBVTkNPTkZJR1VSRURfQkFEKDEpCm1maTA6IDcxNyAoNDQ2 MTcyNjkxcy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpCm1maTA6IDcx OCAoNDQ2MTcyNjkxcy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzE5IChib290ICsgM3MvMHgwMDIwL2luZm8p IC0gRmlybXdhcmUgaW5pdGlhbGl6YXRpb24gc3RhcnRlZCAoUENJIElEIDAwNzMvMTAwMC85MjQw LzEwMDApCm1maTA6IDcyMCAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIHZlcnNp b24gMi4xMzAuMzk0LTI1NTAKbWZpMDogNzIxIChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gUGFj a2FnZSB2ZXJzaW9uIDIwLjEyLjEtMDE1MAptZmkwOiA3MjIgKGJvb3QgKyA1cy8weDAwMjAvaW5m bykgLSBCb2FyZCBSZXZpc2lvbiAwNEMKbWZpMDogNzIzIChib290ICsgMjRzLzB4MDAwNC9pbmZv KSAtIEVuY2xvc3VyZSAoU0VTKSBkaXNjb3ZlcmVkIG9uIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkK bWZpMDogNzI0IChib290ICsgMjVzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSBQRCAxNChjIFBv cnQgMCAtIDMvcDEpIGNvbW11bmljYXRpb24gcmVzdG9yZWQKbWZpMDogNzI1IChib290ICsgMjVz LzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBFbmNsIFBEIDE0Cm1maTA6IDcyNiAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPWQsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3 OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDcyNyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMDgoZTB4MTQvczEpCm1maTA6IDcyOCAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogNzI5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgx NC9zMikKbWZpMDogNzMwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw OShlMHgxNC9zMikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3MzEgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKQptZmkwOiA3MzIgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 MywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDczMyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGIoZTB4MTQvczQpCm1maTA6IDczNCAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMGIoZTB4MTQvczQpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY0LDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogNzM1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgx NC9zNSkKbWZpMDogNzM2IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw YyhlMHgxNC9zNSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjUsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3MzcgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KQptZmkwOiA3MzggKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 NiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDczOSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGUoZTB4MTQvczcpCm1maTA6IDc0MCAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMGUoZTB4MTQvczcpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY3LDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogNzQxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgx NC9zOCkKbWZpMDogNzQyIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw ZihlMHgxNC9zOCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjgsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3NDMgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KQptZmkwOiA3NDQgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDc0NSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMTEoZTB4MTQvczEwKQptZmkwOiA3NDYgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlw ZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmEsMDAwMDAwMDAwMDAwMDAw MAptZmkwOiA3NDcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUw eDE0L3MxMSkKbWZpMDogNzQ4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQ RCAxMihlMHgxNC9zMTEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwg c2FzQWRkcj01MDAxNDM4MDMwNTBmOTZiLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzQ5IChib290 ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpCm1maTA6IDc1 MCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKSBJ bmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAz MDUwZjk2YywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDc1MSAoNDY1NjUwOTQ1cy8weDAwMjAvaW5m bykgLSBUaW1lIGVzdGFibGlzaGVkIGFzIDEwLzAzLzE0IDExOjI5OjA1OyAoMTIwIHNlY29uZHMg c2luY2UgcG93ZXIgb24pCm1maTA6IDc1MiAoNDY1NjUwOTQ2cy8weDAwMjAvV0FSTikgLSBQYXRy b2wgUmVhZCBjYW4ndCBiZSBzdGFydGVkLCBhcyBQRHMgYXJlIGVpdGhlciBub3QgT05MSU5FLCBv ciBhcmUgaW4gYSBWRCB3aXRoIGFuIGFjdGl2ZSBwcm9jZXNzLCBvciBhcmUgaW4gYW4gZXhjbHVk ZWQgVkQKbWZpMDogNzUzIChib290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgaW5pdGlh bGl6YXRpb24gc3RhcnRlZCAoUENJIElEIDAwNzMvMTAwMC85MjQwLzEwMDApCm1maTA6IDc1NCAo Ym9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIHZlcnNpb24gMi4xMzAuMzk0LTI1NTAK bWZpMDogNzU1IChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gUGFja2FnZSB2ZXJzaW9uIDIwLjEy LjEtMDE1MAptZmkwOiA3NTYgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBCb2FyZCBSZXZpc2lv biAwNEMKbWZpMDogNzU3IChib290ICsgMjRzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSAoU0VT KSBkaXNjb3ZlcmVkIG9uIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkKbWZpMDogNzU4IChib290ICsg MjRzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIGNvbW11 bmljYXRpb24gcmVzdG9yZWQKbWZpMDogNzU5IChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBFbmNsIFBEIDE0Cm1maTA6IDc2MCAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PWQsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3OSwwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDc2MSAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4 MTQvczEpCm1maTA6IDc2MiAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MDgoZTB4MTQvczEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2Fz QWRkcj01MDAxNDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzYzIChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikKbWZpMDogNzY0IChi b290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikgSW5mbzog ZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5 NjIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3NjUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0g SW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKQptZmkwOiA3NjYgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MywwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDc2NyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGIoZTB4 MTQvczQpCm1maTA6IDc2OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MGIoZTB4MTQvczQpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2Fz QWRkcj01MDAxNDM4MDMwNTBmOTY0LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzY5IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgxNC9zNSkKbWZpMDogNzcwIChi b290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgxNC9zNSkgSW5mbzog ZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5 NjUsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3NzEgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0g SW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KQptZmkwOiA3NzIgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NiwwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDc3MyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGUoZTB4 MTQvczcpCm1maTA6IDc3NCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MGUoZTB4MTQvczcpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2Fz QWRkcj01MDAxNDM4MDMwNTBmOTY3LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzc1IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgxNC9zOCkKbWZpMDogNzc2IChi b290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgxNC9zOCkgSW5mbzog ZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5 NjgsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3NzcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0g SW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KQptZmkwOiA3NzggKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OSwwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDc3OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTEoZTB4 MTQvczEwKQptZmkwOiA3ODAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDExKGUweDE0L3MxMCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBz YXNBZGRyPTUwMDE0MzgwMzA1MGY5NmEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3ODEgKGJvb3Qg KyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0L3MxMSkKbWZpMDogNzgy IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMihlMHgxNC9zMTEpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTZiLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogNzgzIChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpCm1maTA6IDc4NCAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKSBJbmZvOiBlbmNsUGQ9MTQsIHNj c2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YywwMDAwMDAwMDAw MDAwMDAwCm1maTA6IDc4NSAoNDY1NjUxMzk1cy8weDAwMjAvaW5mbykgLSBUaW1lIGVzdGFibGlz aGVkIGFzIDEwLzAzLzE0IDExOjM2OjM1OyAoMTI0IHNlY29uZHMgc2luY2UgcG93ZXIgb24pCm1m aTA6IDc4NiAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIGluaXRpYWxpemF0aW9u IHN0YXJ0ZWQgKFBDSSBJRCAwMDczLzEwMDAvOTI0MC8xMDAwKQptZmkwOiA3ODcgKGJvb3QgKyAz cy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSB2ZXJzaW9uIDIuMTMwLjM5NC0yNTUwCm1maTA6IDc4 OCAoYm9vdCArIDZzLzB4MDAyMC9pbmZvKSAtIFBhY2thZ2UgdmVyc2lvbiAyMC4xMi4xLTAxNTAK bWZpMDogNzg5IChib290ICsgNnMvMHgwMDIwL2luZm8pIC0gQm9hcmQgUmV2aXNpb24gMDRDCm1m aTA6IDc5MCAoYm9vdCArIDI0cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgKFNFUykgZGlzY292 ZXJlZCBvbiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpCm1maTA6IDc5MSAoYm9vdCArIDI1cy8weDAw MDQvaW5mbykgLSBFbmNsb3N1cmUgUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBjb21tdW5pY2F0aW9u IHJlc3RvcmVkCm1maTA6IDc5MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDog RW5jbCBQRCAxNAptZmkwOiA3OTMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT1kLCBwb3J0 TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NzksMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA3 OTQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKQpt ZmkwOiA3OTUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0 L3MxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2MSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDc5NiAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4MTQvczIpCm1maTA6IDc5NyAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4MTQvczIpIEluZm86IGVuY2xQZD0x NCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYyLDAwMDAw MDAwMDAwMDAwMDAKbWZpMDogNzk4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVk OiBQRCAwYShlMHgxNC9zMykKbWZpMDogNzk5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwYShlMHgxNC9zMykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0 TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjMsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4 MDAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KQpt ZmkwOiA4MDEgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0 L3M0KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2NCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDgwMiAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpCm1maTA6IDgwMyAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpIEluZm86IGVuY2xQZD0x NCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY1LDAwMDAw MDAwMDAwMDAwMDAKbWZpMDogODA0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVk OiBQRCAwZChlMHgxNC9zNikKbWZpMDogODA1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwZChlMHgxNC9zNikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0 TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjYsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4 MDYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KQpt ZmkwOiA4MDcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0 L3M3KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2NywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDgwOCAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpCm1maTA6IDgwOSAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpIEluZm86IGVuY2xQZD0x NCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY4LDAwMDAw MDAwMDAwMDAwMDAKbWZpMDogODEwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVk OiBQRCAxMChlMHgxNC9zOSkKbWZpMDogODExIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAxMChlMHgxNC9zOSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0 TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjksMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4 MTIgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkK bWZpMDogODEzIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMShlMHgx NC9zMTApIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01 MDAxNDM4MDMwNTBmOTZhLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODE0IChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMihlMHgxNC9zMTEpCm1maTA6IDgxNSAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Yiww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDgxNiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMTMoZTB4MTQvczEyKQptZmkwOiA4MTcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmMsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiA4MTggKDQ2NTY3MDE2N3MvMHgwMDIwL2luZm8pIC0gVGltZSBlc3RhYmxpc2hlZCBhcyAx MC8wMy8xNCAxNjo0OToyNzsgKDEzNCBzZWNvbmRzIHNpbmNlIHBvd2VyIG9uKQptZmkwOiA4MTkg KDQ2NTcwNjgwMHMvMHgwMDIwL1dBUk4pIC0gUGF0cm9sIFJlYWQgY2FuJ3QgYmUgc3RhcnRlZCwg YXMgUERzIGFyZSBlaXRoZXIgbm90IE9OTElORSwgb3IgYXJlIGluIGEgVkQgd2l0aCBhbiBhY3Rp dmUgcHJvY2Vzcywgb3IgYXJlIGluIGFuIGV4Y2x1ZGVkIFZECm1maTA6IDgyMCAoYm9vdCArIDNz LzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIGluaXRpYWxpemF0aW9uIHN0YXJ0ZWQgKFBDSSBJRCAw MDczLzEwMDAvOTI0MC8xMDAwKQptZmkwOiA4MjEgKGJvb3QgKyAzcy8weDAwMjAvaW5mbykgLSBG aXJtd2FyZSB2ZXJzaW9uIDIuMTMwLjM5NC0yNTUwCm1maTA6IDgyMiAoYm9vdCArIDVzLzB4MDAy MC9pbmZvKSAtIFBhY2thZ2UgdmVyc2lvbiAyMC4xMi4xLTAxNTAKbWZpMDogODIzIChib290ICsg NXMvMHgwMDIwL2luZm8pIC0gQm9hcmQgUmV2aXNpb24gMDRDCm1maTA6IDgyNCAoYm9vdCArIDI0 cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgKFNFUykgZGlzY292ZXJlZCBvbiBQRCAxNChjIFBv cnQgMCAtIDMvcDEpCm1maTA6IDgyNSAoYm9vdCArIDI1cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1 cmUgUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBjb21tdW5pY2F0aW9uIHJlc3RvcmVkCm1maTA6IDgy NiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogRW5jbCBQRCAxNAptZmkwOiA4 MjcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDE0KGMgUG9ydCAwIC0g My9wMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT1kLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NzksMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4MjggKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKQptZmkwOiA4MjkgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MSwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDgzMCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRl ZDogUEQgMDkoZTB4MTQvczIpCm1maTA6IDgzMSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMDkoZTB4MTQvczIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9y dE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYyLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDog ODMyIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYShlMHgxNC9zMykK bWZpMDogODMzIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYShlMHgx NC9zMykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjMsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4MzQgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KQptZmkwOiA4MzUgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NCwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDgzNiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRl ZDogUEQgMGMoZTB4MTQvczUpCm1maTA6IDgzNyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGMoZTB4MTQvczUpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9y dE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY1LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDog ODM4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZChlMHgxNC9zNikK bWZpMDogODM5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZChlMHgx NC9zNikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjYsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4NDAgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KQptZmkwOiA4NDEgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NywwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDg0MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRl ZDogUEQgMGYoZTB4MTQvczgpCm1maTA6IDg0MyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGYoZTB4MTQvczgpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9y dE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY4LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDog ODQ0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkK bWZpMDogODQ1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMChlMHgx NC9zOSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjksMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4NDYgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkKbWZpMDogODQ3IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMShlMHgxNC9zMTApIEluZm86IGVuY2xQ ZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZhLDAw MDAwMDAwMDAwMDAwMDAKbWZpMDogODQ4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAxMihlMHgxNC9zMTEpCm1maTA6IDg0OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAs IHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YiwwMDAwMDAwMDAwMDAwMDAwCm1m aTA6IDg1MCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQv czEyKQptZmkwOiA4NTEgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEz KGUweDE0L3MxMikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NmMsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4NTIgKDQ2NTkyMzc5 M3MvMHgwMDIwL2luZm8pIC0gVGltZSBlc3RhYmxpc2hlZCBhcyAxMC8wNi8xNCAxNToxNjozMzsg KDEyMSBzZWNvbmRzIHNpbmNlIHBvd2VyIG9uKQptZmkwOiA4NTMgKGJvb3QgKyAzcy8weDAwMjAv aW5mbykgLSBGaXJtd2FyZSBpbml0aWFsaXphdGlvbiBzdGFydGVkIChQQ0kgSUQgMDA3My8xMDAw LzkyNDAvMTAwMCkKbWZpMDogODU0IChib290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUg dmVyc2lvbiAyLjEzMC4zOTQtMjU1MAptZmkwOiA4NTUgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykg LSBQYWNrYWdlIHZlcnNpb24gMjAuMTIuMS0wMTUwCm1maTA6IDg1NiAoYm9vdCArIDVzLzB4MDAy MC9pbmZvKSAtIEJvYXJkIFJldmlzaW9uIDA0QwptZmkwOiA4NTcgKGJvb3QgKyAyNHMvMHgwMDA0 L2luZm8pIC0gRW5jbG9zdXJlIChTRVMpIGRpc2NvdmVyZWQgb24gUEQgMTQoYyBQb3J0IDAgLSAz L3AxKQptZmkwOiA4NTggKGJvb3QgKyAyNHMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIFBEIDE0 KGMgUG9ydCAwIC0gMy9wMSkgY29tbXVuaWNhdGlvbiByZXN0b3JlZAptZmkwOiA4NTkgKGJvb3Qg KyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IEVuY2wgUEQgMTQKbWZpMDogODYwIChib290 ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9ZCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTc5LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODYxIChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkKbWZpMDogODYyIChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBzY3Np VHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAwMDAw MDAwMAptZmkwOiA4NjMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5 KGUweDE0L3MyKQptZmkwOiA4NjQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDA5KGUweDE0L3MyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDg2NSAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpCm1maTA6IDg2 NiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTYzLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODY3IChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkKbWZpMDogODY4IChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkgSW5mbzogZW5jbFBkPTE0LCBzY3Np VHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjQsMDAwMDAwMDAwMDAw MDAwMAptZmkwOiA4NjkgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBj KGUweDE0L3M1KQptZmkwOiA4NzAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDBjKGUweDE0L3M1KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDg3MSAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpCm1maTA6IDg3 MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTY2LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODczIChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykKbWZpMDogODc0IChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykgSW5mbzogZW5jbFBkPTE0LCBzY3Np VHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjcsMDAwMDAwMDAwMDAw MDAwMAptZmkwOiA4NzUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBm KGUweDE0L3M4KQptZmkwOiA4NzYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDBmKGUweDE0L3M4KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDg3NyAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpCm1maTA6IDg3 OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTY5LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODc5IChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAxMShlMHgxNC9zMTApCm1maTA6IDg4MCAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTEoZTB4MTQvczEwKSBJbmZvOiBlbmNsUGQ9MTQsIHNj c2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YSwwMDAwMDAwMDAw MDAwMDAwCm1maTA6IDg4MSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MTIoZTB4MTQvczExKQptZmkwOiA4ODIgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDEyKGUweDE0L3MxMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFw PTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4ODMg KGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikKbWZp MDogODg0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9z MTIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAx NDM4MDMwNTBmOTZjLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogODg1ICg0NjU5MjU1MjhzLzB4MDAy MC9pbmZvKSAtIFRpbWUgZXN0YWJsaXNoZWQgYXMgMTAvMDYvMTQgMTU6NDU6Mjg7ICgxMjAgc2Vj b25kcyBzaW5jZSBwb3dlciBvbikKbWZpMDogODg2IChib290ICsgM3MvMHgwMDIwL2luZm8pIC0g RmlybXdhcmUgaW5pdGlhbGl6YXRpb24gc3RhcnRlZCAoUENJIElEIDAwNzMvMTAwMC85MjQwLzEw MDApCm1maTA6IDg4NyAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIHZlcnNpb24g Mi4xMzAuMzk0LTI1NTAKbWZpMDogODg4IChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gUGFja2Fn ZSB2ZXJzaW9uIDIwLjEyLjEtMDE1MAptZmkwOiA4ODkgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykg LSBCb2FyZCBSZXZpc2lvbiAwNEMKbWZpMDogODkwIChib290ICsgMjRzLzB4MDAwNC9pbmZvKSAt IEVuY2xvc3VyZSAoU0VTKSBkaXNjb3ZlcmVkIG9uIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkKbWZp MDogODkxIChib290ICsgMjRzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSBQRCAxNChjIFBvcnQg MCAtIDMvcDEpIGNvbW11bmljYXRpb24gcmVzdG9yZWQKbWZpMDogODkyIChib290ICsgMjRzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBFbmNsIFBEIDE0Cm1maTA6IDg5MyAoYm9vdCArIDI0cy8w eDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPWQsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3OSww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDg5NCAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMDgoZTB4MTQvczEpCm1maTA6IDg5NSAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwg cG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAKbWZp MDogODk2IChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9z MikKbWZpMDogODk3IChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShl MHgxNC9zMikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRy PTUwMDE0MzgwMzA1MGY5NjIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA4OTggKGJvb3QgKyAyNHMv MHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKQptZmkwOiA4OTkgKGJvb3Qg KyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Myww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDkwMCAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMGIoZTB4MTQvczQpCm1maTA6IDkwMSAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMGIoZTB4MTQvczQpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwg cG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY0LDAwMDAwMDAwMDAwMDAwMDAKbWZp MDogOTAyIChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgxNC9z NSkKbWZpMDogOTAzIChib290ICsgMjRzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhl MHgxNC9zNSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRy PTUwMDE0MzgwMzA1MGY5NjUsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5MDQgKGJvb3QgKyAyNHMv MHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KQptZmkwOiA5MDUgKGJvb3Qg KyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Niww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDkwNiAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMGUoZTB4MTQvczcpCm1maTA6IDkwNyAoYm9vdCArIDI0cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMGUoZTB4MTQvczcpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwg cG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY3LDAwMDAwMDAwMDAwMDAwMDAKbWZp MDogOTA4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgxNC9z OCkKbWZpMDogOTA5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihl MHgxNC9zOCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRy PTUwMDE0MzgwMzA1MGY5NjgsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5MTAgKGJvb3QgKyAyNXMv MHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KQptZmkwOiA5MTEgKGJvb3Qg KyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OSww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDkxMiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMTEoZTB4MTQvczEwKQptZmkwOiA5MTMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmEsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiA5MTQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0 L3MxMSkKbWZpMDogOTE1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAx MihlMHgxNC9zMTEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2Fz QWRkcj01MDAxNDM4MDMwNTBmOTZiLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTE2IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpCm1maTA6IDkxNyAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKSBJbmZv OiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUw Zjk2YywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDkxOCAoNDY1OTI1OTk4cy8weDAwMjAvaW5mbykg LSBUaW1lIGVzdGFibGlzaGVkIGFzIDEwLzA2LzE0IDE1OjUzOjE4OyAoMTIzIHNlY29uZHMgc2lu Y2UgcG93ZXIgb24pCm1maTA6IDkxOSAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJl IGluaXRpYWxpemF0aW9uIHN0YXJ0ZWQgKFBDSSBJRCAwMDczLzEwMDAvOTI0MC8xMDAwKQptZmkw OiA5MjAgKGJvb3QgKyAzcy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSB2ZXJzaW9uIDIuMTMwLjM5 NC0yNTUwCm1maTA6IDkyMSAoYm9vdCArIDVzLzB4MDAyMC9pbmZvKSAtIFBhY2thZ2UgdmVyc2lv biAyMC4xMi4xLTAxNTAKbWZpMDogOTIyIChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gQm9hcmQg UmV2aXNpb24gMDRDCm1maTA6IDkyMyAoYm9vdCArIDI0cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1 cmUgKFNFUykgZGlzY292ZXJlZCBvbiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpCm1maTA6IDkyNCAo Ym9vdCArIDI0cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgUEQgMTQoYyBQb3J0IDAgLSAzL3Ax KSBjb21tdW5pY2F0aW9uIHJlc3RvcmVkCm1maTA6IDkyNSAoYm9vdCArIDI0cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogRW5jbCBQRCAxNAptZmkwOiA5MjYgKGJvb3QgKyAyNHMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT1kLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NzksMDAwMDAwMDAw MDAwMDAwMAptZmkwOiA5MjcgKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDA4KGUweDE0L3MxKQptZmkwOiA5MjggKGJvb3QgKyAyNHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDA4KGUweDE0L3MxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9 MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDkyOSAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4MTQvczIpCm1maTA6 IDkzMCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4MTQvczIp IEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4 MDMwNTBmOTYyLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTMxIChib290ICsgMjVzLzB4MDAwMi9p bmZvKSAtIEluc2VydGVkOiBQRCAwYShlMHgxNC9zMykKbWZpMDogOTMyIChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYShlMHgxNC9zMykgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjMsMDAwMDAwMDAw MDAwMDAwMAptZmkwOiA5MzMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDBiKGUweDE0L3M0KQptZmkwOiA5MzQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDBiKGUweDE0L3M0KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9 MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDkzNSAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpCm1maTA6 IDkzNiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUp IEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4 MDMwNTBmOTY1LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTM3IChib290ICsgMjVzLzB4MDAwMi9p bmZvKSAtIEluc2VydGVkOiBQRCAwZChlMHgxNC9zNikKbWZpMDogOTM4IChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZChlMHgxNC9zNikgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjYsMDAwMDAwMDAw MDAwMDAwMAptZmkwOiA5MzkgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDBlKGUweDE0L3M3KQptZmkwOiA5NDAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDBlKGUweDE0L3M3KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9 MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk0MSAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpCm1maTA6 IDk0MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgp IEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4 MDMwNTBmOTY4LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTQzIChib290ICsgMjVzLzB4MDAwMi9p bmZvKSAtIEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkKbWZpMDogOTQ0IChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjksMDAwMDAwMDAw MDAwMDAwMAptZmkwOiA5NDUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDExKGUweDE0L3MxMCkKbWZpMDogOTQ2IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAxMShlMHgxNC9zMTApIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1h cD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZhLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTQ3 IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMihlMHgxNC9zMTEpCm1m aTA6IDk0OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQv czExKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2YiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk0OSAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKQptZmkwOiA5NTAgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikgSW5mbzogZW5jbFBk PTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmMsMDAw MDAwMDAwMDAwMDAwMAptZmkwOiA5NTEgKDQ2NTkyNjI0N3MvMHgwMDIwL2luZm8pIC0gVGltZSBl c3RhYmxpc2hlZCBhcyAxMC8wNi8xNCAxNTo1NzoyNzsgKDEyMSBzZWNvbmRzIHNpbmNlIHBvd2Vy IG9uKQptZmkwOiA5NTIgKGJvb3QgKyAzcy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSBpbml0aWFs aXphdGlvbiBzdGFydGVkIChQQ0kgSUQgMDA3My8xMDAwLzkyNDAvMTAwMCkKbWZpMDogOTUzIChi b290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgdmVyc2lvbiAyLjEzMC4zOTQtMjU1MApt ZmkwOiA5NTQgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBQYWNrYWdlIHZlcnNpb24gMjAuMTIu MS0wMTUwCm1maTA6IDk1NSAoYm9vdCArIDVzLzB4MDAyMC9pbmZvKSAtIEJvYXJkIFJldmlzaW9u IDA0QwptZmkwOiA5NTYgKGJvb3QgKyAyNHMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIChTRVMp IGRpc2NvdmVyZWQgb24gUEQgMTQoYyBQb3J0IDAgLSAzL3AxKQptZmkwOiA5NTcgKGJvb3QgKyAy NXMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkgY29tbXVu aWNhdGlvbiByZXN0b3JlZAptZmkwOiA5NTggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5z ZXJ0ZWQ6IEVuY2wgUEQgMTQKbWZpMDogOTU5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 ZCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTc5LDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogOTYwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOChlMHgx NC9zMSkKbWZpMDogOTYxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw OChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5NjIgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0L3MyKQptZmkwOiA5NjMgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0L3MyKSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 MiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk2NCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGEoZTB4MTQvczMpCm1maTA6IDk2NSAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYzLDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogOTY2IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYihlMHgx NC9zNCkKbWZpMDogOTY3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw YihlMHgxNC9zNCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjQsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5NjggKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0L3M1KQptZmkwOiA5NjkgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0L3M1KSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 NSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk3MCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMGQoZTB4MTQvczYpCm1maTA6IDk3MSAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY2LDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogOTcyIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZShlMHgx NC9zNykKbWZpMDogOTczIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw ZShlMHgxNC9zNykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjcsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5NzQgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0L3M4KQptZmkwOiA5NzUgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0L3M4KSBJbmZvOiBl bmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2 OCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk3NiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMTAoZTB4MTQvczkpCm1maTA6IDk3NyAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9 MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY5LDAwMDAwMDAwMDAwMDAwMDAK bWZpMDogOTc4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMShlMHgx NC9zMTApCm1maTA6IDk3OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MTEoZTB4MTQvczEwKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNh c0FkZHI9NTAwMTQzODAzMDUwZjk2YSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk4MCAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKQptZmkwOiA5ODEg KGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0L3MxMSkgSW5m bzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1 MGY5NmIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiA5ODIgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikKbWZpMDogOTgzIChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpIEluZm86IGVuY2xQZD0xNCwgc2Nz aVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZjLDAwMDAwMDAwMDAw MDAwMDAKbWZpMDogOTg0ICg0NjU5MjY0NjdzLzB4MDAyMC9pbmZvKSAtIFRpbWUgZXN0YWJsaXNo ZWQgYXMgMTAvMDYvMTQgMTY6MDE6MDc7ICgxMjAgc2Vjb25kcyBzaW5jZSBwb3dlciBvbikKbWZp MDogOTg1IChib290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgaW5pdGlhbGl6YXRpb24g c3RhcnRlZCAoUENJIElEIDAwNzMvMTAwMC85MjQwLzEwMDApCm1maTA6IDk4NiAoYm9vdCArIDNz LzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIHZlcnNpb24gMi4xMzAuMzk0LTI1NTAKbWZpMDogOTg3 IChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gUGFja2FnZSB2ZXJzaW9uIDIwLjEyLjEtMDE1MApt ZmkwOiA5ODggKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBCb2FyZCBSZXZpc2lvbiAwNEMKbWZp MDogOTg5IChib290ICsgMjVzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSAoU0VTKSBkaXNjb3Zl cmVkIG9uIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkKbWZpMDogOTkwIChib290ICsgMjVzLzB4MDAw NC9pbmZvKSAtIEVuY2xvc3VyZSBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIGNvbW11bmljYXRpb24g cmVzdG9yZWQKbWZpMDogOTkxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBF bmNsIFBEIDE0Cm1maTA6IDk5MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDog UEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPWQsIHBvcnRN YXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk5 MyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpCm1m aTA6IDk5NCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQv czEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAx NDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogOTk1IChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikKbWZpMDogOTk2IChib290ICsgMjVz LzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikgSW5mbzogZW5jbFBkPTE0 LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjIsMDAwMDAw MDAwMDAwMDAwMAptZmkwOiA5OTcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDBhKGUweDE0L3MzKQptZmkwOiA5OTggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5z ZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRN YXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDk5 OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGIoZTB4MTQvczQpCm1m aTA6IDEwMDAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0 L3M0KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2NCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwMDEgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0L3M1KQptZmkwOiAxMDAyIChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgxNC9zNSkgSW5mbzogZW5jbFBk PTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjUsMDAw MDAwMDAwMDAwMDAwMAptZmkwOiAxMDAzIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAwZChlMHgxNC9zNikKbWZpMDogMTAwNCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwg cG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY2LDAwMDAwMDAwMDAwMDAwMDAKbWZp MDogMTAwNSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGUoZTB4MTQv czcpCm1maTA6IDEwMDYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBl KGUweDE0L3M3KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2NywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwMDcgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0L3M4KQptZmkwOiAxMDA4IChi b290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgxNC9zOCkgSW5mbzog ZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5 NjgsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDA5IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAt IEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkKbWZpMDogMTAxMCAoYm9vdCArIDI1cy8weDAwMDIv aW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5 cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY5LDAwMDAwMDAwMDAwMDAw MDAKbWZpMDogMTAxMSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTEo ZTB4MTQvczEwKQptZmkwOiAxMDEyIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVk OiBQRCAxMShlMHgxNC9zMTApIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0w MCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZhLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTAxMyAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKQptZmkw OiAxMDE0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMihlMHgxNC9z MTEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAx NDM4MDMwNTBmOTZiLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTAxNSAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKQptZmkwOiAxMDE2IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpIEluZm86IGVuY2xQ ZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZjLDAw MDAwMDAwMDAwMDAwMDAKbWZpMDogMTAxNyAoNDY1OTI2ODYwcy8weDAwMjAvaW5mbykgLSBUaW1l IGVzdGFibGlzaGVkIGFzIDEwLzA2LzE0IDE2OjA3OjQwOyAoMTIxIHNlY29uZHMgc2luY2UgcG93 ZXIgb24pCm1maTA6IDEwMTggKGJvb3QgKyAzcy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSBpbml0 aWFsaXphdGlvbiBzdGFydGVkIChQQ0kgSUQgMDA3My8xMDAwLzkyNDAvMTAwMCkKbWZpMDogMTAx OSAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIHZlcnNpb24gMi4xMzAuMzk0LTI1 NTAKbWZpMDogMTAyMCAoYm9vdCArIDVzLzB4MDAyMC9pbmZvKSAtIFBhY2thZ2UgdmVyc2lvbiAy MC4xMi4xLTAxNTAKbWZpMDogMTAyMSAoYm9vdCArIDVzLzB4MDAyMC9pbmZvKSAtIEJvYXJkIFJl dmlzaW9uIDA0QwptZmkwOiAxMDIyIChib290ICsgMjVzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3Vy ZSAoU0VTKSBkaXNjb3ZlcmVkIG9uIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkKbWZpMDogMTAyMyAo Ym9vdCArIDI1cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgUEQgMTQoYyBQb3J0IDAgLSAzL3Ax KSBjb21tdW5pY2F0aW9uIHJlc3RvcmVkCm1maTA6IDEwMjQgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IEVuY2wgUEQgMTQKbWZpMDogMTAyNSAoYm9vdCArIDI1cy8weDAwMDIv aW5mbykgLSBJbnNlcnRlZDogUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBlbmNsUGQ9MTQs IHNjc2lUeXBlPWQsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3OSwwMDAwMDAw MDAwMDAwMDAwCm1maTA6IDEwMjYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDA4KGUweDE0L3MxKQptZmkwOiAxMDI3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBQRCAwOChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0 TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAx MDI4IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikK bWZpMDogMTAyOSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4 MTQvczIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01 MDAxNDM4MDMwNTBmOTYyLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTAzMCAoYm9vdCArIDI1cy8w eDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpCm1maTA6IDEwMzEgKGJvb3Qg KyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Myww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwMzIgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5z ZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KQptZmkwOiAxMDMzIChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0w LCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjQsMDAwMDAwMDAwMDAwMDAwMApt ZmkwOiAxMDM0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgx NC9zNSkKbWZpMDogMTAzNSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MGMoZTB4MTQvczUpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2Fz QWRkcj01MDAxNDM4MDMwNTBmOTY1LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTAzNiAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpCm1maTA6IDEwMzcg KGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZv OiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUw Zjk2NiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwMzggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KQptZmkwOiAxMDM5IChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykgSW5mbzogZW5jbFBkPTE0LCBzY3Np VHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjcsMDAwMDAwMDAwMDAw MDAwMAptZmkwOiAxMDQwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw ZihlMHgxNC9zOCkKbWZpMDogMTA0MSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRl ZDogUEQgMGYoZTB4MTQvczgpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0w MCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY4LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTA0MiAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpCm1maTA6 IDEwNDMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5 KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQz ODAzMDUwZjk2OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwNDQgKGJvb3QgKyAyNXMvMHgwMDAy L2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkKbWZpMDogMTA0NSAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTEoZTB4MTQvczEwKSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YSwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDEwNDYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDEyKGUweDE0L3MxMSkKbWZpMDogMTA0NyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAs IHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YiwwMDAwMDAwMDAwMDAwMDAwCm1m aTA6IDEwNDggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0 L3MxMikKbWZpMDogMTA0OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MTMoZTB4MTQvczEyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNh c0FkZHI9NTAwMTQzODAzMDUwZjk2YywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwNTAgKDQ2NTk5 NzI0NnMvMHgwMDIwL2luZm8pIC0gVGltZSBlc3RhYmxpc2hlZCBhcyAxMC8wNy8xNCAxMTo0MDo0 NjsgKDEyMCBzZWNvbmRzIHNpbmNlIHBvd2VyIG9uKQptZmkwOiAxMDUxIChib290ICsgM3MvMHgw MDIwL2luZm8pIC0gRmlybXdhcmUgaW5pdGlhbGl6YXRpb24gc3RhcnRlZCAoUENJIElEIDAwNzMv MTAwMC85MjQwLzEwMDApCm1maTA6IDEwNTIgKGJvb3QgKyAzcy8weDAwMjAvaW5mbykgLSBGaXJt d2FyZSB2ZXJzaW9uIDIuMTMwLjM5NC0yNTUwCm1maTA6IDEwNTMgKGJvb3QgKyA1cy8weDAwMjAv aW5mbykgLSBQYWNrYWdlIHZlcnNpb24gMjAuMTIuMS0wMTUwCm1maTA6IDEwNTQgKGJvb3QgKyA1 cy8weDAwMjAvaW5mbykgLSBCb2FyZCBSZXZpc2lvbiAwNEMKbWZpMDogMTA1NSAoYm9vdCArIDI0 cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgKFNFUykgZGlzY292ZXJlZCBvbiBQRCAxNChjIFBv cnQgMCAtIDMvcDEpCm1maTA6IDEwNTYgKGJvb3QgKyAyNXMvMHgwMDA0L2luZm8pIC0gRW5jbG9z dXJlIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkgY29tbXVuaWNhdGlvbiByZXN0b3JlZAptZmkwOiAx MDU3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBFbmNsIFBEIDE0Cm1maTA6 IDEwNTggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDE0KGMgUG9ydCAw IC0gMy9wMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT1kLCBwb3J0TWFwPTAwLCBzYXNBZGRy PTUwMDE0MzgwMzA1MGY5NzksMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDU5IChib290ICsgMjVz LzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkKbWZpMDogMTA2MCAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYx LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTA2MSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogUEQgMDkoZTB4MTQvczIpCm1maTA6IDEwNjIgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0L3MyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBl PTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MiwwMDAwMDAwMDAwMDAwMDAw Cm1maTA6IDEwNjMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUw eDE0L3MzKQptZmkwOiAxMDY0IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQ RCAwYShlMHgxNC9zMykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBz YXNBZGRyPTUwMDE0MzgwMzA1MGY5NjMsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDY1IChib290 ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkKbWZpMDogMTA2 NiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGIoZTB4MTQvczQpIElu Zm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMw NTBmOTY0LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTA2NyAoYm9vdCArIDI1cy8weDAwMDIvaW5m bykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpCm1maTA6IDEwNjggKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0L3M1KSBJbmZvOiBlbmNsUGQ9MTQsIHNj c2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NSwwMDAwMDAwMDAw MDAwMDAwCm1maTA6IDEwNjkgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBE IDBkKGUweDE0L3M2KQptZmkwOiAxMDcwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAwZChlMHgxNC9zNikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFw PTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjYsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDcx IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykKbWZp MDogMTA3MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGUoZTB4MTQv czcpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAx NDM4MDMwNTBmOTY3LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTA3MyAoYm9vdCArIDI1cy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpCm1maTA6IDEwNzQgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0L3M4KSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OCwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDEwNzUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDEwKGUweDE0L3M5KQptZmkwOiAxMDc2IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAt IEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBw b3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjksMDAwMDAwMDAwMDAwMDAwMAptZmkw OiAxMDc3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMShlMHgxNC9z MTApCm1maTA6IDEwNzggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEx KGUweDE0L3MxMCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NmEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDc5IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMihlMHgxNC9zMTEpCm1maTA6IDEwODAg KGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0L3MxMSkgSW5m bzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1 MGY5NmIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDgxIChib290ICsgMjVzLzB4MDAwMi9pbmZv KSAtIEluc2VydGVkOiBQRCAxMyhlMHgxNC9zMTIpCm1maTA6IDEwODIgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmMsMDAwMDAwMDAw MDAwMDAwMAptZmkwOiAxMDgzICg0NjU5OTgzNzBzLzB4MDAyMC9pbmZvKSAtIFRpbWUgZXN0YWJs aXNoZWQgYXMgMTAvMDcvMTQgMTE6NTk6MzA7ICgxMjEgc2Vjb25kcyBzaW5jZSBwb3dlciBvbikK bWZpMDogMTA4NCAoYm9vdCArIDNzLzB4MDAyMC9pbmZvKSAtIEZpcm13YXJlIGluaXRpYWxpemF0 aW9uIHN0YXJ0ZWQgKFBDSSBJRCAwMDczLzEwMDAvOTI0MC8xMDAwKQptZmkwOiAxMDg1IChib290 ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgdmVyc2lvbiAyLjEzMC4zOTQtMjU1MAptZmkw OiAxMDg2IChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gUGFja2FnZSB2ZXJzaW9uIDIwLjEyLjEt MDE1MAptZmkwOiAxMDg3IChib290ICsgNXMvMHgwMDIwL2luZm8pIC0gQm9hcmQgUmV2aXNpb24g MDRDCm1maTA6IDEwODggKGJvb3QgKyAyNHMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIChTRVMp IGRpc2NvdmVyZWQgb24gUEQgMTQoYyBQb3J0IDAgLSAzL3AxKQptZmkwOiAxMDg5IChib290ICsg MjVzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIGNvbW11 bmljYXRpb24gcmVzdG9yZWQKbWZpMDogMTA5MCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJ bnNlcnRlZDogRW5jbCBQRCAxNAptZmkwOiAxMDkxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAt IEluc2VydGVkOiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5 cGU9ZCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTc5LDAwMDAwMDAwMDAwMDAw MDAKbWZpMDogMTA5MiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgo ZTB4MTQvczEpCm1maTA6IDEwOTMgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDA4KGUweDE0L3MxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEwOTQgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0L3MyKQptZmkwOiAx MDk1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikg SW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0Mzgw MzA1MGY5NjIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMDk2IChib290ICsgMjVzLzB4MDAwMi9p bmZvKSAtIEluc2VydGVkOiBQRCAwYShlMHgxNC9zMykKbWZpMDogMTA5NyAoYm9vdCArIDI1cy8w eDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpIEluZm86IGVuY2xQZD0xNCwg c2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYzLDAwMDAwMDAw MDAwMDAwMDAKbWZpMDogMTA5OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDog UEQgMGIoZTB4MTQvczQpCm1maTA6IDEwOTkgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5z ZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRN YXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NCwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEx MDAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBjKGUweDE0L3M1KQpt ZmkwOiAxMTAxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgx NC9zNSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjUsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTAyIChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZChlMHgxNC9zNikKbWZpMDogMTEwMyAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQvczYpIEluZm86IGVuY2xQ ZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY2LDAw MDAwMDAwMDAwMDAwMDAKbWZpMDogMTEwNCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMGUoZTB4MTQvczcpCm1maTA6IDExMDUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAs IHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NywwMDAwMDAwMDAwMDAwMDAwCm1m aTA6IDExMDYgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBmKGUweDE0 L3M4KQptZmkwOiAxMTA3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw ZihlMHgxNC9zOCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjgsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTA4IChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMChlMHgxNC9zOSkKbWZpMDogMTEwOSAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpIEluZm86 IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBm OTY5LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTExMCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMTEoZTB4MTQvczEwKQptZmkwOiAxMTExIChib290ICsgMjVzLzB4MDAw Mi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMShlMHgxNC9zMTApIEluZm86IGVuY2xQZD0xNCwgc2Nz aVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZhLDAwMDAwMDAwMDAw MDAwMDAKbWZpMDogMTExMiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQg MTIoZTB4MTQvczExKQptZmkwOiAxMTEzIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAxMihlMHgxNC9zMTEpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1h cD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZiLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTEx NCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKQpt ZmkwOiAxMTE1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAxMyhlMHgx NC9zMTIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01 MDAxNDM4MDMwNTBmOTZjLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTExNiAoNDY2MDc5MTM4cy8w eDAwMjAvaW5mbykgLSBUaW1lIGVzdGFibGlzaGVkIGFzIDEwLzA4LzE0IDEwOjI1OjM4OyAoMTIx IHNlY29uZHMgc2luY2UgcG93ZXIgb24pCm1maTA6IDExMTcgKDQ2NjA4MDQ1MnMvMHgwMDAyL1dB Uk4pIC0gUmVtb3ZlZDogUEQgMDgoZTB4MTQvczEpCm1maTA6IDExMTggKDQ2NjA4MDQ1MnMvMHgw MDAyL2luZm8pIC0gUmVtb3ZlZDogUEQgMDgoZTB4MTQvczEpIEluZm86IGVuY2xQZD0xNCwgc2Nz aVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYxLDAwMDAwMDAwMDAw MDAwMDAKbWZpMDogMTExOSAoNDY2MDgwNDUycy8weDAwMDIvaW5mbykgLSBTdGF0ZSBjaGFuZ2Ug b24gUEQgMDgoZTB4MTQvczEpIGZyb20gSkJPRCg0MCkgdG8gVU5DT05GSUdVUkVEX0JBRCgxKQpt ZmkwOiAxMTIwICg0NjYwODA0NjVzLzB4MDAwMi9XQVJOKSAtIFJlbW92ZWQ6IFBEIDA5KGUweDE0 L3MyKQptZmkwOiAxMTIxICg0NjYwODA0NjVzLzB4MDAwMi9pbmZvKSAtIFJlbW92ZWQ6IFBEIDA5 KGUweDE0L3MyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2MiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExMjIgKDQ2NjA4MDQ2 NXMvMHgwMDAyL2luZm8pIC0gU3RhdGUgY2hhbmdlIG9uIFBEIDA5KGUweDE0L3MyKSBmcm9tIEpC T0QoNDApIHRvIFVOQ09ORklHVVJFRF9CQUQoMSkKbWZpMDogMTEyMyAoNDY2MDgwNDkwcy8weDAw MDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDgoZTB4MTQvczEpCm1maTA6IDExMjQgKDQ2NjA4MDQ5 MHMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MSwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDExMjUgKDQ2NjA4MDQ5N3MvMHgwMDAyL1dBUk4pIC0gUEQgMGEo ZTB4MTQvczMpIFBhdGggNTAwMTQzODAzMDUwZjk2MyAgcmVzZXQgKFR5cGUgMDMpCm1maTA6IDEx MjYgKDQ2NjA4MDQ5N3MvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA5KGUweDE0L3MyKQpt ZmkwOiAxMTI3ICg0NjYwODA0OTdzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgx NC9zMikgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjIsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTI4ICg0NjYwODA1MDlzLzB4 MDAwMi9XQVJOKSAtIFJlbW92ZWQ6IFBEIDBkKGUweDE0L3M2KQptZmkwOiAxMTI5ICg0NjYwODA1 MDlzLzB4MDAwMi9pbmZvKSAtIFJlbW92ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NiwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDExMzAgKDQ2NjA4MDUwOXMvMHgwMDAyL2luZm8pIC0gU3RhdGUg Y2hhbmdlIG9uIFBEIDBkKGUweDE0L3M2KSBmcm9tIEpCT0QoNDApIHRvIFVOQ09ORklHVVJFRF9C QUQoMSkKbWZpMDogMTEzMSAoNDY2MDgwNTE0cy8weDAwMDIvV0FSTikgLSBSZW1vdmVkOiBQRCAw YyhlMHgxNC9zNSkKbWZpMDogMTEzMiAoNDY2MDgwNTE0cy8weDAwMDIvaW5mbykgLSBSZW1vdmVk OiBQRCAwYyhlMHgxNC9zNSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAw LCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjUsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTMzICg0 NjYwODA1MTRzLzB4MDAwMi9pbmZvKSAtIFN0YXRlIGNoYW5nZSBvbiBQRCAwYyhlMHgxNC9zNSkg ZnJvbSBKQk9EKDQwKSB0byBVTkNPTkZJR1VSRURfQkFEKDEpCm1maTA6IDExMzQgKDQ2NjA4MDUz MXMvMHgwMDAyL1dBUk4pIC0gUmVtb3ZlZDogUEQgMGIoZTB4MTQvczQpCm1maTA6IDExMzUgKDQ2 NjA4MDUzMXMvMHgwMDAyL2luZm8pIC0gUmVtb3ZlZDogUEQgMGIoZTB4MTQvczQpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY0 LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTEzNiAoNDY2MDgwNTMxcy8weDAwMDIvaW5mbykgLSBT dGF0ZSBjaGFuZ2Ugb24gUEQgMGIoZTB4MTQvczQpIGZyb20gSkJPRCg0MCkgdG8gVU5DT05GSUdV UkVEX0JBRCgxKQptZmkwOiAxMTM3ICg0NjYwODA1NTZzLzB4MDAwMi9XQVJOKSAtIFJlbW92ZWQ6 IFBEIDEzKGUweDE0L3MxMikKbWZpMDogMTEzOCAoNDY2MDgwNTU2cy8weDAwMDIvaW5mbykgLSBS ZW1vdmVkOiBQRCAxMyhlMHgxNC9zMTIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9y dE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTZjLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDog MTEzOSAoNDY2MDgwNTU2cy8weDAwMDIvaW5mbykgLSBTdGF0ZSBjaGFuZ2Ugb24gUEQgMTMoZTB4 MTQvczEyKSBmcm9tIEpCT0QoNDApIHRvIFVOQ09ORklHVVJFRF9CQUQoMSkKbWZpMDogMTE0MCAo NDY2MDgwNTU4cy8weDAwMDIvV0FSTikgLSBSZW1vdmVkOiBQRCAxMihlMHgxNC9zMTEpCm1maTA6 IDExNDEgKDQ2NjA4MDU1OHMvMHgwMDAyL2luZm8pIC0gUmVtb3ZlZDogUEQgMTIoZTB4MTQvczEx KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQz ODAzMDUwZjk2YiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExNDIgKDQ2NjA4MDU1OHMvMHgwMDAy L2luZm8pIC0gU3RhdGUgY2hhbmdlIG9uIFBEIDEyKGUweDE0L3MxMSkgZnJvbSBKQk9EKDQwKSB0 byBVTkNPTkZJR1VSRURfQkFEKDEpCm1maTA6IDExNDMgKDQ2NjA4MDU2MHMvMHgwMDAyL1dBUk4p IC0gUmVtb3ZlZDogUEQgMTEoZTB4MTQvczEwKQptZmkwOiAxMTQ0ICg0NjYwODA1NjBzLzB4MDAw Mi9pbmZvKSAtIFJlbW92ZWQ6IFBEIDExKGUweDE0L3MxMCkgSW5mbzogZW5jbFBkPTE0LCBzY3Np VHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NmEsMDAwMDAwMDAwMDAw MDAwMAptZmkwOiAxMTQ1ICg0NjYwODA1NjBzLzB4MDAwMi9pbmZvKSAtIFN0YXRlIGNoYW5nZSBv biBQRCAxMShlMHgxNC9zMTApIGZyb20gSkJPRCg0MCkgdG8gVU5DT05GSUdVUkVEX0JBRCgxKQpt ZmkwOiAxMTQ2ICg0NjYwODA1NjdzLzB4MDAwMi9XQVJOKSAtIFJlbW92ZWQ6IFBEIDBlKGUweDE0 L3M3KQptZmkwOiAxMTQ3ICg0NjYwODA1NjdzLzB4MDAwMi9pbmZvKSAtIFJlbW92ZWQ6IFBEIDBl KGUweDE0L3M3KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2NywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExNDggKDQ2NjA4MDU2 N3MvMHgwMDAyL1dBUk4pIC0gUmVtb3ZlZDogUEQgMGYoZTB4MTQvczgpCm1maTA6IDExNDkgKDQ2 NjA4MDU2N3MvMHgwMDAyL2luZm8pIC0gUmVtb3ZlZDogUEQgMGYoZTB4MTQvczgpIEluZm86IGVu Y2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY4 LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTE1MCAoNDY2MDgwNTY3cy8weDAwMDIvaW5mbykgLSBT dGF0ZSBjaGFuZ2Ugb24gUEQgMGUoZTB4MTQvczcpIGZyb20gSkJPRCg0MCkgdG8gVU5DT05GSUdV UkVEX0JBRCgxKQptZmkwOiAxMTUxICg0NjYwODA1NjdzLzB4MDAwMi9pbmZvKSAtIFN0YXRlIGNo YW5nZSBvbiBQRCAwZihlMHgxNC9zOCkgZnJvbSBKQk9EKDQwKSB0byBVTkNPTkZJR1VSRURfQkFE KDEpCm1maTA6IDExNTIgKDQ2NjA4MDU2OXMvMHgwMDAyL1dBUk4pIC0gUmVtb3ZlZDogUEQgMTAo ZTB4MTQvczkpCm1maTA6IDExNTMgKDQ2NjA4MDU2OXMvMHgwMDAyL2luZm8pIC0gUmVtb3ZlZDog UEQgMTAoZTB4MTQvczkpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwg c2FzQWRkcj01MDAxNDM4MDMwNTBmOTY5LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTE1NCAoNDY2 MDgwNTY5cy8weDAwMDIvaW5mbykgLSBTdGF0ZSBjaGFuZ2Ugb24gUEQgMTAoZTB4MTQvczkpIGZy b20gSkJPRCg0MCkgdG8gVU5DT05GSUdVUkVEX0JBRCgxKQptZmkwOiAxMTU1ICg0NjYwODA1NzNz LzB4MDAwMi9XQVJOKSAtIFJlbW92ZWQ6IFBEIDA5KGUweDE0L3MyKQptZmkwOiAxMTU2ICg0NjYw ODA1NzNzLzB4MDAwMi9pbmZvKSAtIFJlbW92ZWQ6IFBEIDA5KGUweDE0L3MyKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Miww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDExNTcgKDQ2NjA4MDU3M3MvMHgwMDAyL2luZm8pIC0gU3Rh dGUgY2hhbmdlIG9uIFBEIDA5KGUweDE0L3MyKSBmcm9tIEpCT0QoNDApIHRvIFVOQ09ORklHVVJF RF9CQUQoMSkKbWZpMDogMTE1OCAoNDY2MDgwNTc0cy8weDAwMDIvV0FSTikgLSBSZW1vdmVkOiBQ RCAwOChlMHgxNC9zMSkKbWZpMDogMTE1OSAoNDY2MDgwNTc0cy8weDAwMDIvaW5mbykgLSBSZW1v dmVkOiBQRCAwOChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFw PTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTYw ICg0NjYwODA1NzRzLzB4MDAwMi9pbmZvKSAtIFN0YXRlIGNoYW5nZSBvbiBQRCAwOChlMHgxNC9z MSkgZnJvbSBKQk9EKDQwKSB0byBVTkNPTkZJR1VSRURfQkFEKDEpCm1maTA6IDExNjEgKGJvb3Qg KyAzcy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSBpbml0aWFsaXphdGlvbiBzdGFydGVkIChQQ0kg SUQgMDA3My8xMDAwLzkyNDAvMTAwMCkKbWZpMDogMTE2MiAoYm9vdCArIDNzLzB4MDAyMC9pbmZv KSAtIEZpcm13YXJlIHZlcnNpb24gMi4xMzAuMzk0LTI1NTAKbWZpMDogMTE2MyAoYm9vdCArIDVz LzB4MDAyMC9pbmZvKSAtIFBhY2thZ2UgdmVyc2lvbiAyMC4xMi4xLTAxNTAKbWZpMDogMTE2NCAo Ym9vdCArIDVzLzB4MDAyMC9pbmZvKSAtIEJvYXJkIFJldmlzaW9uIDA0QwptZmkwOiAxMTY1IChi b290ICsgMjRzLzB4MDAwNC9pbmZvKSAtIEVuY2xvc3VyZSAoU0VTKSBkaXNjb3ZlcmVkIG9uIFBE IDE0KGMgUG9ydCAwIC0gMy9wMSkKbWZpMDogMTE2NiAoYm9vdCArIDI1cy8weDAwMDQvaW5mbykg LSBFbmNsb3N1cmUgUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBjb21tdW5pY2F0aW9uIHJlc3RvcmVk Cm1maTA6IDExNjcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IEVuY2wgUEQg MTQKbWZpMDogMTE2OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTQo YyBQb3J0IDAgLSAzL3AxKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPWQsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk3OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExNjkgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKQptZmkwOiAx MTcwIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkg SW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0Mzgw MzA1MGY5NjEsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTcxIChib290ICsgMjVzLzB4MDAwMi9p bmZvKSAtIEluc2VydGVkOiBQRCAwOShlMHgxNC9zMikKbWZpMDogMTE3MiAoYm9vdCArIDI1cy8w eDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMDkoZTB4MTQvczIpIEluZm86IGVuY2xQZD0xNCwg c2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYyLDAwMDAwMDAw MDAwMDAwMDAKbWZpMDogMTE3MyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDog UEQgMGEoZTB4MTQvczMpCm1maTA6IDExNzQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5z ZXJ0ZWQ6IFBEIDBhKGUweDE0L3MzKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRN YXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2MywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEx NzUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KQpt ZmkwOiAxMTc2IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYihlMHgx NC9zNCkgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUw MDE0MzgwMzA1MGY5NjQsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTc3IChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYyhlMHgxNC9zNSkKbWZpMDogMTE3OCAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpIEluZm86IGVuY2xQ ZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY1LDAw MDAwMDAwMDAwMDAwMDAKbWZpMDogMTE3OSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMGQoZTB4MTQvczYpCm1maTA6IDExODAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8p IC0gSW5zZXJ0ZWQ6IFBEIDBkKGUweDE0L3M2KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAs IHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2NiwwMDAwMDAwMDAwMDAwMDAwCm1m aTA6IDExODEgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0 L3M3KQptZmkwOiAxMTgyIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAw ZShlMHgxNC9zNykgSW5mbzogZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNB ZGRyPTUwMDE0MzgwMzA1MGY5NjcsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMTgzIChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZihlMHgxNC9zOCkKbWZpMDogMTE4NCAo Ym9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpIEluZm86 IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBm OTY4LDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTE4NSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMTAoZTB4MTQvczkpCm1maTA6IDExODYgKGJvb3QgKyAyNXMvMHgwMDAy L2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEwKGUweDE0L3M5KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lU eXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OSwwMDAwMDAwMDAwMDAw MDAwCm1maTA6IDExODcgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEx KGUweDE0L3MxMCkKbWZpMDogMTE4OCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRl ZDogUEQgMTEoZTB4MTQvczEwKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9 MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExODkg KGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0L3MxMSkKbWZp MDogMTE5MCAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQv czExKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2YiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDExOTEgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEzKGUweDE0L3MxMikKbWZpMDogMTE5MiAoYm9vdCAr IDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKSBJbmZvOiBlbmNs UGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2Yyww MDAwMDAwMDAwMDAwMDAwCm1maTA6IDExOTMgKDQ2NjA5Mjg1NHMvMHgwMDIwL2luZm8pIC0gVGlt ZSBlc3RhYmxpc2hlZCBhcyAxMC8wOC8xNCAxNDoxNDoxNDsgKDEyMCBzZWNvbmRzIHNpbmNlIHBv d2VyIG9uKQptZmkwOiAxMTk0ICg0NjYzMTE2MDBzLzB4MDAyMC9XQVJOKSAtIFBhdHJvbCBSZWFk IGNhbid0IGJlIHN0YXJ0ZWQsIGFzIFBEcyBhcmUgZWl0aGVyIG5vdCBPTkxJTkUsIG9yIGFyZSBp biBhIFZEIHdpdGggYW4gYWN0aXZlIHByb2Nlc3MsIG9yIGFyZSBpbiBhbiBleGNsdWRlZCBWRApt ZmkwOiAxMTk1IChib290ICsgM3MvMHgwMDIwL2luZm8pIC0gRmlybXdhcmUgaW5pdGlhbGl6YXRp b24gc3RhcnRlZCAoUENJIElEIDAwNzMvMTAwMC85MjQwLzEwMDApCm1maTA6IDExOTYgKGJvb3Qg KyAzcy8weDAwMjAvaW5mbykgLSBGaXJtd2FyZSB2ZXJzaW9uIDIuMTMwLjM5NC0yNTUwCm1maTA6 IDExOTcgKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBQYWNrYWdlIHZlcnNpb24gMjAuMTIuMS0w MTUwCm1maTA6IDExOTggKGJvb3QgKyA1cy8weDAwMjAvaW5mbykgLSBCb2FyZCBSZXZpc2lvbiAw NEMKbWZpMDogMTE5OSAoYm9vdCArIDI0cy8weDAwMDQvaW5mbykgLSBFbmNsb3N1cmUgKFNFUykg ZGlzY292ZXJlZCBvbiBQRCAxNChjIFBvcnQgMCAtIDMvcDEpCm1maTA6IDEyMDAgKGJvb3QgKyAy NXMvMHgwMDA0L2luZm8pIC0gRW5jbG9zdXJlIFBEIDE0KGMgUG9ydCAwIC0gMy9wMSkgY29tbXVu aWNhdGlvbiByZXN0b3JlZAptZmkwOiAxMjAxIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIElu c2VydGVkOiBFbmNsIFBEIDE0CnJhbmRvbTogdW5ibG9ja2luZyBkZXZpY2UuCnVzYnVzMDogNDgw TWJwcyBIaWdoIFNwZWVkIFVTQiB2Mi4wCnVzYnVzMTogMTJNYnBzIEZ1bGwgU3BlZWQgVVNCIHYx LjAKdXNidXMyOiA0ODBNYnBzIEhpZ2ggU3BlZWQgVVNCIHYyLjAKdWdlbjIuMTogPEludGVsPiBh dCB1c2J1czIKdWh1YjA6IDxJbnRlbCBFSENJIHJvb3QgSFVCLCBjbGFzcyA5LzAsIHJldiAyLjAw LzEuMDAsIGFkZHIgMT4gb24gdXNidXMyCnVnZW4xLjE6IDwweDEwM2M+IGF0IHVzYnVzMQp1aHVi MTogPDB4MTAzYyBVSENJIHJvb3QgSFVCLCBjbGFzcyA5LzAsIHJldiAxLjAwLzEuMDAsIGFkZHIg MT4gb24gdXNidXMxCnVnZW4wLjE6IDxJbnRlbD4gYXQgdXNidXMwCnVodWIyOiA8SW50ZWwgRUhD SSByb290IEhVQiwgY2xhc3MgOS8wLCByZXYgMi4wMC8xLjAwLCBhZGRyIDE+IG9uIHVzYnVzMApt ZmlzeXNwZDAgb24gbWZpMAptZmlzeXNwZDA6IDM4MTU0NDdNQiAoNzgxNDAzNzE2OCBzZWN0b3Jz KSBTWVNQRCB2b2x1bWUgKGRldmljZWlkOiA4KQptZmlzeXNwZDA6ICBTWVNQRCB2b2x1bWUgYXR0 YWNoZWQKbWZpc3lzcGQxIG9uIG1maTAKbWZpc3lzcGQxOiAzODE1NDQ3TUIgKDc4MTQwMzcxNjgg c2VjdG9ycykgU1lTUEQgdm9sdW1lIChkZXZpY2VpZDogOSkKbWZpc3lzcGQxOiAgU1lTUEQgdm9s dW1lIGF0dGFjaGVkCm1maXN5c3BkMiBvbiBtZmkwCm1maXN5c3BkMjogMzgxNTQ0N01CICg3ODE0 MDM3MTY4IHNlY3RvcnMpIFNZU1BEIHZvbHVtZSAoZGV2aWNlaWQ6IDEwKQptZmlzeXNwZDI6ICBT WVNQRCB2b2x1bWUgYXR0YWNoZWQKbWZpc3lzcGQzIG9uIG1maTAKbWZpc3lzcGQzOiAzODE1NDQ3 TUIgKDc4MTQwMzcxNjggc2VjdG9ycykgU1lTUEQgdm9sdW1lIChkZXZpY2VpZDogMTEpCm1maXN5 c3BkMzogIFNZU1BEIHZvbHVtZSBhdHRhY2hlZAptZmlzeXNwZDQgb24gbWZpMAptZmlzeXNwZDQ6 IDM4MTU0NDdNQiAoNzgxNDAzNzE2OCBzZWN0b3JzKSBTWVNQRCB2b2x1bWUgKGRldmljZWlkOiAx MikKbWZpc3lzcGQ0OiAgU1lTUEQgdm9sdW1lIGF0dGFjaGVkCm1maXN5c3BkNSBvbiBtZmkwCm1m aXN5c3BkNTogMzgxNTQ0N01CICg3ODE0MDM3MTY4IHNlY3RvcnMpIFNZU1BEIHZvbHVtZSAoZGV2 aWNlaWQ6IDEzKQptZmlzeXNwZDU6ICBTWVNQRCB2b2x1bWUgYXR0YWNoZWQKbWZpc3lzcGQ2IG9u IG1maTAKbWZpc3lzcGQ2OiAzODE1NDQ3TUIgKDc4MTQwMzcxNjggc2VjdG9ycykgU1lTUEQgdm9s dW1lIChkZXZpY2VpZDogMTQpCm1maXN5c3BkNjogIFNZU1BEIHZvbHVtZSBhdHRhY2hlZAptZmlz eXNwZDcgb24gbWZpMAptZmlzeXNwZDc6IDM4MTU0NDdNQiAoNzgxNDAzNzE2OCBzZWN0b3JzKSBT WVNQRCB2b2x1bWUgKGRldmljZWlkOiAxNSkKbWZpc3lzcGQ3OiAgU1lTUEQgdm9sdW1lIGF0dGFj aGVkCm1maXN5c3BkOCBvbiBtZmkwCm1maXN5c3BkODogMzgxNTQ0N01CICg3ODE0MDM3MTY4IHNl Y3RvcnMpIFNZU1BEIHZvbHVtZSAoZGV2aWNlaWQ6IDE2KQptZmlzeXNwZDg6ICBTWVNQRCB2b2x1 bWUgYXR0YWNoZWQKbWZpc3lzcGQ5IG9uIG1maTAKbWZpc3lzcGQ5OiAzODE1NDQ3TUIgKDc4MTQw MzcxNjggc2VjdG9ycykgU1lTUEQgdm9sdW1lIChkZXZpY2VpZDogMTcpCm1maXN5c3BkOTogIFNZ U1BEIHZvbHVtZSBhdHRhY2hlZAptZmlzeXNwZDEwIG9uIG1maTAKbWZpc3lzcGQxMDogMzgxNTQ0 N01CICg3ODE0MDM3MTY4IHNlY3RvcnMpIFNZU1BEIHZvbHVtZSAoZGV2aWNlaWQ6IDE4KQptZmlz eXNwZDEwOiAgU1lTUEQgdm9sdW1lIGF0dGFjaGVkCm1maXN5c3BkMTEgb24gbWZpMAptZmlzeXNw ZDExOiAzODE1NDQ3TUIgKDc4MTQwMzcxNjggc2VjdG9ycykgU1lTUEQgdm9sdW1lIChkZXZpY2Vp ZDogMTkpCm1maXN5c3BkMTE6ICBTWVNQRCB2b2x1bWUgYXR0YWNoZWQKbWZpMDogMTIwMiAoYm9v dCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTQoYyBQb3J0IDAgLSAzL3AxKSBJ bmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPWQsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAz MDUwZjk3OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEyMDMgKGJvb3QgKyAyNXMvMHgwMDAyL2lu Zm8pIC0gSW5zZXJ0ZWQ6IFBEIDA4KGUweDE0L3MxKQptZmkwOiAxMjA0IChib290ICsgMjVzLzB4 MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwOChlMHgxNC9zMSkgSW5mbzogZW5jbFBkPTE0LCBz Y3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjEsMDAwMDAwMDAw MDAwMDAwMAptZmkwOiAxMjA1IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQ RCAwOShlMHgxNC9zMikKbWZpMDogMTIwNiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNl cnRlZDogUEQgMDkoZTB4MTQvczIpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwgcG9ydE1h cD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTYyLDAwMDAwMDAwMDAwMDAwMDAKbWZpMDogMTIw NyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGEoZTB4MTQvczMpCm1m aTA6IDEyMDggKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBhKGUweDE0 L3MzKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAw MTQzODAzMDUwZjk2MywwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEyMDkgKGJvb3QgKyAyNXMvMHgw MDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBiKGUweDE0L3M0KQptZmkwOiAxMjEwIChib290ICsg MjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwYihlMHgxNC9zNCkgSW5mbzogZW5jbFBk PTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5NjQsMDAw MDAwMDAwMDAwMDAwMAptZmkwOiAxMjExIChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2Vy dGVkOiBQRCAwYyhlMHgxNC9zNSkKbWZpMDogMTIxMiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMGMoZTB4MTQvczUpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5cGU9MCwg cG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY1LDAwMDAwMDAwMDAwMDAwMDAKbWZp MDogMTIxMyAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMGQoZTB4MTQv czYpCm1maTA6IDEyMTQgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBk KGUweDE0L3M2KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0Fk ZHI9NTAwMTQzODAzMDUwZjk2NiwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEyMTUgKGJvb3QgKyAy NXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDBlKGUweDE0L3M3KQptZmkwOiAxMjE2IChi b290ICsgMjVzLzB4MDAwMi9pbmZvKSAtIEluc2VydGVkOiBQRCAwZShlMHgxNC9zNykgSW5mbzog ZW5jbFBkPTE0LCBzY3NpVHlwZT0wLCBwb3J0TWFwPTAwLCBzYXNBZGRyPTUwMDE0MzgwMzA1MGY5 NjcsMDAwMDAwMDAwMDAwMDAwMAptZmkwOiAxMjE3IChib290ICsgMjVzLzB4MDAwMi9pbmZvKSAt IEluc2VydGVkOiBQRCAwZihlMHgxNC9zOCkKbWZpMDogMTIxOCAoYm9vdCArIDI1cy8weDAwMDIv aW5mbykgLSBJbnNlcnRlZDogUEQgMGYoZTB4MTQvczgpIEluZm86IGVuY2xQZD0xNCwgc2NzaVR5 cGU9MCwgcG9ydE1hcD0wMCwgc2FzQWRkcj01MDAxNDM4MDMwNTBmOTY4LDAwMDAwMDAwMDAwMDAw MDAKbWZpMDogMTIxOSAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTAo ZTB4MTQvczkpCm1maTA6IDEyMjAgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6 IFBEIDEwKGUweDE0L3M5KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAs IHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2OSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEyMjEgKGJv b3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDExKGUweDE0L3MxMCkKbWZpMDog MTIyMiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTEoZTB4MTQvczEw KSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQz ODAzMDUwZjk2YSwwMDAwMDAwMDAwMDAwMDAwCm1maTA6IDEyMjMgKGJvb3QgKyAyNXMvMHgwMDAy L2luZm8pIC0gSW5zZXJ0ZWQ6IFBEIDEyKGUweDE0L3MxMSkKbWZpMDogMTIyNCAoYm9vdCArIDI1 cy8weDAwMDIvaW5mbykgLSBJbnNlcnRlZDogUEQgMTIoZTB4MTQvczExKSBJbmZvOiBlbmNsUGQ9 MTQsIHNjc2lUeXBlPTAsIHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YiwwMDAw MDAwMDAwMDAwMDAwCm1maTA6IDEyMjUgKGJvb3QgKyAyNXMvMHgwMDAyL2luZm8pIC0gSW5zZXJ0 ZWQ6IFBEIDEzKGUweDE0L3MxMikKbWZpMDogMTIyNiAoYm9vdCArIDI1cy8weDAwMDIvaW5mbykg LSBJbnNlcnRlZDogUEQgMTMoZTB4MTQvczEyKSBJbmZvOiBlbmNsUGQ9MTQsIHNjc2lUeXBlPTAs IHBvcnRNYXA9MDAsIHNhc0FkZHI9NTAwMTQzODAzMDUwZjk2YywwMDAwMDAwMDAwMDAwMDAwCm1m aTA6IDEyMjcgKDQ2NjUzMzc5MHMvMHgwMDIwL2luZm8pIC0gVGltZSBlc3RhYmxpc2hlZCBhcyAx MC8xMy8xNCAxNjo0MzoxMDsgKDEyMiBzZWNvbmRzIHNpbmNlIHBvd2VyIG9uKQpzZXMwIGF0IGFo Y2llbTAgYnVzIDAgc2NidXM2IHRhcmdldCAwIGx1biAwCnNlczA6IDxBSENJIFNHUElPIEVuY2xv c3VyZSAxLjAwIDAwMDE+IFNFTUIgUy1FLVMgMi4wMCBkZXZpY2UKc2VzMDogU0VNQiBTRVMgRGV2 aWNlCm52bWUwOiBTRVQgRkVBVFVSRVMgKDA5KSBzcWlkOjAgY2lkOjkgbnNpZDowIGNkdzEwOjAw MDAwMDgwIGNkdzExOjAwMDAwMDAwCm52bWUwOiBJTlRFUk5BTCBERVZJQ0UgRVJST1IgKDAwLzA2 KSBzcWlkOjAgY2lkOjkgY2R3MDowCk5ldHZzYyBpbml0aWFsaXppbmcuLi4gU01QOiBBUCBDUFUg IzEgTGF1bmNoZWQhClNNUDogQVAgQ1BVICMzIExhdW5jaGVkIQpTTVA6IEFQIENQVSAjMTEgTGF1 bmNoZWQhClNNUDogQVAgQ1BVICM1IExhdW5jaGVkIQpTTVA6IEFQIENQVSAjMiBMYXVuY2hlZCEK U01QOiBBUCBDUFUgIzYgTGF1bmNoZWQhClNNUDogQVAgQ1BVICM4IExhdW5jaGVkIQpTTVA6IEFQ IENQVSAjMTAgTGF1bmNoZWQhClNNUDogQVAgQ1BVICM5IExhdW5jaGVkIQpTTVA6IEFQIENQVSAj NyBMYXVuY2hlZCEKU01QOiBBUCBDUFUgIzQgTGF1bmNoZWQhClRpbWVjb3VudGVyICJUU0MtbG93 IiBmcmVxdWVuY3kgMTA5NzM4MDU0MiBIeiBxdWFsaXR5IDEwMDAKdWh1YjE6IDIgcG9ydHMgd2l0 aCAyIHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkClJvb3QgbW91bnQgd2FpdGluZyBmb3I6IHVzYnVz MiB1c2J1czEgdXNidXMwCnVodWIwOiAyIHBvcnRzIHdpdGggMiByZW1vdmFibGUsIHNlbGYgcG93 ZXJlZAp1aHViMjogMiBwb3J0cyB3aXRoIDIgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQKdWdlbjEu MjogPEhQPiBhdCB1c2J1czEKdWtiZDA6IDxWaXJ0dWFsIEtleWJvYXJkID4gb24gdXNidXMxCmti ZDIgYXQgdWtiZDAKdWdlbjIuMjogPHZlbmRvciAweDgwODc+IGF0IHVzYnVzMgp1aHViMzogPHZl bmRvciAweDgwODcgcHJvZHVjdCAweDAwMjQsIGNsYXNzIDkvMCwgcmV2IDIuMDAvMC4wMCwgYWRk ciAyPiBvbiB1c2J1czIKdWdlbjAuMjogPHZlbmRvciAweDgwODc+IGF0IHVzYnVzMAp1aHViNDog PHZlbmRvciAweDgwODcgcHJvZHVjdCAweDAwMjQsIGNsYXNzIDkvMCwgcmV2IDIuMDAvMC4wMCwg YWRkciAyPiBvbiB1c2J1czAKUm9vdCBtb3VudCB3YWl0aW5nIGZvcjogdXNidXMyIHVzYnVzMAp1 aHViNDogNiBwb3J0cyB3aXRoIDYgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQKdWh1YjM6IDggcG9y dHMgd2l0aCA4IHJlbW92YWJsZSwgc2VsZiBwb3dlcmVkCnVnZW4yLjM6IDx2ZW5kb3IgMHgwNDI0 PiBhdCB1c2J1czIKdWh1YjU6IDx2ZW5kb3IgMHgwNDI0IHByb2R1Y3QgMHgyNjYwLCBjbGFzcyA5 LzAsIHJldiAyLjAwLzguMDEsIGFkZHIgMz4gb24gdXNidXMyClJvb3QgbW91bnQgd2FpdGluZyBm b3I6IHVzYnVzMgp1aHViNTogMiBwb3J0cyB3aXRoIDEgcmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQK Um9vdCBtb3VudCB3YWl0aW5nIGZvcjogdXNidXMyCnVnZW4yLjQ6IDxHZW5lcmljPiBhdCB1c2J1 czIKdW1hc3MwOiA8R2VuZXJpYyBVbHRyYSBGYXN0IE1lZGlhIFJlYWRlciwgY2xhc3MgMC8wLCBy ZXYgMi4wMC8yLjA5LCBhZGRyIDQ+IG9uIHVzYnVzMgp1bWFzczA6ICBTQ1NJIG92ZXIgQnVsay1P bmx5OyBxdWlya3MgPSAweDQxMDAKdW1hc3MwOjc6MDotMTogQXR0YWNoZWQgdG8gc2NidXM3CmRh MCBhdCB1bWFzcy1zaW0wIGJ1cyAwIHNjYnVzNyB0YXJnZXQgMCBsdW4gMApkYTA6IDxIUCBpTE8g SW50ZXJuYWwgU0QtQ0FSRCAyLjA5PiBGaXhlZCBEaXJlY3QgQWNjZXNzIFNDU0ktMCBkZXZpY2Ug CmRhMDogU2VyaWFsIE51bWJlciAwMDAwMDI2NjBBMDEKZGEwOiA0MC4wMDBNQi9zIHRyYW5zZmVy cwpkYTA6IDc2MDJNQiAoMTU1Njg4OTYgNTEyIGJ5dGUgc2VjdG9yczogMjU1SCA2M1MvVCA5NjlD KQpkYTA6IHF1aXJrcz0weDI8Tk9fNl9CWVRFPgpUcnlpbmcgdG8gbW91bnQgcm9vdCBmcm9tIHpm czp6cm9vdC9ST09UL2RlZmF1bHQgW10uLi4KdW1zMDogPFZpcnR1YWwgTW91c2U+IG9uIHVzYnVz MQpwaWQgMjA4MyAobGFkdmQpLCB1aWQgMDogZXhpdGVkIG9uIHNpZ25hbCA2IChjb3JlIGR1bXBl ZCkKbnZkMDogPElOVEVMIFNTRFBFRE1FNDAwRzQ+IE5WTWUgbmFtZXNwYWNlCm52ZDA6IDM4MTU1 NE1CICg3ODE0MjI3NjggNTEyIGJ5dGUgc2VjdG9ycykKdWdlbjEuMjogPEhQPiBhdCB1c2J1czEg KGRpc2Nvbm5lY3RlZCkKdWtiZDA6IGF0IHVodWIxLCBwb3J0IDEsIGFkZHIgMiAoZGlzY29ubmVj dGVkKQp1bXMwOiBhdCB1aHViMSwgcG9ydCAxLCBhZGRyIDIgKGRpc2Nvbm5lY3RlZCkK --_004_D062AFA3206A0maikelverheijenredwoodcom_-- From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 09:50:29 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABF0B490 for ; Tue, 14 Oct 2014 09:50:29 +0000 (UTC) Received: from smtp3-g21.free.fr (smtp3-g21.free.fr [212.27.42.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 690F1242 for ; Tue, 14 Oct 2014 09:50:29 +0000 (UTC) Received: from yggdrasil.alkumuna.eu (unknown [IPv6:2a01:e35:8a74:6e70:232:36ff:fe5c:3a87]) by smtp3-g21.free.fr (Postfix) with ESMTP id 3B2C0A623A for ; Tue, 14 Oct 2014 11:49:13 +0200 (CEST) Received: from ist-159-28.ujf-grenoble.fr (ist-159-28.ujf-grenoble.fr [152.77.159.28]) (authenticated bits=0) by yggdrasil.alkumuna.eu (8.14.7/8.14.7) with ESMTP id s9E9oIHE069696 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 14 Oct 2014 11:50:19 +0200 (CEST) (envelope-from mazhe@alkumuna.eu) From: Matthieu Volat Content-Type: multipart/signed; boundary="Apple-Mail=_BA8326CE-A77C-495C-BA68-732324C97F4F"; protocol="application/pgp-signature"; micalg=pgp-sha1 Message-Id: <2553F6A9-0363-491B-9585-E12AD32CF0F6@alkumuna.eu> Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Date: Tue, 14 Oct 2014 11:50:21 +0200 Subject: VT switch to console disabled once X starts To: freebsd-stable@freebsd.org X-Mailer: Apple Mail (2.1878.6) Cc: troyax@pholos.eu X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 09:50:29 -0000 --Apple-Mail=_BA8326CE-A77C-495C-BA68-732324C97F4F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hello, I've had this problem with a HP Z400 workstation which is currently = running 10.1-RC2, but had it since I put FreeBSD 10 on it: using either = syscons or vt, I can see the console output when the system boots. But once the X server start (through xdm, using the nvidia driver), if I = try to switch to another screen, I get only a black screen, but I can = get back to vt #9 and get my X session back. If I don't set xdm in = /etc/ttys, I can use the consoles... I have no relevant information in dmesg and little in syslog: only = messages from console-kit-daemon giving up on the consoles (Device not = configured) and devd messages: devd: check_clients: dropping disconnected clients when switching. Does anybody have an idea of what is wrong with my setup? Thanks, -- Matthieu Volat --Apple-Mail=_BA8326CE-A77C-495C-BA68-732324C97F4F Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlQ88eEACgkQ+ENDeYKZi369CQCgx8dmE2UA1sSHsKeBhiKovuSD NSEAn3nWH1+ZiWyYaXBFImxTBeN2v+0k =/Khk -----END PGP SIGNATURE----- --Apple-Mail=_BA8326CE-A77C-495C-BA68-732324C97F4F-- From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 10:24:11 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D1132C4D for ; Tue, 14 Oct 2014 10:24:11 +0000 (UTC) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 925F27B3 for ; Tue, 14 Oct 2014 10:24:11 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1XdzGt-0006Q3-9t for freebsd-stable@freebsd.org; Tue, 14 Oct 2014 12:24:08 +0200 Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes To: freebsd-stable@freebsd.org Subject: Re: VT switch to console disabled once X starts References: <2553F6A9-0363-491B-9585-E12AD32CF0F6@alkumuna.eu> Date: Tue, 14 Oct 2014 12:24:02 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: "Ronald Klop" Message-ID: In-Reply-To: <2553F6A9-0363-491B-9585-E12AD32CF0F6@alkumuna.eu> User-Agent: Opera Mail/12.17 (Win32) X-Authenticated-As-Hash: 398f5522cb258ce43cb679602f8cfe8b62a256d1 X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: -- X-Spam-Score: -2.9 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED, BAYES_00 autolearn=disabled version=3.3.2 X-Scan-Signature: 9090f8a1960d7f777b94d17b6f36e747 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 10:24:11 -0000 On Tue, 14 Oct 2014 11:50:21 +0200, Matthieu Volat wrote: > Hello, > > I've had this problem with a HP Z400 workstation which is currently > running 10.1-RC2, but had it since I put FreeBSD 10 on it: using either > syscons or vt, I can see the console output when the system boots. > > But once the X server start (through xdm, using the nvidia driver), if I > try to switch to another screen, I get only a black screen, but I can > get back to vt #9 and get my X session back. If I don't set xdm in > /etc/ttys, I can use the consoles... > > I have no relevant information in dmesg and little in syslog: only > messages from console-kit-daemon giving up on the consoles (Device not > configured) and devd messages: > devd: check_clients: dropping disconnected clients > when switching. > > Does anybody have an idea of what is wrong with my setup? > > Thanks, > > -- Matthieu Volat > People will like a little bit more information about your hardware and settings. The output of 'dmesg' is helpfull for a start. And the content of /var/log/Xorg.0.log. Regards, Ronald. From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 11:20:06 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0EEEACF5; Tue, 14 Oct 2014 11:20:06 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 775F7C84; Tue, 14 Oct 2014 11:20:04 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 99E9F20E70942; Tue, 14 Oct 2014 11:20:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id E634D20E70940; Tue, 14 Oct 2014 11:20:00 +0000 (UTC) Message-ID: <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "K. Macy" References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 12:19:58 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: "freebsd-fs@FreeBSD.org" , FreeBSD Stable , mark X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:20:06 -0000 ----- Original Message ----- From: "Steven Hartland" To: "K. Macy" Cc: "freebsd-fs@FreeBSD.org" ; "mark" ; "FreeBSD Stable" Sent: Tuesday, October 14, 2014 9:14 AM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > ----- Original Message ----- > From: "K. Macy" > > >>>> Thank you both for analysis and effort! >>>> >>>> I can't rule out the possibility that my main system pool >>>> on a SSD was low on space at some point in time, but the >>>> three 4 GiB cloned pools (sys1boot and its brothers) were all >>>> created as a zfs send / receive copies of the main / (root) >>>> file system and I haven't noticed anything unusual during >>>> syncing. This syncing was done manually (using zxfer) and >>>> independently from the upgrade on the system - on a steady/quiet >>>> system, when the source file system definitely had sufficient >>>> free space. >>>> >>>> The source file system now shows 1.2 GiB of usage shown >>>> by df: >>>> shiny/ROOT 61758388 1271620 60486768 2% / >>>> Seems unlikely that the 1.2 GiB has grown to 4 GiB space >>>> on a cloned filesystem. >>>> >>>> Will try to import the main two pools after re-creating >>>> a sane boot pool... >>> >>> >>> Yer zfs list only shows around 2-3GB used too but zpool list >>> shows the pool is out of space. Cant rule out an accounting >>> issue though. >>> >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > Its not clear but I believe it could just be fragmention even though > its ashift=9. > > I sent the last snapshot to another pool of the same size and it > resulted in: > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - > > I believe FRAG is 0% as the feature wasn't enabled for the lifetime of > the pool hence its simply not showing a valid value. > > zfs list -t all -r sys1boot > NAME USED AVAIL REFER MOUNTPOINT > sys1boot 1.76G 2.08G 11K /sys1boot > sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT > sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - .. Well interesting issue I left this pool alone this morning literally doing nothing, and its now out of space. zpool list NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH ALTROOT sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - There's something very wrong here as nothing has been accessing the pool. pool: zfs state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://illumos.org/msg/ZFS-8000-HC scan: none requested config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 2 0 md1 ONLINE 0 0 0 I tried destroying the pool and ever that failed, presumably because the pool has suspended IO. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 11:27:00 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6F1B4235; Tue, 14 Oct 2014 11:27:00 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id ED1FCD89; Tue, 14 Oct 2014 11:26:59 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jHDx56fGxz1Dh; Tue, 14 Oct 2014 13:26:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413286005; x=1415878006; bh=JAltATHwlSD2F6q4NoNW9s7EL aNvB/11Qu7cvsHFvzE=; b=eReAxFzeG9+BpS7T/Ll5GYTK2ejDQwafFC51KiYlX CFmlYC8EyIPIeWvA+hqDzmau5S2q2J1lCQy8mZvwIsnMw3MCy6roV/UIMLGFjED4 YW/ErPxn7ZhszP4FCAXNTF91W8QlQVW0jKER/1uuhtBQYeEYUQV9U9S9LI9vTj2l yc= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id UUa2MEOZHAAm; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jHDx128FMzBt; Tue, 14 Oct 2014 13:26:45 +0200 (CEST) Message-ID: <543D0874.9080809@ijs.si> Date: Tue, 14 Oct 2014 13:26:44 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:27:00 -0000 On 10/14/2014 10:14, Steven Hartland wrote: >>> Yer zfs list only shows around 2-3GB used too but zpool list >>> shows the pool is out of space. Cant rule out an accounting >>> issue though. >> >> What is using the extra space in the pool? Is there an unmounted >> dataset or snapshot? Do you know how to easily tell? Unlike txg and >> zio processing I don't have the luxury of having just read that part >> of the codebase. > > Its not clear but I believe it could just be fragmention even though > its ashift=9. > > I sent the last snapshot to another pool of the same size and it > resulted in: > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH > ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.47G 512M 72% - 87% 1.00x ONLINE - Yes, that's it! Fragmentation. > I believe FRAG is 0% as the feature wasn't enabled for the lifetime of > the pool hence its simply not showing a valid value. Indeed. The pool has a long lifetime and the feature was only recently made available. > zfs list -t all -r sys1boot > NAME USED AVAIL REFER MOUNTPOINT > sys1boot 1.76G 2.08G 11K /sys1boot > sys1boot/ROOT 1.72G 2.08G 1.20G /sys1boot/ROOT > sys1boot/ROOT@auto-2014-08-16_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-17_04.00 1K - 1.19G - > sys1boot/ROOT@auto-2014-08-19_04.00 1K - 1.19G - > [...] > sys1boot/ROOT@auto-2014-09-19_22.20 1K - 1.20G - > sys1boot/ROOT@auto-2014-09-19_22.30 0 - 1.20G - So snapshots were not consuming much, it was fragmentation. Thanks! Mark From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 11:30:33 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A3F196B2; Tue, 14 Oct 2014 11:30:33 +0000 (UTC) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) by mx1.freebsd.org (Postfix) with ESMTP id 2FF25DDF; Tue, 14 Oct 2014 11:30:33 +0000 (UTC) Received: from amavis-proxy-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3jHF1N4138z1Gc; Tue, 14 Oct 2014 13:30:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:organization :from:from:date:date:message-id:received:received:received; s= jakla4; t=1413286227; x=1415878228; bh=zLHNxpSja50NlONXqgXlIWA5a gJI6zlDpAKAum9KRSE=; b=HAYylhABxzW6VWCfGn9b6LoSwYcUJNfTmPn3i0nX4 ePT8QdCCpmpBbtHW5HkdogcMjcwkrWljoxn7+9/KfzAvMKsaB6iWurKO1n00vaDJ 0AhgKSz59CUml2BZhnVAMaM7hAeXBtmO2JkHa38y6lUhc7KmzUiKauTLoEUYEuZM vk= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-proxy-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10012) with ESMTP id YIifCGDKsbao; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Received: from sleepy.ijs.si (msleepy-1-pt.tunnel.tserv27.prg1.ipv6.he.net [IPv6:2001:470:6e:18e::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mildred.ijs.si (Postfix) with ESMTPSA id 3jHF1H48xzzDy; Tue, 14 Oct 2014 13:30:27 +0200 (CEST) Message-ID: <543D0953.1070604@ijs.si> Date: Tue, 14 Oct 2014 13:30:27 +0200 From: Mark Martinec Organization: J. Stefan Institute User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> In-Reply-To: <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:30:33 -0000 On 10/14/2014 13:19, Steven Hartland wrote: > Well interesting issue I left this pool alone this morning literally doing > nothing, and its now out of space. > zpool list > NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH > ALTROOT > sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - > sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - > > There's something very wrong here as nothing has been accessing the pool. > > pool: zfs > state: ONLINE > status: One or more devices are faulted in response to IO failures. > action: Make sure the affected devices are connected, then run 'zpool > clear'. > see: http://illumos.org/msg/ZFS-8000-HC > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > zfs ONLINE 0 2 0 > md1 ONLINE 0 0 0 > > I tried destroying the pool and ever that failed, presumably because > the pool has suspended IO. That's exactly how trouble started here. Got the "One or more devices are faulted in response to IO failures" on all three small cloned boot pools one day, out of the blue. There was no activity there, except for periodic snapshoting every 10 minutes. Mark From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 11:40:51 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07ED7A4D; Tue, 14 Oct 2014 11:40:51 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id B5FF3F4F; Tue, 14 Oct 2014 11:40:50 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 980F420E70953; Tue, 14 Oct 2014 11:40:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 0729020E70942; Tue, 14 Oct 2014 11:40:47 +0000 (UTC) Message-ID: From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Tue, 14 Oct 2014 12:40:45 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 11:40:51 -0000 ----- Original Message ----- From: "Mark Martinec" > On 10/14/2014 13:19, Steven Hartland wrote: >> Well interesting issue I left this pool alone this morning literally doing >> nothing, and its now out of space. >> zpool list >> NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH >> ALTROOT >> sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - >> sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - >> >> There's something very wrong here as nothing has been accessing the pool. >> >> pool: zfs >> state: ONLINE >> status: One or more devices are faulted in response to IO failures. >> action: Make sure the affected devices are connected, then run 'zpool >> clear'. >> see: http://illumos.org/msg/ZFS-8000-HC >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> zfs ONLINE 0 2 0 >> md1 ONLINE 0 0 0 >> >> I tried destroying the pool and ever that failed, presumably because >> the pool has suspended IO. > > That's exactly how trouble started here. Got the > "One or more devices are faulted in response to IO failures" > on all three small cloned boot pools one day, out of the blue. > There was no activity there, except for periodic snapshoting > every 10 minutes. Yer this isn't fragmentation, this is something else. I've started a thread on the openzfs list to discuss this as theres something quite odd going on. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 12:53:42 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A811D479 for ; Tue, 14 Oct 2014 12:53:42 +0000 (UTC) Received: from mail-lb0-x22b.google.com (mail-lb0-x22b.google.com [IPv6:2a00:1450:4010:c04::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC6549A7 for ; Tue, 14 Oct 2014 12:53:41 +0000 (UTC) Received: by mail-lb0-f171.google.com with SMTP id z12so8097988lbi.2 for ; Tue, 14 Oct 2014 05:53:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=iObcGhQhzgWWtIyY2SOCmFUreE2t54khWLpG6Eb5ltg=; b=lmohWhLRZ51Pnqvr5u7D1d5AQbufsIsuaq3usprHts2PejY5kBHR7HCpkb2Ay+NMc/ RYPmT1sDP7H7kRkQpnpm56rt28ak5QycCGY75MwO1H54PUIsLWszecrihxUo9UJdoSNS N7JsGq6P/GOFA4GzamraK/KnXfihD7Cs4FRfMVcnjTKvyrVqnPTvPV6I1wz2ihxnnGr3 OhzD8OIhv94DP2e0wNknq4cyTQ02Wpm8ImVxzc+ULZhOjBiB+gh8zw5YSRlrzyvkkCJs JfEn16cAELKbLYsikRjRw0rmbSvjZrxOfSALRs3tjAllla6Z9JeHQLw0Qrm1A8fwx7Pm Qa+g== MIME-Version: 1.0 X-Received: by 10.112.169.66 with SMTP id ac2mr5072226lbc.73.1413291219239; Tue, 14 Oct 2014 05:53:39 -0700 (PDT) Received: by 10.25.166.17 with HTTP; Tue, 14 Oct 2014 05:53:38 -0700 (PDT) Date: Tue, 14 Oct 2014 16:53:38 +0400 Message-ID: Subject: Poll timeouts on Marvell 88SE9123 From: KOT MATPOCKuH To: freebsd-stable@freebsd.org Content-Type: multipart/mixed; boundary=001a11c38da01668dd05056181d1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 12:53:42 -0000 --001a11c38da01668dd05056181d1 Content-Type: text/plain; charset=UTF-8 Hi all! After upgrading from r251990 I got unusable all my HDD that connected to Marvell 88SE9123 PCI-E card. Probing fails with messages like this: Oct 14 06:58:17 green kernel: ahcich7: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 50 serr 00000000 cmd 10000016 Oct 14 06:58:17 green kernel: (aprobe2:ahcich7:0:0:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 Oct 14 06:58:17 green kernel: (aprobe2:ahcich7:0:0:0): CAM status: Command timeout Oct 14 06:58:17 green kernel: (aprobe2:ahcich7:0:0:0): Error 5, Retries exhausted How I can debug or solve this problem? Full boot log from messages and pciconf -lv output attached. -- MATPOCKuH --001a11c38da01668dd05056181d1 Content-Type: application/octet-stream; name=messages Content-Disposition: attachment; filename=messages Content-Transfer-Encoding: base64 X-Attachment-Id: f_i19911gg0 T2N0IDE0IDA2OjU4OjE3IGdyZWVuIHN5c2xvZ2Q6IGtlcm5lbCBib290IGZpbGUgaXMgL2Jvb3Qv a2VybmVsL2tlcm5lbApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBDb3B5cmlnaHQgKGMp IDE5OTItMjAxNCBUaGUgRnJlZUJTRCBQcm9qZWN0LgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBDb3B5cmlnaHQgKGMpIDE5NzksIDE5ODAsIDE5ODMsIDE5ODYsIDE5ODgsIDE5ODksIDE5 OTEsIDE5OTIsIDE5OTMsIDE5OTQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogVGhlIFJl Z2VudHMgb2YgdGhlIFVuaXZlcnNpdHkgb2YgQ2FsaWZvcm5pYS4gQWxsIHJpZ2h0cyByZXNlcnZl ZC4KT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogRnJlZUJTRCBpcyBhIHJlZ2lzdGVyZWQg dHJhZGVtYXJrIG9mIFRoZSBGcmVlQlNEIEZvdW5kYXRpb24uCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IEZyZWVCU0QgOS4zLVNUQUJMRSAjOTQgcjI3MjA5Mk06IFR1ZSBPY3QgMTQgMDA6 Mzk6NTEgTVNLIDIwMTQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcm9vdEBncmVlbjov dXNyL29iai91c3Ivc3JjL3N5cy9ncmVlbiBhbWQ2NApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBnY2MgdmVyc2lvbiA0LjIuMSAyMDA3MDgzMSBwYXRjaGVkIFtGcmVlQlNEXQpPY3QgMTQg MDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBtb2R1bGVfcmVnaXN0ZXI6IG1vZHVsZSBybC9taWlidXMg YWxyZWFkeSBleGlzdHMhCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IE1vZHVsZSBybC9t aWlidXMgZmFpbGVkIHRvIHJlZ2lzdGVyOiAxNwpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVs OiBtb2R1bGVfcmVnaXN0ZXI6IG1vZHVsZSBjYXJkYnVzL3JsIGFscmVhZHkgZXhpc3RzIQpPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBNb2R1bGUgY2FyZGJ1cy9ybCBmYWlsZWQgdG8gcmVn aXN0ZXI6IDE3Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IG1vZHVsZV9yZWdpc3Rlcjog bW9kdWxlIHBjaS9ybCBhbHJlYWR5IGV4aXN0cyEKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5l bDogTW9kdWxlIHBjaS9ybCBmYWlsZWQgdG8gcmVnaXN0ZXI6IDE3Ck9jdCAxNCAwNjo1ODoxNyBn cmVlbiBrZXJuZWw6IENQVTogQU1EIEF0aGxvbih0bSkgNjQgWDIgRHVhbCBDb3JlIFByb2Nlc3Nv ciAzODAwKyAoMjAwOS4xOS1NSHogSzgtY2xhc3MgQ1BVKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBPcmlnaW4gPSAiQXV0aGVudGljQU1EIiAgSWQgPSAweDQwZmIyICBGYW1pbHkgPSAw eGYgIE1vZGVsID0gMHg0YiAgU3RlcHBpbmcgPSAyCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJu ZWw6IEZlYXR1cmVzPTB4MTc4YmZiZmY8RlBVLFZNRSxERSxQU0UsVFNDLE1TUixQQUUsTUNFLENY OCxBUElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsQ0xGTFVTSCxNTVgsRlhTUixT U0UsU1NFMixIVFQ+Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IEZlYXR1cmVzMj0weDIw MDE8U1NFMyxDWDE2PgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBBTUQgRmVhdHVyZXM9 MHhlYTUwMDgwMDxTWVNDQUxMLE5YLE1NWCssRkZYU1IsUkRUU0NQLExNLDNETm93ISssM0ROb3ch PgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBBTUQgRmVhdHVyZXMyPTB4MWY8TEFIRixD TVAsU1ZNLEV4dEFQSUMsQ1I4PgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiByZWFsIG1l bW9yeSAgPSA4NTg5OTM0NTkyICg4MTkyIE1CKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVs OiBhdmFpbCBtZW1vcnkgPSA4MTY2NDYxNDQwICg3Nzg4IE1CKQpPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBFdmVudCB0aW1lciAiTEFQSUMiIHF1YWxpdHkgNDAwCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IEFDUEkgQVBJQyBUYWJsZTogPEdCVCAgICBBV1JEQUNQST4KT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogRnJlZUJTRC9TTVA6IE11bHRpcHJvY2Vzc29yIFN5c3Rl bSBEZXRlY3RlZDogMiBDUFVzCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IEZyZWVCU0Qv U01QOiAxIHBhY2thZ2UocykgeCAyIGNvcmUocykKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5l bDogY3B1MCAoQlNQKTogQVBJQyBJRDogIDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDog Y3B1MSAoQVApOiBBUElDIElEOiAgMQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBTZWN1 cml0eSBwb2xpY3kgbG9hZGVkOiBUcnVzdGVkQlNEIE1BQy9wb3J0YWNsIChtYWNfcG9ydGFjbCkK T2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogU2VjdXJpdHkgcG9saWN5IGxvYWRlZDogVHJ1 c3RlZEJTRCBNQUMvc2Vlb3RoZXJ1aWRzIChtYWNfc2Vlb3RoZXJ1aWRzKQpPY3QgMTQgMDY6NTg6 MTcgZ3JlZW4ga2VybmVsOiBpb2FwaWMwOiBDaGFuZ2luZyBBUElDIElEIHRvIDIKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogaW9hcGljMCA8VmVyc2lvbiAxLjE+IGlycXMgMC0yMyBvbiBt b3RoZXJib2FyZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBrYmQxIGF0IGtiZG11eDAK T2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWNwaTA6IDxHQlQgQVdSREFDUEk+IG9uIG1v dGhlcmJvYXJkCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFjcGkwOiBQb3dlciBCdXR0 b24gKGZpeGVkKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhY3BpMDogcmVzZXJ2YXRp b24gb2YgMCwgYTAwMDAgKDMpIGZhaWxlZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBh Y3BpMDogcmVzZXJ2YXRpb24gb2YgMTAwMDAwLCBiYmVmMDAwMCAoMykgZmFpbGVkCk9jdCAxNCAw Njo1ODoxNyBncmVlbiBrZXJuZWw6IGNwdTA6IDxBQ1BJIENQVT4gb24gYWNwaTAKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogY3B1MTogPEFDUEkgQ1BVPiBvbiBhY3BpMApPY3QgMTQgMDY6 NTg6MTcgZ3JlZW4ga2VybmVsOiBhdHRpbWVyMDogPEFUIHRpbWVyPiBwb3J0IDB4NDAtMHg0MyBp cnEgMCBvbiBhY3BpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBUaW1lY291bnRlciAi aTgyNTQiIGZyZXF1ZW5jeSAxMTkzMTgyIEh6IHF1YWxpdHkgMApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBhdHRpbWVyMDogQ2FuJ3QgbWFwIGludGVycnVwdC4KT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogYXRydGMwOiA8QVQgcmVhbHRpbWUgY2xvY2s+IHBvcnQgMHg3MC0weDcz IGlycSA4IG9uIGFjcGkwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IEV2ZW50IHRpbWVy ICJSVEMiIGZyZXF1ZW5jeSAzMjc2OCBIeiBxdWFsaXR5IDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVu IGtlcm5lbDogYWNwaV9idXR0b24wOiA8UG93ZXIgQnV0dG9uPiBvbiBhY3BpMApPY3QgMTQgMDY6 NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2liMDogPEFDUEkgSG9zdC1QQ0kgYnJpZGdlPiBwb3J0IDB4 Y2Y4LTB4Y2ZmIG9uIGFjcGkwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaTA6IDxB Q1BJIFBDSSBidXM+IG9uIHBjaWIwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaTA6 IDxtZW1vcnksIFJBTT4gYXQgZGV2aWNlIDAuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQpPY3QgMTQg MDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2kwOiA8bWVtb3J5LCBSQU0+IGF0IGRldmljZSAwLjEg KG5vIGRyaXZlciBhdHRhY2hlZCkKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcGNpMDog PG1lbW9yeSwgUkFNPiBhdCBkZXZpY2UgMC4yIChubyBkcml2ZXIgYXR0YWNoZWQpCk9jdCAxNCAw Njo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaTA6IDxtZW1vcnksIFJBTT4gYXQgZGV2aWNlIDAuMyAo bm8gZHJpdmVyIGF0dGFjaGVkKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2kwOiA8 bWVtb3J5LCBSQU0+IGF0IGRldmljZSAwLjQgKG5vIGRyaXZlciBhdHRhY2hlZCkKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogcGNpMDogPG1lbW9yeSwgUkFNPiBhdCBkZXZpY2UgMC41IChu byBkcml2ZXIgYXR0YWNoZWQpCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaTA6IDxt ZW1vcnksIFJBTT4gYXQgZGV2aWNlIDAuNiAobm8gZHJpdmVyIGF0dGFjaGVkKQpPY3QgMTQgMDY6 NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2kwOiA8bWVtb3J5LCBSQU0+IGF0IGRldmljZSAwLjcgKG5v IGRyaXZlciBhdHRhY2hlZCkKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcGNpYjE6IDxB Q1BJIFBDSS1QQ0kgYnJpZGdlPiBhdCBkZXZpY2UgMi4wIG9uIHBjaTAKT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogcGNpNDogPEFDUEkgUENJIGJ1cz4gb24gcGNpYjEKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogYWhjaTA6IDxNYXJ2ZWxsIDg4U0U5MTJ4IEFIQ0kgU0FUQSBjb250 cm9sbGVyPiBwb3J0IDB4ODAwMC0weDgwMDcsMHg4NDAwLTB4ODQwMywweDg4MDAtMHg4ODA3LDB4 OGMwMC0weDhjMDMsMHg5MDAwLTB4OTAwZiBtZW0gMHhlYTAwMDAwMC0weGVhMDAwN2ZmIGlycSAx NiBhdCBkZXZpY2UgMC4wIG9uIHBjaTQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhj aTA6IEFIQ0kgdjEuMjAgd2l0aCA4IDZHYnBzIHBvcnRzLCBQb3J0IE11bHRpcGxpZXIgbm90IHN1 cHBvcnRlZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpMDogcXVpcmtzPTB4OTAw PE5PQlNZUkVTLEFMVFNJRz4KT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoMDog PEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAwIG9uIGFoY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGFoY2ljaDE6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhaGNpMApP Y3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2gyOiA8QUhDSSBjaGFubmVsPiBhdCBj aGFubmVsIDIgb24gYWhjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoMzog PEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCAzIG9uIGFoY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGFoY2ljaDQ6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgNCBvbiBhaGNpMApP Y3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2g1OiA8QUhDSSBjaGFubmVsPiBhdCBj aGFubmVsIDUgb24gYWhjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoNjog PEFIQ0kgY2hhbm5lbD4gYXQgY2hhbm5lbCA2IG9uIGFoY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGFoY2ljaDc6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgNyBvbiBhaGNpMApP Y3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhdGFwY2kwOiA8TWFydmVsbCA4OFNFOTEyeCBV RE1BMTMzIGNvbnRyb2xsZXI+IHBvcnQgMHg5NDAwLTB4OTQwNywweDk4MDAtMHg5ODAzLDB4OWMw MC0weDljMDcsMHhhMDAwLTB4YTAwMywweGE0MDAtMHhhNDBmIG1lbSAweGVhMDAxMDAwLTB4ZWEw MDEwMGYgaXJxIDE2IGF0IGRldmljZSAwLjEgb24gcGNpNApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBwY2liMjogPEFDUEkgUENJLVBDSSBicmlkZ2U+IGF0IGRldmljZSAzLjAgb24gcGNp MApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2kzOiA8QUNQSSBQQ0kgYnVzPiBvbiBw Y2liMgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpMTogPE1hcnZlbGwgODhTRTkx MnggQUhDSSBTQVRBIGNvbnRyb2xsZXI+IHBvcnQgMHg1MDAwLTB4NTAwNywweDU0MDAtMHg1NDAz LDB4NTgwMC0weDU4MDcsMHg1YzAwLTB4NWMwMywweDYwMDAtMHg2MDBmIG1lbSAweGU4MDAwMDAw LTB4ZTgwMDA3ZmYgaXJxIDE2IGF0IGRldmljZSAwLjAgb24gcGNpMwpPY3QgMTQgMDY6NTg6MTcg Z3JlZW4ga2VybmVsOiBhaGNpMTogQUhDSSB2MS4yMCB3aXRoIDggNkdicHMgcG9ydHMsIFBvcnQg TXVsdGlwbGllciBub3Qgc3VwcG9ydGVkCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFo Y2kxOiBxdWlya3M9MHg5MDA8Tk9CU1lSRVMsQUxUU0lHPgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBhaGNpY2g4OiA8QUhDSSBjaGFubmVsPiBhdCBjaGFubmVsIDAgb24gYWhjaTEKT2N0 IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoOTogPEFIQ0kgY2hhbm5lbD4gYXQgY2hh bm5lbCAxIG9uIGFoY2kxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFoY2ljaDEwOiA8 QUhDSSBjaGFubmVsPiBhdCBjaGFubmVsIDIgb24gYWhjaTEKT2N0IDE0IDA2OjU4OjE3IGdyZWVu IGtlcm5lbDogYWhjaWNoMTE6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgMyBvbiBhaGNpMQpP Y3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2gxMjogPEFIQ0kgY2hhbm5lbD4gYXQg Y2hhbm5lbCA0IG9uIGFoY2kxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFoY2ljaDEz OiA8QUhDSSBjaGFubmVsPiBhdCBjaGFubmVsIDUgb24gYWhjaTEKT2N0IDE0IDA2OjU4OjE3IGdy ZWVuIGtlcm5lbDogYWhjaWNoMTQ6IDxBSENJIGNoYW5uZWw+IGF0IGNoYW5uZWwgNiBvbiBhaGNp MQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2gxNTogPEFIQ0kgY2hhbm5lbD4g YXQgY2hhbm5lbCA3IG9uIGFoY2kxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGF0YXBj aTE6IDxNYXJ2ZWxsIDg4U0U5MTJ4IFVETUExMzMgY29udHJvbGxlcj4gcG9ydCAweDY0MDAtMHg2 NDA3LDB4NjgwMC0weDY4MDMsMHg2YzAwLTB4NmMwNywweDcwMDAtMHg3MDAzLDB4NzQwMC0weDc0 MGYgbWVtIDB4ZTgwMDEwMDAtMHhlODAwMTAwZiBpcnEgMTYgYXQgZGV2aWNlIDAuMSBvbiBwY2kz Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaWIzOiA8QUNQSSBQQ0ktUENJIGJyaWRn ZT4gYXQgZGV2aWNlIDQuMCBvbiBwY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBj aTI6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWIzCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6 IHZnYXBjaTA6IDxWR0EtY29tcGF0aWJsZSBkaXNwbGF5PiBtZW0gMHhlMjAwMDAwMC0weGUyZmZm ZmZmLDB4ZDAwMDAwMDAtMHhkZmZmZmZmZiwweGU1MDAwMDAwLTB4ZTVmZmZmZmYgaXJxIDE2IGF0 IGRldmljZSA1LjAgb24gcGNpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiB2Z2FwY2kw OiBCb290IHZpZGVvIGRldmljZQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBwY2kwOiA8 bWVtb3J5LCBSQU0+IGF0IGRldmljZSA5LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogaXNhYjA6IDxQQ0ktSVNBIGJyaWRnZT4gYXQgZGV2aWNlIDEw LjAgb24gcGNpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBpc2EwOiA8SVNBIGJ1cz4g b24gaXNhYjAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogbmZzbWIwOiA8bkZvcmNlMi8z LzQgTUNQIFNNQnVzIENvbnRyb2xsZXI+IHBvcnQgMHgxYzAwLTB4MWMzZiwweDFjODAtMHgxY2Jm IGlycSAyMCBhdCBkZXZpY2UgMTAuMSBvbiBwY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJu ZWw6IHNtYnVzMDogPFN5c3RlbSBNYW5hZ2VtZW50IEJ1cz4gb24gbmZzbWIwCk9jdCAxNCAwNjo1 ODoxNyBncmVlbiBrZXJuZWw6IHNtYjA6IDxTTUJ1cyBnZW5lcmljIEkvTz4gb24gc21idXMwCk9j dCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IG5mc21iMTogPG5Gb3JjZTIvMy80IE1DUCBTTUJ1 cyBDb250cm9sbGVyPiBvbiBuZnNtYjAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogc21i dXMxOiA8U3lzdGVtIE1hbmFnZW1lbnQgQnVzPiBvbiBuZnNtYjEKT2N0IDE0IDA2OjU4OjE3IGdy ZWVuIGtlcm5lbDogc21iMTogPFNNQnVzIGdlbmVyaWMgSS9PPiBvbiBzbWJ1czEKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogcGNpMDogPG1lbW9yeSwgUkFNPiBhdCBkZXZpY2UgMTAuMiAo bm8gZHJpdmVyIGF0dGFjaGVkKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBvaGNpMDog PE9IQ0kgKGdlbmVyaWMpIFVTQiBjb250cm9sbGVyPiBtZW0gMHhlYjEwNjAwMC0weGViMTA2ZmZm IGlycSAyMSBhdCBkZXZpY2UgMTEuMCBvbiBwY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJu ZWw6IHVzYnVzMCBvbiBvaGNpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBlaGNpMDog PEVIQ0kgKGdlbmVyaWMpIFVTQiAyLjAgY29udHJvbGxlcj4gbWVtIDB4ZWIxMDcwMDAtMHhlYjEw NzBmZiBpcnEgMjIgYXQgZGV2aWNlIDExLjEgb24gcGNpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiB1c2J1czE6IEVIQ0kgdmVyc2lvbiAxLjAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtl cm5lbDogdXNidXMxIG9uIGVoY2kwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGF0YXBj aTI6IDxuVmlkaWEgbkZvcmNlIE1DUDUxIFVETUExMzMgY29udHJvbGxlcj4gcG9ydCAweDFmMC0w eDFmNywweDNmNiwweDE3MC0weDE3NywweDM3NiwweGYwMDAtMHhmMDBmIGF0IGRldmljZSAxMy4w IG9uIHBjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYXRhMDogPEFUQSBjaGFubmVs PiBhdCBjaGFubmVsIDAgb24gYXRhcGNpMgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBh dGExOiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhdGFwY2kyCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IGF0YXBjaTM6IDxuVmlkaWEgbkZvcmNlIE1DUDUxIFNBVEEzMDAgY29u dHJvbGxlcj4gcG9ydCAweDlmMC0weDlmNywweGJmMC0weGJmMywweDk3MC0weDk3NywweGI3MC0w eGI3MywweGQwMDAtMHhkMDBmIG1lbSAweGViMTA1MDAwLTB4ZWIxMDVmZmYgaXJxIDIzIGF0IGRl dmljZSAxNC4wIG9uIHBjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYXRhMjogPEFU QSBjaGFubmVsPiBhdCBjaGFubmVsIDAgb24gYXRhcGNpMwpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBhdGEzOiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhdGFwY2kzCk9jdCAx NCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGF0YXBjaTQ6IDxuVmlkaWEgbkZvcmNlIE1DUDUxIFNB VEEzMDAgY29udHJvbGxlcj4gcG9ydCAweDllMC0weDllNywweGJlMC0weGJlMywweDk2MC0weDk2 NywweGI2MC0weGI2MywweGJjMDAtMHhiYzBmIG1lbSAweGViMTA0MDAwLTB4ZWIxMDRmZmYgaXJx IDIwIGF0IGRldmljZSAxNS4wIG9uIHBjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDog YXRhNDogPEFUQSBjaGFubmVsPiBhdCBjaGFubmVsIDAgb24gYXRhcGNpNApPY3QgMTQgMDY6NTg6 MTcgZ3JlZW4ga2VybmVsOiBhdGE1OiA8QVRBIGNoYW5uZWw+IGF0IGNoYW5uZWwgMSBvbiBhdGFw Y2k0Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBjaWI0OiA8QUNQSSBQQ0ktUENJIGJy aWRnZT4gYXQgZGV2aWNlIDE2LjAgb24gcGNpMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVs OiBwY2kxOiA8QUNQSSBQQ0kgYnVzPiBvbiBwY2liNApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBhdGgwOiA8QXRoZXJvcyA1MjEyPiBtZW0gMHhlNDEwMDAwMC0weGU0MTBmZmZmIGlycSAx NiBhdCBkZXZpY2UgNi4wIG9uIHBjaTEKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYXRo MDogQVIyNDEzIG1hYyA3LjkgUkYyNDEzIHBoeSA0LjUKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtl cm5lbDogZnhwMDogPEludGVsIDgyNTU3IFByby8xMDAgRXRoZXJuZXQ+IHBvcnQgMHg0MDAwLTB4 NDAxZiBtZW0gMHhlYjAwMDAwMC0weGViMDAwZmZmLDB4ZTQwMDAwMDAtMHhlNDBmZmZmZiBpcnEg MTcgYXQgZGV2aWNlIDcuMCBvbiBwY2kxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGZ4 cDA6IEVuYWJsaW5nIFJ4IGxvY2stdXAgd29ya2Fyb3VuZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBtaWlidXMwOiA8TUlJIGJ1cz4gb24gZnhwMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBpbnBoeTA6IDxpODI1NTUgMTAvMTAwIG1lZGlhIGludGVyZmFjZT4gUEhZIDEgb24g bWlpYnVzMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBpbnBoeTA6ICAxMGJhc2VULCAx MGJhc2VULUZEWCwgMTAwYmFzZVRYLCAxMDBiYXNlVFgtRkRYLCBhdXRvCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IGZ4cDA6IEV0aGVybmV0IGFkZHJlc3M6IDAwOmEwOmM5OjVkOjQyOjk3 Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGZ3b2hjaTA6IDxUZXhhcyBJbnN0cnVtZW50 cyBUU0I0M0FCMjM+IG1lbSAweGU0MTE0MDAwLTB4ZTQxMTQ3ZmYsMHhlNDExMDAwMC0weGU0MTEz ZmZmIGlycSAxOCBhdCBkZXZpY2UgMTQuMCBvbiBwY2kxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBr ZXJuZWw6IGZ3b2hjaTA6IE9IQ0kgdmVyc2lvbiAxLjEwIChST009MSkKT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogZndvaGNpMDogTm8uIG9mIElzb2Nocm9ub3VzIGNoYW5uZWxzIGlzIDQu Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGZ3b2hjaTA6IEVVSTY0IDAwOjBmOmVhOjAw OjAwOjVhOmY4OjAxCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGZ3b2hjaTA6IFBoeSAx Mzk0YSBhdmFpbGFibGUgUzQwMCwgMyBwb3J0cy4KT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5l bDogZndvaGNpMDogTGluayBTNDAwLCBtYXhfcmVjIDIwNDggYnl0ZXMuCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IGZpcmV3aXJlMDogPElFRUUxMzk0KEZpcmVXaXJlKSBidXM+IG9uIGZ3 b2hjaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogZGNvbnNfY3JvbTA6IDxkY29ucyBj b25maWd1cmF0aW9uIFJPTT4gb24gZmlyZXdpcmUwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJu ZWw6IGRjb25zX2Nyb20wOiBidXNfYWRkciAweGJiZmMwMDAwCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGZ3b2hjaTA6IEluaXRpYXRlIGJ1cyByZXNldApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBmd29oY2kwOiBmd29oY2lfaW50cl9jb3JlOiBCVVMgcmVzZXQKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogZndvaGNpMDogZndvaGNpX2ludHJfY29yZTogbm9kZV9pZD0w eDAwMDAwMDAwLCBTZWxmSUQgQ291bnQ9MSwgQ1lDTEVNQVNURVIgbW9kZQpPY3QgMTQgMDY6NTg6 MTcgZ3JlZW4ga2VybmVsOiBoZGFjMDogPE5WSURJQSBNQ1A1MSBIREEgQ29udHJvbGxlcj4gbWVt IDB4ZWIxMDAwMDAtMHhlYjEwM2ZmZiBpcnEgMjEgYXQgZGV2aWNlIDE2LjEgb24gcGNpMApPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBuZmUwOiA8TlZJRElBIG5Gb3JjZSA0MzAgTUNQMTMg TmV0d29ya2luZyBBZGFwdGVyPiBwb3J0IDB4ZGMwMC0weGRjMDcgbWVtIDB4ZWIxMDgwMDAtMHhl YjEwOGZmZiBpcnEgMjIgYXQgZGV2aWNlIDIwLjAgb24gcGNpMApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBtaWlidXMxOiA8TUlJIGJ1cz4gb24gbmZlMApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBlMTAwMHBoeTA6IDxNYXJ2ZWxsIDg4RTExMTYgR2lnYWJpdCBQSFk+IFBIWSAx IG9uIG1paWJ1czEKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogZTEwMDBwaHkwOiAgbm9u ZSwgMTBiYXNlVCwgMTBiYXNlVC1GRFgsIDEwMGJhc2VUWCwgMTAwYmFzZVRYLUZEWCwgMTAwMGJh c2VULCAxMDAwYmFzZVQtbWFzdGVyLCAxMDAwYmFzZVQtRkRYLCAxMDAwYmFzZVQtRkRYLW1hc3Rl ciwgYXV0bywgYXV0by1mbG93Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IG5mZTA6IEV0 aGVybmV0IGFkZHJlc3M6IDAwOjBmOmVhOjVhOmQxOjBkCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBr ZXJuZWw6IGFtZHRlbXAwOiA8QU1EIENQVSBPbi1EaWUgVGhlcm1hbCBTZW5zb3JzPiBvbiBob3N0 YjMKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogdWFydDA6IDwxNjU1MCBvciBjb21wYXRp YmxlPiBwb3J0IDB4M2Y4LTB4M2ZmIGlycSA0IGZsYWdzIDB4MTAgb24gYWNwaTAKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogcHBjMDogPFBhcmFsbGVsIHBvcnQ+IHBvcnQgMHgzNzgtMHgz N2YgaXJxIDcgb24gYWNwaTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcHBjMDogR2Vu ZXJpYyBjaGlwc2V0IChOSUJCTEUtb25seSkgaW4gQ09NUEFUSUJMRSBtb2RlCk9jdCAxNCAwNjo1 ODoxNyBncmVlbiBrZXJuZWw6IHBwYnVzMDogPFBhcmFsbGVsIHBvcnQgYnVzPiBvbiBwcGMwCk9j dCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGxwdDA6IDxQcmludGVyPiBvbiBwcGJ1czAKT2N0 IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogbHB0MDogSW50ZXJydXB0LWRyaXZlbiBwb3J0Ck9j dCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBwaTA6IDxQYXJhbGxlbCBJL08+IG9uIHBwYnVz MApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBvcm0wOiA8SVNBIE9wdGlvbiBST01zPiBh dCBpb21lbSAweGQwMDAwLTB4ZDNmZmYsMHhkNDAwMC0weGQ2ZmZmLDB4ZDcwMDAtMHhkOWZmZiBv biBpc2EwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHNjMDogPFN5c3RlbSBjb25zb2xl PiBhdCBmbGFncyAweDEwMCBvbiBpc2EwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHNj MDogVkdBIDwxNiB2aXJ0dWFsIGNvbnNvbGVzLCBmbGFncz0weDMwMD4KT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogdmdhMDogPEdlbmVyaWMgSVNBIFZHQT4gYXQgcG9ydCAweDNjMC0weDNk ZiBpb21lbSAweGEwMDAwLTB4YmZmZmYgb24gaXNhMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBhdGtiZGMwOiA8S2V5Ym9hcmQgY29udHJvbGxlciAoaTgwNDIpPiBhdCBwb3J0IDB4NjAs MHg2NCBvbiBpc2EwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGF0a2JkMDogPEFUIEtl eWJvYXJkPiBpcnEgMSBvbiBhdGtiZGMwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGti ZDAgYXQgYXRrYmQwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGF0a2JkMDogW0dJQU5U LUxPQ0tFRF0KT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcG93ZXJub3cwOiA8UG93ZXJO b3chIEs4PiBvbiBjcHUwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHBvd2Vybm93MTog PFBvd2VyTm93ISBLOD4gb24gY3B1MQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBmaXJl d2lyZTA6IDEgbm9kZXMsIG1heGhvcCA8PSAwIGNhYmxlIElSTSBpcm0oMCkgIChtZSkgCk9jdCAx NCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGZpcmV3aXJlMDogYnVzIG1hbmFnZXIgMCAKT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogdXNidXMwOiAxMk1icHMgRnVsbCBTcGVlZCBVU0IgdjEu MApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBaRlMgZmlsZXN5c3RlbSB2ZXJzaW9uOiA1 Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IFpGUyBzdG9yYWdlIHBvb2wgdmVyc2lvbjog ZmVhdHVyZXMgc3VwcG9ydCAoNTAwMCkKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogVGlt ZWNvdW50ZXJzIHRpY2sgZXZlcnkgMS4wMDAgbXNlYwpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBoZGFjYzA6IDxSZWFsdGVrIEFMQzg4MyBIREEgQ09ERUM+IGF0IGNhZCAwIG9uIGhkYWMw Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGhkYWEwOiA8UmVhbHRlayBBTEM4ODMgQXVk aW8gRnVuY3Rpb24gR3JvdXA+IGF0IG5pZCAxIG9uIGhkYWNjMApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiBwY20wOiA8UmVhbHRlayBBTEM4ODMgKEFuYWxvZyA3LjEvMi4wKT4gYXQgbmlk IDIwLDIyLDIxLDIzIGFuZCAyNCwyNiwyOCBvbiBoZGFhMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBwY20xOiA8UmVhbHRlayBBTEM4ODMgKEZyb250IEFuYWxvZyk+IGF0IG5pZCAyNyBh bmQgMjUgb24gaGRhYTAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogcGNtMjogPFJlYWx0 ZWsgQUxDODgzIChSZWFyIERpZ2l0YWwpPiBhdCBuaWQgMzAgYW5kIDMxIG9uIGhkYWEwCk9jdCAx NCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHVzYnVzMTogNDgwTWJwcyBIaWdoIFNwZWVkIFVTQiB2 Mi4wCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHVnZW4wLjE6IDxuVmlkaWE+IGF0IHVz YnVzMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiB1aHViMDogPG5WaWRpYSBPSENJIHJv b3QgSFVCLCBjbGFzcyA5LzAsIHJldiAxLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNidXMwCk9jdCAx NCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IHVnZW4xLjE6IDxuVmlkaWE+IGF0IHVzYnVzMQpPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiB1aHViMTogPG5WaWRpYSBFSENJIHJvb3QgSFVCLCBj bGFzcyA5LzAsIHJldiAyLjAwLzEuMDAsIGFkZHIgMT4gb24gdXNidXMxCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IHVodWIwOiA4IHBvcnRzIHdpdGggOCByZW1vdmFibGUsIHNlbGYgcG93 ZXJlZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiB1aHViMTogOCBwb3J0cyB3aXRoIDgg cmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhj aWNoNzogUG9sbCB0aW1lb3V0IG9uIHNsb3QgMCBwb3J0IDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVu IGtlcm5lbDogYWhjaWNoNzogaXMgMDAwMDAwMDAgY3MgMDAwMDAwMDEgc3MgMDAwMDAwMDAgcnMg MDAwMDAwMDEgdGZkIDUwIHNlcnIgMDAwMDAwMDAgY21kIDEwMDAwMDE2Ck9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IChhcHJvYmUyOmFoY2ljaDc6MDowOjApOiBOT1AuIEFDQjogMDAgMDAg MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5l bDogKGFwcm9iZTI6YWhjaWNoNzowOjA6MCk6IENBTSBzdGF0dXM6IENvbW1hbmQgdGltZW91dApP Y3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlMjphaGNpY2g3OjA6MDowKTogRXJy b3IgNSwgUmV0cmllcyBleGhhdXN0ZWQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhj aWNoMDogVGltZW91dCBvbiBzbG90IDAgcG9ydCAwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJu ZWw6IGFoY2ljaDA6IGlzIDAwMDAwMDAxIGNzIDAwMDAwMDAwIHNzIDAwMDAwMDAwIHJzIDAwMDAw MDAxIHRmZCA1MCBzZXJyIDAwMDAwMDAwIGNtZCAxMDAwODAxNwpPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiAoYXByb2JlMDphaGNpY2gwOjA6MDowKTogQVRBX0lERU5USUZZLiBBQ0I6IGVj IDAwIDAwIDAwIDAwIDQwIDAwIDAwIDAwIDAwIDAwIDAwCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBr ZXJuZWw6IChhcHJvYmUwOmFoY2ljaDA6MDowOjApOiBDQU0gc3RhdHVzOiBDb21tYW5kIHRpbWVv dXQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTA6YWhjaWNoMDowOjA6MCk6 IEVycm9yIDUsIFJldHJ5IHdhcyBibG9ja2VkCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6 IGFoY2ljaDE6IFRpbWVvdXQgb24gc2xvdCAwIHBvcnQgMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBhaGNpY2gxOiBpcyAwMDAwMDAwMSBjcyAwMDAwMDAwMCBzcyAwMDAwMDAwMCBycyAw MDAwMDAwMSB0ZmQgNTAgc2VyciAwMDAwMDAwMCBjbWQgMTAwMDgwMTcKT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogKGFwcm9iZTE6YWhjaWNoMTowOjA6MCk6IEFUQV9JREVOVElGWS4gQUNC OiBlYyAwMCAwMCAwMCAwMCA0MCAwMCAwMCAwMCAwMCAwMCAwMApPY3QgMTQgMDY6NTg6MTcgZ3Jl ZW4ga2VybmVsOiAoYXByb2JlMTphaGNpY2gxOjA6MDowKTogQ0FNIHN0YXR1czogQ29tbWFuZCB0 aW1lb3V0Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUxOmFoY2ljaDE6MDow OjApOiBFcnJvciA1LCBSZXRyeSB3YXMgYmxvY2tlZApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBhaGNpY2gxNTogUG9sbCB0aW1lb3V0IG9uIHNsb3QgMCBwb3J0IDAKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoMTU6IGlzIDAwMDAwMDAwIGNzIDAwMDAwMDAxIHNzIDAw MDAwMDAwIHJzIDAwMDAwMDAxIHRmZCA1MCBzZXJyIDAwMDAwMDAwIGNtZCAxMDAwMDAxNgpPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlNDphaGNpY2gxNTowOjA6MCk6IE5PUC4g QUNCOiAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMApPY3QgMTQgMDY6NTg6MTcg Z3JlZW4ga2VybmVsOiAoYXByb2JlNDphaGNpY2gxNTowOjA6MCk6IENBTSBzdGF0dXM6IENvbW1h bmQgdGltZW91dApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlNDphaGNpY2gx NTowOjA6MCk6IEVycm9yIDUsIFJldHJpZXMgZXhoYXVzdGVkCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGFoY2ljaDg6IFRpbWVvdXQgb24gc2xvdCAwIHBvcnQgMApPY3QgMTQgMDY6NTg6 MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2g4OiBpcyAwMDAwMDAwMSBjcyAwMDAwMDAwMCBzcyAwMDAw MDAwMCBycyAwMDAwMDAwMSB0ZmQgNTAgc2VyciAwMDAwMDAwMCBjbWQgMTAwMDgwMTcKT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTI6YWhjaWNoODowOjA6MCk6IEFUQV9JREVO VElGWS4gQUNCOiBlYyAwMCAwMCAwMCAwMCA0MCAwMCAwMCAwMCAwMCAwMCAwMApPY3QgMTQgMDY6 NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlMjphaGNpY2g4OjA6MDowKTogQ0FNIHN0YXR1czog Q29tbWFuZCB0aW1lb3V0Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUyOmFo Y2ljaDg6MDowOjApOiBFcnJvciA1LCBSZXRyeSB3YXMgYmxvY2tlZApPY3QgMTQgMDY6NTg6MTcg Z3JlZW4ga2VybmVsOiBhaGNpY2g5OiBUaW1lb3V0IG9uIHNsb3QgMCBwb3J0IDAKT2N0IDE0IDA2 OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoOTogaXMgMDAwMDAwMDEgY3MgMDAwMDAwMDAgc3Mg MDAwMDAwMDAgcnMgMDAwMDAwMDEgdGZkIDUwIHNlcnIgMDAwMDAwMDAgY21kIDEwMDA4MDE3Ck9j dCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUzOmFoY2ljaDk6MDowOjApOiBBVEFf SURFTlRJRlkuIEFDQjogZWMgMDAgMDAgMDAgMDAgNDAgMDAgMDAgMDAgMDAgMDAgMDAKT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTM6YWhjaWNoOTowOjA6MCk6IENBTSBzdGF0 dXM6IENvbW1hbmQgdGltZW91dApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2Jl MzphaGNpY2g5OjA6MDowKTogRXJyb3IgNSwgUmV0cnkgd2FzIGJsb2NrZWQKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogcnVuX2ludGVycnVwdF9kcml2ZW5faG9va3M6IHN0aWxsIHdhaXRp bmcgYWZ0ZXIgNjAgc2Vjb25kcyBmb3IgeHB0X2NvbmZpZwpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBhaGNpY2gwOiBUaW1lb3V0IG9uIHNsb3QgMCBwb3J0IDAKT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogYWhjaWNoMDogaXMgMDAwMDAwMDEgY3MgMDAwMDAwMDAgc3MgMDAwMDAw MDAgcnMgMDAwMDAwMDEgdGZkIDUwIHNlcnIgMDAwMDAwMDAgY21kIDEwMDA4MDE3Ck9jdCAxNCAw Njo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUwOmFoY2ljaDA6MDowOjApOiBBVEFfSURFTlRJ RlkuIEFDQjogZWMgMDAgMDAgMDAgMDAgNDAgMDAgMDAgMDAgMDAgMDAgMDAKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTA6YWhjaWNoMDowOjA6MCk6IENBTSBzdGF0dXM6IENv bW1hbmQgdGltZW91dApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlMDphaGNp Y2gwOjA6MDowKTogRXJyb3IgNSwgUmV0cnkgd2FzIGJsb2NrZWQKT2N0IDE0IDA2OjU4OjE3IGdy ZWVuIGtlcm5lbDogYWhjaWNoMTogVGltZW91dCBvbiBzbG90IDAgcG9ydCAwCk9jdCAxNCAwNjo1 ODoxNyBncmVlbiBrZXJuZWw6IGFoY2ljaDE6IGlzIDAwMDAwMDAxIGNzIDAwMDAwMDAwIHNzIDAw MDAwMDAwIHJzIDAwMDAwMDAxIHRmZCA1MCBzZXJyIDAwMDAwMDAwIGNtZCAxMDAwODAxNwpPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlMTphaGNpY2gxOjA6MDowKTogQVRBX0lE RU5USUZZLiBBQ0I6IGVjIDAwIDAwIDAwIDAwIDQwIDAwIDAwIDAwIDAwIDAwIDAwCk9jdCAxNCAw Njo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUxOmFoY2ljaDE6MDowOjApOiBDQU0gc3RhdHVz OiBDb21tYW5kIHRpbWVvdXQKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTE6 YWhjaWNoMTowOjA6MCk6IEVycm9yIDUsIFJldHJ5IHdhcyBibG9ja2VkCk9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IGFoY2ljaDg6IFRpbWVvdXQgb24gc2xvdCAwIHBvcnQgMApPY3QgMTQg MDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2g4OiBpcyAwMDAwMDAwMSBjcyAwMDAwMDAwMCBz cyAwMDAwMDAwMCBycyAwMDAwMDAwMSB0ZmQgNTAgc2VyciAwMDAwMDAwMCBjbWQgMTAwMDgwMTcK T2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTI6YWhjaWNoODowOjA6MCk6IEFU QV9JREVOVElGWS4gQUNCOiBlYyAwMCAwMCAwMCAwMCA0MCAwMCAwMCAwMCAwMCAwMCAwMApPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAoYXByb2JlMjphaGNpY2g4OjA6MDowKTogQ0FNIHN0 YXR1czogQ29tbWFuZCB0aW1lb3V0Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJv YmUyOmFoY2ljaDg6MDowOjApOiBFcnJvciA1LCBSZXRyeSB3YXMgYmxvY2tlZApPY3QgMTQgMDY6 NTg6MTcgZ3JlZW4ga2VybmVsOiBhaGNpY2g5OiBUaW1lb3V0IG9uIHNsb3QgMCBwb3J0IDAKT2N0 IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWhjaWNoOTogaXMgMDAwMDAwMDEgY3MgMDAwMDAw MDAgc3MgMDAwMDAwMDAgcnMgMDAwMDAwMDEgdGZkIDUwIHNlcnIgMDAwMDAwMDAgY21kIDEwMDA4 MDE3Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IChhcHJvYmUzOmFoY2ljaDk6MDowOjAp OiBBVEFfSURFTlRJRlkuIEFDQjogZWMgMDAgMDAgMDAgMDAgNDAgMDAgMDAgMDAgMDAgMDAgMDAK T2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogKGFwcm9iZTM6YWhjaWNoOTowOjA6MCk6IENB TSBzdGF0dXM6IENvbW1hbmQgdGltZW91dApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiAo YXByb2JlMzphaGNpY2g5OjA6MDowKTogRXJyb3IgNSwgUmV0cnkgd2FzIGJsb2NrZWQKT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWRhMCBhdCBhdGEyIGJ1cyAwIHNjYnVzMTggdGFyZ2V0 IDAgbHVuIDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogY2QwIGF0IGF0YTAgYnVzIDAg c2NidXMxNiB0YXJnZXQgMCBsdW4gMApPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBjZDA6 IDwgQ1JXLTQ4MjRBIDEuMDA+IFJlbW92YWJsZSBDRC1ST00gU0NTSS0wIGRldmljZSAKT2N0IDE0 IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogY2QwOiAzMy4zMDBNQi9zIHRyYW5zZmVycyAoVURNQTIs IEFUQVBJIDEyYnl0ZXMsIFBJTyA2NTUzNGJ5dGVzKQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2Vy bmVsOiBjZDA6IEF0dGVtcHQgdG8gcXVlcnkgZGV2aWNlIHNpemUgZmFpbGVkOiBOT1QgUkVBRFks IE1lZGl1bSBub3QgcHJlc2VudCAtIHRyYXkgY2xvc2VkCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBr ZXJuZWw6IGFkYTA6IDxTVDEwMDBOQzAwMS0xRFkxNjIgQ04wMj4gQVRBLTggU0FUQSAzLnggZGV2 aWNlCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTA6IFNlcmlhbCBOdW1iZXIgWjFE QUFRTEYKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWRhMDogMzAwLjAwME1CL3MgdHJh bnNmZXJzIChTQVRBIDIueCwgVURNQTUsIFBJTyA4MTkyYnl0ZXMpCk9jdCAxNCAwNjo1ODoxNyBn cmVlbiBrZXJuZWw6IGFkYTA6IDk1Mzg2OE1CICgxOTUzNTIzMDU1IDUxMiBieXRlIHNlY3RvcnM6 IDE2SCA2M1MvVCAxNjM4M0MpCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTA6IFBy ZXZpb3VzbHkgd2FzIGtub3duIGFzIGFkMzYKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDog YWRhMSBhdCBhdGEzIGJ1cyAwIHNjYnVzMTkgdGFyZ2V0IDAgbHVuIDAKT2N0IDE0IDA2OjU4OjE3 IGdyZWVuIGtlcm5lbDogYWRhMTogPFdEQyBXRDIwRUFSWC0wMFBBU0IwIDUxLjBBQjUxPiBBVEEt OCBTQVRBIDMueCBkZXZpY2UKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWRhMTogU2Vy aWFsIE51bWJlciBXRC1XQ0FaQUU5OTg1MzkKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDog YWRhMTogMTUwLjAwME1CL3MgdHJhbnNmZXJzIChTQVRBIDEueCwgVURNQTUsIFBJTyA4MTkyYnl0 ZXMpCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTE6IDE5MDc3MjlNQiAoMzkwNzAy OTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAxNkggNjNTL1QgMTYzODNDKQpPY3QgMTQgMDY6NTg6MTcg Z3JlZW4ga2VybmVsOiBhZGExOiBxdWlya3M9MHgxPDRLPgpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4g a2VybmVsOiBhZGExOiBQcmV2aW91c2x5IHdhcyBrbm93biBhcyBhZDM4Ck9jdCAxNCAwNjo1ODox NyBncmVlbiBrZXJuZWw6IGFkYTIgYXQgYXRhNCBidXMgMCBzY2J1czIwIHRhcmdldCAwIGx1biAw Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTI6IDxTVDMxMDAwNTI0TlMgU04xMj4g QVRBLTggU0FUQSAyLnggZGV2aWNlCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTI6 IFNlcmlhbCBOdW1iZXIgOVdLNExKMzIKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWRh MjogMzAwLjAwME1CL3MgdHJhbnNmZXJzIChTQVRBIDIueCwgVURNQTUsIFBJTyA4MTkyYnl0ZXMp Ck9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTI6IDk1Mzg2OU1CICgxOTUzNTI1MTY4 IDUxMiBieXRlIHNlY3RvcnM6IDE2SCA2M1MvVCAxNjM4M0MpCk9jdCAxNCAwNjo1ODoxNyBncmVl biBrZXJuZWw6IGFkYTI6IFByZXZpb3VzbHkgd2FzIGtub3duIGFzIGFkNDAKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogYWRhMyBhdCBhdGE1IGJ1cyAwIHNjYnVzMjEgdGFyZ2V0IDAgbHVu IDAKT2N0IDE0IDA2OjU4OjE3IGdyZWVuIGtlcm5lbDogYWRhMzogPFdEQyBXRDIwRUFSUy0wME1W V0IwIDUxLjBBQjUxPiBBVEEtOCBTQVRBIDIueCBkZXZpY2UKT2N0IDE0IDA2OjU4OjE3IGdyZWVu IGtlcm5lbDogYWRhMzogU2VyaWFsIE51bWJlciBXRC1XTUFaQTMzMzAzMjIKT2N0IDE0IDA2OjU4 OjE3IGdyZWVuIGtlcm5lbDogYWRhMzogMzAwLjAwME1CL3MgdHJhbnNmZXJzIChTQVRBIDIueCwg VURNQTUsIFBJTyA4MTkyYnl0ZXMpCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IGFkYTM6 IDE5MDc3MjlNQiAoMzkwNzAyOTE2OCA1MTIgYnl0ZSBzZWN0b3JzOiAxNkggNjNTL1QgMTYzODND KQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhZGEzOiBxdWlya3M9MHgxPDRLPgpPY3Qg MTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBhZGEzOiBQcmV2aW91c2x5IHdhcyBrbm93biBhcyBh ZDQyCk9jdCAxNCAwNjo1ODoxNyBncmVlbiBrZXJuZWw6IFNNUDogQVAgQ1BVICMxIExhdW5jaGVk IQpPY3QgMTQgMDY6NTg6MTcgZ3JlZW4ga2VybmVsOiBUcnlpbmcgdG8gbW91bnQgcm9vdCBmcm9t IHpmczpncmVlbi9yb290IFtdLi4uCg== --001a11c38da01668dd05056181d1 Content-Type: application/octet-stream; name="pciconf-lv.out" Content-Disposition: attachment; filename="pciconf-lv.out" Content-Transfer-Encoding: base64 X-Attachment-Id: f_i1992ilw1 bm9uZTBAcGNpMDowOjA6MDoJY2xhc3M9MHgwNTAwMDAgY2FyZD0weDUwMDAxNDU4IGNoaXA9MHgw MmYxMTBkZSByZXY9MHhhMiBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9ICduVmlkaWEgQ29ycG9y YXRpb24nCiAgICBkZXZpY2UgICAgID0gJ0M1MSBIb3N0IEJyaWRnZScKICAgIGNsYXNzICAgICAg PSBtZW1vcnkKICAgIHN1YmNsYXNzICAgPSBSQU0Kbm9uZTFAcGNpMDowOjA6MToJY2xhc3M9MHgw NTAwMDAgY2FyZD0weDUwMDAxNDU4IGNoaXA9MHgwMmZhMTBkZSByZXY9MHhhMiBoZHI9MHgwMAog ICAgdmVuZG9yICAgICA9ICduVmlkaWEgQ29ycG9yYXRpb24nCiAgICBkZXZpY2UgICAgID0gJ0M1 MSBNZW1vcnkgQ29udHJvbGxlciAwJwogICAgY2xhc3MgICAgICA9IG1lbW9yeQogICAgc3ViY2xh c3MgICA9IFJBTQpub25lMkBwY2kwOjA6MDoyOgljbGFzcz0weDA1MDAwMCBjYXJkPTB4NTAwMDE0 NTggY2hpcD0weDAyZmUxMGRlIHJldj0weGEyIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25W aWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnQzUxIE1lbW9yeSBDb250cm9sbGVy IDEnCiAgICBjbGFzcyAgICAgID0gbWVtb3J5CiAgICBzdWJjbGFzcyAgID0gUkFNCm5vbmUzQHBj aTA6MDowOjM6CWNsYXNzPTB4MDUwMDAwIGNhcmQ9MHg1MDAwMTQ1OCBjaGlwPTB4MDJmODEwZGUg cmV2PTB4YTIgaGRyPTB4MDAKICAgIHZlbmRvciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwog ICAgZGV2aWNlICAgICA9ICdDNTEgTWVtb3J5IENvbnRyb2xsZXIgNScKICAgIGNsYXNzICAgICAg PSBtZW1vcnkKICAgIHN1YmNsYXNzICAgPSBSQU0Kbm9uZTRAcGNpMDowOjA6NDoJY2xhc3M9MHgw NTAwMDAgY2FyZD0weDUwMDAxNDU4IGNoaXA9MHgwMmY5MTBkZSByZXY9MHhhMiBoZHI9MHgwMAog ICAgdmVuZG9yICAgICA9ICduVmlkaWEgQ29ycG9yYXRpb24nCiAgICBkZXZpY2UgICAgID0gJ0M1 MSBNZW1vcnkgQ29udHJvbGxlciA0JwogICAgY2xhc3MgICAgICA9IG1lbW9yeQogICAgc3ViY2xh c3MgICA9IFJBTQpub25lNUBwY2kwOjA6MDo1OgljbGFzcz0weDA1MDAwMCBjYXJkPTB4NTAwMDE0 NTggY2hpcD0weDAyZmYxMGRlIHJldj0weGEyIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25W aWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnQzUxIEhvc3QgQnJpZGdlJwogICAg Y2xhc3MgICAgICA9IG1lbW9yeQogICAgc3ViY2xhc3MgICA9IFJBTQpub25lNkBwY2kwOjA6MDo2 OgljbGFzcz0weDA1MDAwMCBjYXJkPTB4NTAwMDE0NTggY2hpcD0weDAyN2YxMGRlIHJldj0weGEy IGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmlj ZSAgICAgPSAnQzUxIE1lbW9yeSBDb250cm9sbGVyIDMnCiAgICBjbGFzcyAgICAgID0gbWVtb3J5 CiAgICBzdWJjbGFzcyAgID0gUkFNCm5vbmU3QHBjaTA6MDowOjc6CWNsYXNzPTB4MDUwMDAwIGNh cmQ9MHg1MDAwMTQ1OCBjaGlwPTB4MDI3ZTEwZGUgcmV2PTB4YTIgaGRyPTB4MDAKICAgIHZlbmRv ciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9ICdDNTEgTWVtb3J5 IENvbnRyb2xsZXIgMicKICAgIGNsYXNzICAgICAgPSBtZW1vcnkKICAgIHN1YmNsYXNzICAgPSBS QU0KcGNpYjFAcGNpMDowOjI6MDoJY2xhc3M9MHgwNjA0MDAgY2FyZD0weDAwMDAxMGRlIGNoaXA9 MHgwMmZjMTBkZSByZXY9MHhhMSBoZHI9MHgwMQogICAgdmVuZG9yICAgICA9ICduVmlkaWEgQ29y cG9yYXRpb24nCiAgICBkZXZpY2UgICAgID0gJ0M1MSBQQ0kgRXhwcmVzcyBCcmlkZ2UnCiAgICBj bGFzcyAgICAgID0gYnJpZGdlCiAgICBzdWJjbGFzcyAgID0gUENJLVBDSQpwY2liMkBwY2kwOjA6 MzowOgljbGFzcz0weDA2MDQwMCBjYXJkPTB4MDAwMDEwZGUgY2hpcD0weDAyZmQxMGRlIHJldj0w eGExIGhkcj0weDAxCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRl dmljZSAgICAgPSAnQzUxIFBDSSBFeHByZXNzIEJyaWRnZScKICAgIGNsYXNzICAgICAgPSBicmlk Z2UKICAgIHN1YmNsYXNzICAgPSBQQ0ktUENJCnBjaWIzQHBjaTA6MDo0OjA6CWNsYXNzPTB4MDYw NDAwIGNhcmQ9MHgwMDAwMTBkZSBjaGlwPTB4MDJmYjEwZGUgcmV2PTB4YTEgaGRyPTB4MDEKICAg IHZlbmRvciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9ICdDNTEg UENJIEV4cHJlc3MgQnJpZGdlJwogICAgY2xhc3MgICAgICA9IGJyaWRnZQogICAgc3ViY2xhc3Mg ICA9IFBDSS1QQ0kKdmdhcGNpMEBwY2kwOjA6NTowOgljbGFzcz0weDAzMDAwMCBjYXJkPTB4ZDAw MDE0NTggY2hpcD0weDAyNDIxMGRlIHJldj0weGEyIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0g J25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnQzUxRyBbR2VGb3JjZSA2MTAw XScKICAgIGNsYXNzICAgICAgPSBkaXNwbGF5CiAgICBzdWJjbGFzcyAgID0gVkdBCm5vbmU4QHBj aTA6MDo5OjA6CWNsYXNzPTB4MDUwMDAwIGNhcmQ9MHg1MDAxMTQ1OCBjaGlwPTB4MDI3MDEwZGUg cmV2PTB4YTIgaGRyPTB4MDAKICAgIHZlbmRvciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwog ICAgZGV2aWNlICAgICA9ICdNQ1A1MSBIb3N0IEJyaWRnZScKICAgIGNsYXNzICAgICAgPSBtZW1v cnkKICAgIHN1YmNsYXNzICAgPSBSQU0KaXNhYjBAcGNpMDowOjEwOjA6CWNsYXNzPTB4MDYwMTAw IGNhcmQ9MHg1MDAxMTQ1OCBjaGlwPTB4MDI2MDEwZGUgcmV2PTB4YTMgaGRyPTB4MDAKICAgIHZl bmRvciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9ICdNQ1A1MSBM UEMgQnJpZGdlJwogICAgY2xhc3MgICAgICA9IGJyaWRnZQogICAgc3ViY2xhc3MgICA9IFBDSS1J U0EKbmZzbWIwQHBjaTA6MDoxMDoxOgljbGFzcz0weDBjMDUwMCBjYXJkPTB4MDI2NDE0NTggY2hp cD0weDAyNjQxMGRlIHJldj0weGEzIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBD b3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnTUNQNTEgU01CdXMnCiAgICBjbGFzcyAgICAg ID0gc2VyaWFsIGJ1cwogICAgc3ViY2xhc3MgICA9IFNNQnVzCm5vbmU5QHBjaTA6MDoxMDoyOglj bGFzcz0weDA1MDAwMCBjYXJkPTB4MDI2NDE0NTggY2hpcD0weDAyNzIxMGRlIHJldj0weGEzIGhk cj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAg ICAgPSAnTUNQNTEgTWVtb3J5IENvbnRyb2xsZXIgMCcKICAgIGNsYXNzICAgICAgPSBtZW1vcnkK ICAgIHN1YmNsYXNzICAgPSBSQU0Kb2hjaTBAcGNpMDowOjExOjA6CWNsYXNzPTB4MGMwMzEwIGNh cmQ9MHg1MDA0MTQ1OCBjaGlwPTB4MDI2ZDEwZGUgcmV2PTB4YTMgaGRyPTB4MDAKICAgIHZlbmRv ciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9ICdNQ1A1MSBVU0Ig Q29udHJvbGxlcicKICAgIGNsYXNzICAgICAgPSBzZXJpYWwgYnVzCiAgICBzdWJjbGFzcyAgID0g VVNCCmVoY2kwQHBjaTA6MDoxMToxOgljbGFzcz0weDBjMDMyMCBjYXJkPTB4NTAwNDE0NTggY2hp cD0weDAyNmUxMGRlIHJldj0weGEzIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBD b3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnTUNQNTEgVVNCIENvbnRyb2xsZXInCiAgICBj bGFzcyAgICAgID0gc2VyaWFsIGJ1cwogICAgc3ViY2xhc3MgICA9IFVTQgphdGFwY2k0QHBjaTA6 MDoxMzowOgljbGFzcz0weDAxMDE4YSBjYXJkPTB4YjAwMDE0NTggY2hpcD0weDAyNjUxMGRlIHJl dj0weGExIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAg IGRldmljZSAgICAgPSAnTUNQNTEgSURFJwogICAgY2xhc3MgICAgICA9IG1hc3Mgc3RvcmFnZQog ICAgc3ViY2xhc3MgICA9IEFUQQphdGFwY2k1QHBjaTA6MDoxNDowOgljbGFzcz0weDAxMDE4NSBj YXJkPTB4YjAwMjE0NTggY2hpcD0weDAyNjYxMGRlIHJldj0weGExIGhkcj0weDAwCiAgICB2ZW5k b3IgICAgID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnTUNQNTEgU2Vy aWFsIEFUQSBDb250cm9sbGVyJwogICAgY2xhc3MgICAgICA9IG1hc3Mgc3RvcmFnZQogICAgc3Vi Y2xhc3MgICA9IEFUQQphdGFwY2k2QHBjaTA6MDoxNTowOgljbGFzcz0weDAxMDE4NSBjYXJkPTB4 YjAwMjE0NTggY2hpcD0weDAyNjcxMGRlIHJldj0weGExIGhkcj0weDAwCiAgICB2ZW5kb3IgICAg ID0gJ25WaWRpYSBDb3Jwb3JhdGlvbicKICAgIGRldmljZSAgICAgPSAnTUNQNTEgU2VyaWFsIEFU QSBDb250cm9sbGVyJwogICAgY2xhc3MgICAgICA9IG1hc3Mgc3RvcmFnZQogICAgc3ViY2xhc3Mg ICA9IEFUQQpwY2liNEBwY2kwOjA6MTY6MDoJY2xhc3M9MHgwNjA0MDEgY2FyZD0weDAwMDAwMDAw IGNoaXA9MHgwMjZmMTBkZSByZXY9MHhhMiBoZHI9MHgwMQogICAgdmVuZG9yICAgICA9ICduVmlk aWEgQ29ycG9yYXRpb24nCiAgICBkZXZpY2UgICAgID0gJ01DUDUxIFBDSSBCcmlkZ2UnCiAgICBj bGFzcyAgICAgID0gYnJpZGdlCiAgICBzdWJjbGFzcyAgID0gUENJLVBDSQpoZGFjMEBwY2kwOjA6 MTY6MToJY2xhc3M9MHgwNDAzMDAgY2FyZD0weGEwMDIxNDU4IGNoaXA9MHgwMjZjMTBkZSByZXY9 MHhhMiBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9ICduVmlkaWEgQ29ycG9yYXRpb24nCiAgICBk ZXZpY2UgICAgID0gJ01DUDUxIEhpZ2ggRGVmaW5pdGlvbiBBdWRpbycKICAgIGNsYXNzICAgICAg PSBtdWx0aW1lZGlhCiAgICBzdWJjbGFzcyAgID0gSERBCm5mZTBAcGNpMDowOjIwOjA6CWNsYXNz PTB4MDY4MDAwIGNhcmQ9MHhlMDAwMTQ1OCBjaGlwPTB4MDI2OTEwZGUgcmV2PTB4YTMgaGRyPTB4 MDAKICAgIHZlbmRvciAgICAgPSAnblZpZGlhIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9 ICdNQ1A1MSBFdGhlcm5ldCBDb250cm9sbGVyJwogICAgY2xhc3MgICAgICA9IGJyaWRnZQpob3N0 YjBAcGNpMDowOjI0OjA6CWNsYXNzPTB4MDYwMDAwIGNhcmQ9MHgwMDAwMDAwMCBjaGlwPTB4MTEw MDEwMjIgcmV2PTB4MDAgaGRyPTB4MDAKICAgIHZlbmRvciAgICAgPSAnQWR2YW5jZWQgTWljcm8g RGV2aWNlcyBbQU1EXScKICAgIGRldmljZSAgICAgPSAnSzggW0F0aGxvbjY0L09wdGVyb25dIEh5 cGVyVHJhbnNwb3J0IFRlY2hub2xvZ3kgQ29uZmlndXJhdGlvbicKICAgIGNsYXNzICAgICAgPSBi cmlkZ2UKICAgIHN1YmNsYXNzICAgPSBIT1NULVBDSQpob3N0YjFAcGNpMDowOjI0OjE6CWNsYXNz PTB4MDYwMDAwIGNhcmQ9MHgwMDAwMDAwMCBjaGlwPTB4MTEwMTEwMjIgcmV2PTB4MDAgaGRyPTB4 MDAKICAgIHZlbmRvciAgICAgPSAnQWR2YW5jZWQgTWljcm8gRGV2aWNlcyBbQU1EXScKICAgIGRl dmljZSAgICAgPSAnSzggW0F0aGxvbjY0L09wdGVyb25dIEFkZHJlc3MgTWFwJwogICAgY2xhc3Mg ICAgICA9IGJyaWRnZQogICAgc3ViY2xhc3MgICA9IEhPU1QtUENJCmhvc3RiMkBwY2kwOjA6MjQ6 MjoJY2xhc3M9MHgwNjAwMDAgY2FyZD0weDAwMDAwMDAwIGNoaXA9MHgxMTAyMTAyMiByZXY9MHgw MCBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9ICdBZHZhbmNlZCBNaWNybyBEZXZpY2VzIFtBTURd JwogICAgZGV2aWNlICAgICA9ICdLOCBbQXRobG9uNjQvT3B0ZXJvbl0gRFJBTSBDb250cm9sbGVy JwogICAgY2xhc3MgICAgICA9IGJyaWRnZQogICAgc3ViY2xhc3MgICA9IEhPU1QtUENJCmhvc3Ri M0BwY2kwOjA6MjQ6MzoJY2xhc3M9MHgwNjAwMDAgY2FyZD0weDAwMDAwMDAwIGNoaXA9MHgxMTAz MTAyMiByZXY9MHgwMCBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9ICdBZHZhbmNlZCBNaWNybyBE ZXZpY2VzIFtBTURdJwogICAgZGV2aWNlICAgICA9ICdLOCBbQXRobG9uNjQvT3B0ZXJvbl0gTWlz Y2VsbGFuZW91cyBDb250cm9sJwogICAgY2xhc3MgICAgICA9IGJyaWRnZQogICAgc3ViY2xhc3Mg ICA9IEhPU1QtUENJCmF0YXBjaTBAcGNpMDo0OjA6MDoJY2xhc3M9MHgwMTA2MDEgY2FyZD0weDkx MjMxYjRiIGNoaXA9MHg5MTIzMWI0YiByZXY9MHgxMSBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9 ICdNYXJ2ZWxsIFRlY2hub2xvZ3kgR3JvdXAgTHRkLicKICAgIGRldmljZSAgICAgPSAnODhTRTkx MjMgUENJZSBTQVRBIDYuMCBHYi9zIGNvbnRyb2xsZXInCiAgICBjbGFzcyAgICAgID0gbWFzcyBz dG9yYWdlCiAgICBzdWJjbGFzcyAgID0gU0FUQQphdGFwY2kxQHBjaTA6NDowOjE6CWNsYXNzPTB4 MDEwMThmIGNhcmQ9MHg5MWE0MWI0YiBjaGlwPTB4OTFhNDFiNGIgcmV2PTB4MTEgaGRyPTB4MDAK ICAgIHZlbmRvciAgICAgPSAnTWFydmVsbCBUZWNobm9sb2d5IEdyb3VwIEx0ZC4nCiAgICBkZXZp Y2UgICAgID0gJzg4U0U5MUE0IFNBVEEgNkdiL3MgQ29udHJvbGxlcicKICAgIGNsYXNzICAgICAg PSBtYXNzIHN0b3JhZ2UKICAgIHN1YmNsYXNzICAgPSBBVEEKYXRhcGNpMkBwY2kwOjM6MDowOglj bGFzcz0weDAxMDYwMSBjYXJkPTB4OTEyMzFiNGIgY2hpcD0weDkxMjMxYjRiIHJldj0weDExIGhk cj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ01hcnZlbGwgVGVjaG5vbG9neSBHcm91cCBMdGQuJwog ICAgZGV2aWNlICAgICA9ICc4OFNFOTEyMyBQQ0llIFNBVEEgNi4wIEdiL3MgY29udHJvbGxlcicK ICAgIGNsYXNzICAgICAgPSBtYXNzIHN0b3JhZ2UKICAgIHN1YmNsYXNzICAgPSBTQVRBCmF0YXBj aTNAcGNpMDozOjA6MToJY2xhc3M9MHgwMTAxOGYgY2FyZD0weDkxYTQxYjRiIGNoaXA9MHg5MWE0 MWI0YiByZXY9MHgxMSBoZHI9MHgwMAogICAgdmVuZG9yICAgICA9ICdNYXJ2ZWxsIFRlY2hub2xv Z3kgR3JvdXAgTHRkLicKICAgIGRldmljZSAgICAgPSAnODhTRTkxQTQgU0FUQSA2R2IvcyBDb250 cm9sbGVyJwogICAgY2xhc3MgICAgICA9IG1hc3Mgc3RvcmFnZQogICAgc3ViY2xhc3MgICA9IEFU QQphdGgwQHBjaTA6MTo2OjA6CWNsYXNzPTB4MDIwMDAwIGNhcmQ9MHgzYTczMTE4NiBjaGlwPTB4 MDAxMzE2OGMgcmV2PTB4MDEgaGRyPTB4MDAKICAgIHZlbmRvciAgICAgPSAnQXRoZXJvcyBDb21t dW5pY2F0aW9ucyBJbmMuJwogICAgZGV2aWNlICAgICA9ICdBdGhlcm9zIEFSNTAwMVgrIFdpcmVs ZXNzIE5ldHdvcmsgQWRhcHRlcicKICAgIGNsYXNzICAgICAgPSBuZXR3b3JrCiAgICBzdWJjbGFz cyAgID0gZXRoZXJuZXQKZnhwMEBwY2kwOjE6NzowOgljbGFzcz0weDAyMDAwMCBjYXJkPTB4MDAw MTgwODYgY2hpcD0weDEyMjk4MDg2IHJldj0weDAyIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0g J0ludGVsIENvcnBvcmF0aW9uJwogICAgZGV2aWNlICAgICA9ICc4MjU1Ny84LzkvMC8xIEV0aGVy bmV0IFBybyAxMDAnCiAgICBjbGFzcyAgICAgID0gbmV0d29yawogICAgc3ViY2xhc3MgICA9IGV0 aGVybmV0CmZ3b2hjaTBAcGNpMDoxOjE0OjA6CWNsYXNzPTB4MGMwMDEwIGNhcmQ9MHgxMDAwMTQ1 OCBjaGlwPTB4ODAyNDEwNGMgcmV2PTB4MDAgaGRyPTB4MDAKICAgIHZlbmRvciAgICAgPSAnVGV4 YXMgSW5zdHJ1bWVudHMnCiAgICBkZXZpY2UgICAgID0gJ1RTQjQzQUIyMyBJRUVFLTEzOTRhLTIw MDAgQ29udHJvbGxlciAoUEhZL0xpbmspJwogICAgY2xhc3MgICAgICA9IHNlcmlhbCBidXMKICAg IHN1YmNsYXNzICAgPSBGaXJlV2lyZQo= --001a11c38da01668dd05056181d1-- From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 14:06:10 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AC68E79B for ; Tue, 14 Oct 2014 14:06:10 +0000 (UTC) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2a02:b90:3002:e550::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7647E169 for ; Tue, 14 Oct 2014 14:06:10 +0000 (UTC) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6] helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Xe2jn-000AK7-Pa for freebsd-stable@freebsd.org; Tue, 14 Oct 2014 14:06:07 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Xe2jn-0000vW-NP for freebsd-stable@freebsd.org; Tue, 14 Oct 2014 15:06:07 +0100 To: freebsd-stable@freebsd.org Subject: STMicroelectronics USB serial controller Message-Id: From: Pete French Date: Tue, 14 Oct 2014 15:06:07 +0100 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 14:06:10 -0000 Has anybody got any expereinec with these ? I am playing around with a small device (an scrypt ASIC) which presents itself to the OS as a serial port, using this chipset. When I plug it in I get this: ugen0.2: at usbus0 umodem0: on usbus0 umodem0: data interface 1, has no CM over data, has no break So it thinks it is a USB modem as far as I can make out ? I get a /dev/ttyU) device in /dev, but I do not think the device should present istelf as a modem. I have Linux software which is supposed to talk to the chip via a tty, and this does nothing when presented with the /dev/ttyU0 device. I also cannot get anything out of it. I havent dug very far into it yet, but was wondering if anyone had any ideas - this is the first time I've seen 'umodem' come up. cheers, -pete. From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 14:19:32 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 72BABBCF for ; Tue, 14 Oct 2014 14:19:32 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 472CD2AD for ; Tue, 14 Oct 2014 14:19:31 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1Xe2we-0003EJ-RF; Tue, 14 Oct 2014 14:19:24 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s9EEJNYr046874; Tue, 14 Oct 2014 08:19:23 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX18sG0cRTRE1lBV/6NygwkY5 X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: STMicroelectronics USB serial controller From: Ian Lepore To: Pete French In-Reply-To: References: Content-Type: text/plain; charset="us-ascii" Date: Tue, 14 Oct 2014 08:19:23 -0600 Message-ID: <1413296363.12052.384.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 14:19:32 -0000 On Tue, 2014-10-14 at 15:06 +0100, Pete French wrote: > Has anybody got any expereinec with these ? I am > playing around with a small device (an scrypt ASIC) > which presents itself to the OS as a serial port, using > this chipset. When I plug it in I get this: > > ugen0.2: at usbus0 > umodem0: on usbus0 > umodem0: data interface 1, has no CM over data, has no break > > So it thinks it is a USB modem as far as I can make out ? > I get a /dev/ttyU) device in /dev, but I do not think > the device should present istelf as a modem. I have > Linux software which is supposed to talk to > the chip via a tty, and this does nothing when > presented with the /dev/ttyU0 device. I also > cannot get anything out of it. > > I havent dug very far into it yet, but was wondering > if anyone had any ideas - this is the first time I've seen > 'umodem' come up. > > cheers, > > -pete. Try pointing that linux software at /dev/cuaU0 instead of ttyU0. The cua devices are "callout" and tty are "dialin" and the distinction is that the tty layer will block the open of a dialin tty until the modem carrier-detect is asserted. Since that isn't a real modem that's unlikely to happen, but it should always work to open the cua device. -- Ian From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 14:24:41 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25898D15; Tue, 14 Oct 2014 14:24:41 +0000 (UTC) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2a02:b90:3002:e550::3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E2566383; Tue, 14 Oct 2014 14:24:40 +0000 (UTC) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6] helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1Xe31i-000Bis-PF; Tue, 14 Oct 2014 14:24:38 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Xe31i-0001gW-N2; Tue, 14 Oct 2014 15:24:38 +0100 To: ian@FreeBSD.org, petefrench@ingresso.co.uk Subject: Re: STMicroelectronics USB serial controller In-Reply-To: <1413296363.12052.384.camel@revolution.hippie.lan> Message-Id: From: Pete French Date: Tue, 14 Oct 2014 15:24:38 +0100 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 14:24:41 -0000 > Try pointing that linux software at /dev/cuaU0 instead of ttyU0. The > cua devices are "callout" and tty are "dialin" and the distinction is > that the tty layer will block the open of a dialin tty until the modem > carrier-detect is asserted. Since that isn't a real modem that's > unlikely to happen, but it should always work to open the cua device. Gah! Yes, of course - I even knew that, from way back in the 80's - just didnt think to look for the 'cu' devices. Feel very stupid know. Works like a charm, thanks! -pete. From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 17:09:24 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8EE8CF7D for ; Tue, 14 Oct 2014 17:09:24 +0000 (UTC) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5EEF1900 for ; Tue, 14 Oct 2014 17:09:24 +0000 (UTC) Received: by mail-ig0-f173.google.com with SMTP id h18so15461617igc.0 for ; Tue, 14 Oct 2014 10:09:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=vGIZndRTCV60cOSS0oedvkPij5OjwKxM+AnLuHyiO4I=; b=tPPXMlie489FzsmBJfJCOUQIpnN1luSSeLs23oSl1osPYVf52CUbuX+SPOOEpLs4QV BiqkuCYjhnP96hUVot3xPFp+K11XJjCDsyaB1nWiKvFuC+Fi2AiphfxmuPfdqFEdxV2w YHULhNBPXiGPMvCpXFL91D/0pBUAQrFrKOnxcsOxAA5eCINqA469+5tvZRlXl9mTCFC6 hVnyuJnKSV1uJKVjG72gw6MH4Eu9cTR2JKZKzEfTKgdvF1p+PV7GbShUmAGzbxir/WIL kDKEs24CLF3fW3U/95P6uoq/pQ82BvoqiWOPgJOdxQd215NgqqUUWiaSmgKBkMANOshq rFEg== MIME-Version: 1.0 X-Received: by 10.50.128.137 with SMTP id no9mr8385980igb.0.1413306563601; Tue, 14 Oct 2014 10:09:23 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.11.141 with HTTP; Tue, 14 Oct 2014 10:09:23 -0700 (PDT) Date: Tue, 14 Oct 2014 10:09:23 -0700 X-Google-Sender-Auth: NG9hPRQFekVk-s83EXNIyjHNL0U Message-ID: Subject: System hang on shutdown when running freebsd-update From: Kevin Oberman To: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 17:09:24 -0000 I thought that this was just a fluke, but it has now happened three times, so I guess it's now out of the "fluke" class. I have upgraded several times recently to each 10.1 BETA and RC. After the first install pass t install the kernel and modules, the system shutdown freezes at the very end. I see the buffers synced to the disks and get the "All buffers synced" message. Then it just hangs. The disks are not marked as clean and are fscked after a reset and boot. There is not much between the "All buffers synced" message and the call to vfs_unmountall(), so I suspect it is hanging in that call. I admit that I am pretty much lost whenever I look at the VFS code and I have not put a lot of effort going further. Just hoping that someone familiar with it might have an idea. I have tried several reboots and all run normally. The problem only seems to appear when upgrading the OS. It happened repeatedly when I tried to reboot before doing the second "install" pass of freebsd-update, but not after, so the kernel and world are not in sync. I am baffled as to what could be going on, but it means I need to be at the system (a baby server) when I upgrade, but not every time I upgrade. I know it happened on the 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. Has anyone else seen this? The system is an Asus VivoPC VM40B-2. CPU: Intel(R) Celeron(R) CPU 1007U @ 1.50GHz (1496.63-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a Stepping =9 Features=0xbfebfbff Features2=0xdbae3bf AMD Features=0x28100800 AMD Features2=0x1 Structured Extended Features=0x281 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 4014346240 (3828 MB) R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 17:16:50 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D9CA326 for ; Tue, 14 Oct 2014 17:16:50 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 11D4AA07 for ; Tue, 14 Oct 2014 17:16:49 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 06AEE20E708FD; Tue, 14 Oct 2014 17:16:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 8AF4A20E7089C; Tue, 14 Oct 2014 17:16:46 +0000 (UTC) Message-ID: <9950D9D780BD4975ACEF9FA5D0F71CCC@multiplay.co.uk> From: "Steven Hartland" To: "Kevin Oberman" , "FreeBSD-STABLE Mailing List" References: Subject: Re: System hang on shutdown when running freebsd-update Date: Tue, 14 Oct 2014 18:16:44 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 17:16:50 -0000 ----- Original Message ----- From: "Kevin Oberman" To: "FreeBSD-STABLE Mailing List" Sent: Tuesday, October 14, 2014 6:09 PM Subject: System hang on shutdown when running freebsd-update >I thought that this was just a fluke, but it has now happened three times, > so I guess it's now out of the "fluke" class. > > I have upgraded several times recently to each 10.1 BETA and RC. After the > first install pass t install the kernel and modules, the system shutdown > freezes at the very end. I see the buffers synced to the disks and get the > "All buffers synced" message. Then it just hangs. The disks are not marked > as clean and are fscked after a reset and boot. > > There is not much between the "All buffers synced" message and the call to > vfs_unmountall(), so I suspect it is hanging in that call. I admit that I > am pretty much lost whenever I look at the VFS code and I have not put a > lot of effort going further. Just hoping that someone familiar with it > might have an idea. > > I have tried several reboots and all run normally. The problem only seems > to appear when upgrading the OS. It happened repeatedly when I tried to > reboot before doing the second "install" pass of freebsd-update, but not > after, so the kernel and world are not in sync. I am baffled as to what > could be going on, but it means I need to be at the system (a baby server) > when I upgrade, but not every time I upgrade. I know it happened on the > 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. > > Has anyone else seen this? > > The system is an Asus VivoPC VM40B-2. > CPU: Intel(R) Celeron(R) CPU 1007U @ 1.50GHz (1496.63-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a > Stepping =9 > > Features=0xbfebfbff > > Features2=0xdbae3bf > AMD Features=0x28100800 > AMD Features2=0x1 > Structured Extended Features=0x281 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID > TSC: P-state invariant, performance statistics > real memory = 4294967296 (4096 MB) > avail memory = 4014346240 (3828 MB) My usual question for this is does the following help especially if you have IPMI loaded at the time. sysctl hw.usb.no_shutdown_wait=1 Regards Steve From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 18:41:12 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35ADEB9F for ; Tue, 14 Oct 2014 18:41:12 +0000 (UTC) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 02AE35E2 for ; Tue, 14 Oct 2014 18:41:11 +0000 (UTC) Received: by mail-ig0-f169.google.com with SMTP id uq10so18382295igb.4 for ; Tue, 14 Oct 2014 11:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=BPuAXz3gnDIH0Z6vLvwKeokt5jUIkmLgPmbQBOO9Hm8=; b=gyMH51Z/BbRSvwvtqfjCOkC5TK2Vsl4Jt9n7+3JqZjy1Kfm6Ly2vX2xOZo5R8x7hm8 AkfXk0sI0lbVKCwCt1y9QAR8P9wSqsmMZXoj3/M1sLLFtBXj11fmX3uyZGxZmSVB75Bs xzqsGkpc+pgQJh4gz23Me+oolMrH2q2zKJUIGoJNXNG1qtxz2H/i6gJoPjopkWMn48A5 I5NxBi0Y3ekoKn4t5wdzXoje2RmStb3jmXt+WhyiB+0FiTEMg8tonqMQULgTICyxHhgm MeLfzG2v8ODNQMELJXvGDTglMbdvfM+m8Bdjn6PmrNfjtR7TU1P15QWZuNrGlKua36te pGYw== MIME-Version: 1.0 X-Received: by 10.43.75.138 with SMTP id za10mr6731327icb.23.1413312071316; Tue, 14 Oct 2014 11:41:11 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.11.141 with HTTP; Tue, 14 Oct 2014 11:41:11 -0700 (PDT) In-Reply-To: <201410141456.s9EEuror026763@mech-as221.men.bris.ac.uk> References: <201410141456.s9EEuror026763@mech-as221.men.bris.ac.uk> Date: Tue, 14 Oct 2014 11:41:11 -0700 X-Google-Sender-Auth: 269uYBzyy36DiDuuQ4yV7jUIt9M Message-ID: Subject: Re: vt in amd64/10.0-RELEASE-p9 #0? From: Kevin Oberman To: mexas@bris.ac.uk Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 18:41:12 -0000 On Tue, Oct 14, 2014 at 7:56 AM, Anton Shterenlikht wrote: > Regarding /usr/ports/UPDATING from 20141001 and 20141003, > I'm following instructions but cannot get vt. > I've set kern.vty=vt in /boot/loader.conf, > but on reboot I get: > > % sysctl kern.vty > sysctl: unknown oid 'kern.vty': No such file or directory > > and the switching from the graphicsl console to > the text does not work. I can type commands blindly, > just as described. > > https://wiki.freebsd.org/Newcons > says this should work > "in GENERIC on amd64/i386 as of r268045". > Perhaps my 10.0-RELEASE-p9 #0 is earlier than that? > > Please advise > > Thanks > > Anton > _______________________________________________ > freebsd-ports@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" > Not sure what this is doing in ports as it is a kernel issue. It belongs on stale. 10.0 does not include vt(4). It was added shortly after the 10.0 release. The various "-p" releases only contain security patches, not functional updates. To get vt(4), you will need to either update to a STABLE after the specified revision or update to 10.1-RC2. I'd recommend the latter as it can be fairly quickly accomplished, especially if you are running GENERIC. Or wait for 10.1-RELEASE which is PROBABLY not far off. freebsd-update upgrade -r 10.1-RC2 Go over the edits to etc (this can be a bit of a pain as the merge editing is just done with vi) freebsd-update install reboot (does not need to be to single user) freebsd-update install The upgrade will make several thousand patches, so it does take a while to run, but you can just leave it alone for the time it takes to run and your system is fully functional during that time. When it is finished, it prints out instructions for how to proceed. It may recommend re-installing all packages, but that is almost certainly not required as no system shareables have had versions updated. You will need to re-install any packages that provide kernel modules, though. -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 18:53:13 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6C58FD6 for ; Tue, 14 Oct 2014 18:53:13 +0000 (UTC) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A25C576D for ; Tue, 14 Oct 2014 18:53:13 +0000 (UTC) Received: by mail-ig0-f177.google.com with SMTP id a13so15862178igq.10 for ; Tue, 14 Oct 2014 11:53:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=7HzjvJTjE6LX6+W3i6PMPmLnMzL7ZM4IOjVyNx2jb0E=; b=M08yV0m33kAjEWFOgRWutd2egmEthqF4JF5EvcrUKzj//N5002zHK+Gs2S1wdNLhlh ve4EBotDP4YoarrgZ9zG7uGyK3/JvTGZoYnFTnTptFVKyeNSg85XvyNW42HOwKbs4pIL 1Rs8FxrgvrLwZrvekD8ORCvo/i4wtY7edhna2YcmvdSRWycEhg5Wa3CipaDVS3lbizdc hq8hpe1xSolpvuFJOzZfiCsgWDgzmA9iXgQE4Plcxtl/mJ5ojrgTacL1SzBCYI6Jn829 1BptDf39k1s5/5/qLngfu18GD6F+6U4qqPKMvmxkU5TDiWonPdA3X3z1Zi1nUJ0hoFHQ Rl6g== MIME-Version: 1.0 X-Received: by 10.50.122.106 with SMTP id lr10mr8872291igb.32.1413312792998; Tue, 14 Oct 2014 11:53:12 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.11.141 with HTTP; Tue, 14 Oct 2014 11:53:12 -0700 (PDT) In-Reply-To: <9950D9D780BD4975ACEF9FA5D0F71CCC@multiplay.co.uk> References: <9950D9D780BD4975ACEF9FA5D0F71CCC@multiplay.co.uk> Date: Tue, 14 Oct 2014 11:53:12 -0700 X-Google-Sender-Auth: TZncQrJct8dkmCJTt8GKWktkuWg Message-ID: Subject: Re: System hang on shutdown when running freebsd-update From: Kevin Oberman To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 18:53:14 -0000 On Tue, Oct 14, 2014 at 10:16 AM, Steven Hartland wrote: > > ----- Original Message ----- From: "Kevin Oberman" > To: "FreeBSD-STABLE Mailing List" > Sent: Tuesday, October 14, 2014 6:09 PM > Subject: System hang on shutdown when running freebsd-update > > > > I thought that this was just a fluke, but it has now happened three times, >> so I guess it's now out of the "fluke" class. >> >> I have upgraded several times recently to each 10.1 BETA and RC. After the >> first install pass t install the kernel and modules, the system shutdown >> freezes at the very end. I see the buffers synced to the disks and get the >> "All buffers synced" message. Then it just hangs. The disks are not marked >> as clean and are fscked after a reset and boot. >> >> There is not much between the "All buffers synced" message and the call to >> vfs_unmountall(), so I suspect it is hanging in that call. I admit that I >> am pretty much lost whenever I look at the VFS code and I have not put a >> lot of effort going further. Just hoping that someone familiar with it >> might have an idea. >> >> I have tried several reboots and all run normally. The problem only seems >> to appear when upgrading the OS. It happened repeatedly when I tried to >> reboot before doing the second "install" pass of freebsd-update, but not >> after, so the kernel and world are not in sync. I am baffled as to what >> could be going on, but it means I need to be at the system (a baby server) >> when I upgrade, but not every time I upgrade. I know it happened on the >> 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. >> >> Has anyone else seen this? >> >> The system is an Asus VivoPC VM40B-2. >> CPU: Intel(R) Celeron(R) CPU 1007U @ 1.50GHz (1496.63-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a >> Stepping =9 >> >> Features=0xbfebfbff> APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI, >> MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> >> Features2=0xdbae3bf> VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2, >> x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE> >> AMD Features=0x28100800 >> AMD Features2=0x1 >> Structured Extended Features=0x281 >> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID >> TSC: P-state invariant, performance statistics >> real memory = 4294967296 (4096 MB) >> avail memory = 4014346240 (3828 MB) >> > > My usual question for this is does the following help especially if you > have IPMI loaded at the time. > sysctl hw.usb.no_shutdown_wait=1 > > Regards > Steve > No IPMI. No USB disks. It's a little tiny Celeron system in a box with a 500GB disk that I use as a low-power server at home. It does use a USB keyboard and mouse, so, even though it seems unlikely, the sysctl might solve the issue and, since I don't have any other USB devices, I think I'll set the sysctl. Won't know if it's resolved until at least the next RC or maybe 10.1-RELEASE. I should have mentioned that the system is on a single UFS GPT partition originally installed from the 10.0-RELEASE memstick distro. I did need to update the re driver to the one is stable to get my network running, but it is now running straight 10.1-RC2 with no modifications. Thanks for the suggestion, Steve! -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 19:31:33 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 398AFAE6 for ; Tue, 14 Oct 2014 19:31:33 +0000 (UTC) Received: from mail-lb0-x22e.google.com (mail-lb0-x22e.google.com [IPv6:2a00:1450:4010:c04::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4C1FB7B for ; Tue, 14 Oct 2014 19:31:32 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id p9so8707385lbv.5 for ; Tue, 14 Oct 2014 12:31:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=W1hxLtRZkos6AxOMf+J2xJogJ79RzjXkWNjiGkGkvII=; b=QEJbk6+02rmO2KdGxz/jE2BcUnrirS0Y3UrBmQv/WgSL5tlDQpX84xlKnzPjYje+y+ LxwRrIauBxOIUp956WijBrABB1fuJ1mTvUo9lpoPTFwuscyuzjluIZ9pC8/eHL/dw7Mx xt5sf6I56MpLL7U7j8NGk5a5WoBfesu+XzivjoUrfbQwuJNzLSmz/xoJkGO/YP7T8ygN fec7N01pX2d2FR3HQZ6pgXGVa6kCRnrzOS6CmhJjywt4KwLXTXXQkFGgwIaowy1q0rlf lX0nOuZYBdGN17MbyfQK6RW9BSlOGEuQ1eK3w8dgbrIhmk4ZRQjiMbIfYhk4jk/MPPT2 tnDA== X-Received: by 10.152.202.135 with SMTP id ki7mr7616680lac.40.1413315090563; Tue, 14 Oct 2014 12:31:30 -0700 (PDT) MIME-Version: 1.0 Sender: royce.williams@gmail.com Received: by 10.112.171.73 with HTTP; Tue, 14 Oct 2014 12:31:10 -0700 (PDT) In-Reply-To: References: <9950D9D780BD4975ACEF9FA5D0F71CCC@multiplay.co.uk> From: Royce Williams Date: Tue, 14 Oct 2014 12:31:10 -0700 X-Google-Sender-Auth: 9WwWrIgbEW75Px-X6dsPhGWf1Co Message-ID: Subject: Re: System hang on shutdown when running freebsd-update To: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 19:31:33 -0000 On Tue, Oct 14, 2014 at 11:53 AM, Kevin Oberman wrote: > On Tue, Oct 14, 2014 at 10:16 AM, Steven Hartland > wrote: >> >> ----- Original Message ----- From: "Kevin Oberman" >> To: "FreeBSD-STABLE Mailing List" >> Sent: Tuesday, October 14, 2014 6:09 PM >> Subject: System hang on shutdown when running freebsd-update >> >> I thought that this was just a fluke, but it has now happened three times, >>> so I guess it's now out of the "fluke" class. >>> >>> I have upgraded several times recently to each 10.1 BETA and RC. After the >>> first install pass t install the kernel and modules, the system shutdown >>> freezes at the very end. I see the buffers synced to the disks and get the >>> "All buffers synced" message. Then it just hangs. The disks are not marked >>> as clean and are fscked after a reset and boot. >>> >>> There is not much between the "All buffers synced" message and the call to >>> vfs_unmountall(), so I suspect it is hanging in that call. I admit that I >>> am pretty much lost whenever I look at the VFS code and I have not put a >>> lot of effort going further. Just hoping that someone familiar with it >>> might have an idea. >>> >>> I have tried several reboots and all run normally. The problem only seems >>> to appear when upgrading the OS. It happened repeatedly when I tried to >>> reboot before doing the second "install" pass of freebsd-update, but not >>> after, so the kernel and world are not in sync. I am baffled as to what >>> could be going on, but it means I need to be at the system (a baby server) >>> when I upgrade, but not every time I upgrade. I know it happened on the >>> 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. >>> >>> Has anyone else seen this? >>> >>> The system is an Asus VivoPC VM40B-2. >>> CPU: Intel(R) Celeron(R) CPU 1007U @ 1.50GHz (1496.63-MHz K8-class CPU) >>> Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a >>> Stepping =9 >>> >>> Features=0xbfebfbff>> APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI, >>> MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >>> >>> Features2=0xdbae3bf>> VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2, >>> x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE> >>> AMD Features=0x28100800 >>> AMD Features2=0x1 >>> Structured Extended Features=0x281 >>> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID >>> TSC: P-state invariant, performance statistics >>> real memory = 4294967296 (4096 MB) >>> avail memory = 4014346240 (3828 MB) >>> >> >> My usual question for this is does the following help especially if you >> have IPMI loaded at the time. >> sysctl hw.usb.no_shutdown_wait=1 >> >> Regards >> Steve >> > > No IPMI. No USB disks. It's a little tiny Celeron system in a box with a > 500GB disk that I use as a low-power server at home. > > It does use a USB keyboard and mouse, so, even though it seems unlikely, > the sysctl might solve the issue and, since I don't have any other USB > devices, I think I'll set the sysctl. Won't know if it's resolved until at > least the next RC or maybe 10.1-RELEASE. > > I should have mentioned that the system is on a single UFS GPT partition > originally installed from the 10.0-RELEASE memstick distro. I did need to > update the re driver to the one is stable to get my network running, but it > is now running straight 10.1-RC2 with no modifications. This freebsd-update hang at "All buffers synced" also happens to me regularly as well - in Virtualbox. The most recent one was bumping from 10.0-RELEASE to 10.1-RC2. The virtual hardware specs are vanilla: obviously no IPMI, configured to emualte PIIX3 chipset, I/O APIC enabled. The only drive in the virtual system is set up as IDE PIIX4, and using the host's I/O cache. Royce From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 19:51:32 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EB6FBD0 for ; Tue, 14 Oct 2014 19:51:32 +0000 (UTC) Received: from smtp3-g21.free.fr (smtp3-g21.free.fr [212.27.42.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 927F2CC5 for ; Tue, 14 Oct 2014 19:51:31 +0000 (UTC) Received: from yggdrasil.alkumuna.eu (unknown [IPv6:2a01:e35:8a74:6e70:232:36ff:fe5c:3a87]) by smtp3-g21.free.fr (Postfix) with ESMTP id 5BE09A6348 for ; Tue, 14 Oct 2014 21:50:20 +0200 (CEST) Received: from freedom ([192.168.10.100]) (authenticated bits=0) by yggdrasil.alkumuna.eu (8.14.7/8.14.7) with ESMTP id s9EJpRYG075760 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NOT); Tue, 14 Oct 2014 21:51:28 +0200 (CEST) (envelope-from mazhe@alkumuna.eu) Date: Tue, 14 Oct 2014 21:51:22 +0200 From: Matthieu Volat Cc: freebsd-stable@freebsd.org Subject: Re: VT switch to console disabled once X starts Message-ID: <20141014215122.2a6e7243@freedom> In-Reply-To: References: <2553F6A9-0363-491B-9585-E12AD32CF0F6@alkumuna.eu> X-Mailer: Claws Mail 3.10.1 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/8vuO.1KU8wyOZpmp+ff02nT"; protocol="application/pgp-signature" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 19:51:33 -0000 --Sig_/8vuO.1KU8wyOZpmp+ff02nT Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Tue, 14 Oct 2014 12:24:02 +0200 "Ronald Klop" wrote: > [...] >=20 > People will like a little bit more information about your hardware and =20 > settings. > The output of 'dmesg' is helpfull for a start. > And the content of /var/log/Xorg.0.log. Sorry, a bit more info: /var/run/dmesg.boot Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.1-RC2 #0 r272876: Fri Oct 10 01:12:21 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 VT: running with driver "vga". CPU: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz (2666.72-MHz K8-class = CPU) Origin =3D "GenuineIntel" Id =3D 0x106a5 Family =3D 0x6 Model =3D 0x1a= Stepping =3D 5 Features=3D0xbfebfbff Features2=3D0x9ce3bd AMD Features=3D0x28100800 AMD Features2=3D0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,VPID TSC: P-state invariant, performance statistics real memory =3D 8592031744 (8194 MB) avail memory =3D 8236482560 (7854 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 ioapic0: Changing APIC ID to 1 ioapic1: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard random: initialized module_register_init: MOD_LOAD (vesa, 0xffffffff80d92410, 0) error 19 kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of fed40000, 12c1ffe (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, cff00000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet1: iomem 0xfed00000-0xfed003ff on acpi0 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf808-0xf80b on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 device_attach: hpet0 attach returned 12 pcib0: on acpi0 pci63: on pcib0 pcib1: port 0xcf8-0xcff on acpi0 pci0: on pcib1 pcib2: at device 1.0 on pci0 pci3: on pcib2 pcib3: at device 3.0 on pci0 pci15: on pcib3 vgapci0: port 0xe000-0xe07f mem 0xe2000000-0xe2fff= fff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 24 at device 0.0 on pci= 15 nvidia0: on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: child nvidia0 requested pci_enable_io vgapci0: Boot video device pcib4: at device 7.0 on pci0 pci40: on pcib4 pci0: at device 16.0 (no driver att= ached) pci0: at device 16.1 (no driver att= ached) pci0: at device 17.0 (no driver att= ached) pci0: at device 17.1 (no driver att= ached) pci0: at device 20.0 (no driver att= ached) pci0: at device 20.1 (no driver att= ached) pci0: at device 20.2 (no driver att= ached) uhci0: port 0xd000-0xd01f irq = 20 at device 26.0 on pci0 usbus0 on uhci0 uhci1: port 0xd020-0xd03f irq = 21 at device 26.1 on pci0 usbus1 on uhci1 uhci2: port 0xd040-0xd05f irq = 22 at device 26.2 on pci0 usbus2 on uhci2 ehci0: mem 0xe3204800-0xe3= 204bff irq 22 at device 26.7 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 hdac0: mem 0xe3200000-0xe3203fff irq 21 at d= evice 27.0 on pci0 pcib5: at device 28.0 on pci0 pci28: on pcib5 pcib6: irq 21 at device 28.5 on pci0 pci1: on pcib6 bge0: m= em 0xe3100000-0xe310ffff irq 17 at device 0.0 on pci1 bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E miibus0: on bge0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000ba= seT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: f4:ce:46:2d:aa:a6 uhci3: port 0xd060-0xd07f irq = 20 at device 29.0 on pci0 usbus4 on uhci3 uhci4: port 0xd080-0xd09f irq = 21 at device 29.1 on pci0 usbus5 on uhci4 uhci5: port 0xd0a0-0xd0bf irq = 22 at device 29.2 on pci0 usbus6 on uhci5 ehci1: mem 0xe3204c00-0xe3= 204fff irq 20 at device 29.7 on pci0 usbus7: EHCI version 1.0 usbus7 on ehci1 pcib7: at device 30.0 on pci0 pci55: on pcib7 pci55: at device 5.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xd100-0xd107,0xd110-0xd113,0= xd108-0xd10f,0xd114-0xd117,0xd0c0-0xd0df mem 0xe3204000-0xe32047ff irq 18 a= t device 31.2 on pci0 ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 acpi_button0: on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 qpi0: on motherboard orm0: at iomem 0xe9000-0xeffff on isa0 ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 random: unblocking device. usbus0: 12Mbps Full Speed USB v1.0 fuse-freebsd: version 0.4.4, FUSE ABI 7.8 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 21 and 24,25,26 on hdaa0 pcm1: at nid 27 on hdaa0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 ugen1.1: at usbus1 uhub0: on usbus0 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 ugen3.1: at usbus3 ugen4.1: at usbus4 uhub3: on usbus3 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8cd0 at ahcich1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device=20 cd0: Serial Number R3786GBSB3631300 cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - t= ray closed SATA 2.x device ada0: Serial Number 9VM9ERCY ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: Command Queueing enabled ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 1333358395 Hz quality 1000 Root mount waiting for: usbus7 usbus3 Root mount waiting for: usbus7 usbus3 uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Trying to mount root from zfs:zroot []... ugen6.2: at usbus6 ums0: o= n usbus6 ums0: 3 buttons and [XYZ] coordinates ID=3D0 /var/log/Xorg.0.log: [ 30.179]=20 X.Org X Server 1.12.4 Release Date: 2012-08-27 [ 30.179] X Protocol Version 11, Revision 0 [ 30.179] Build Operating System: FreeBSD 10.1-BETA3 amd64=20 [ 30.179] Current Operating System: FreeBSD ist-159-23 10.1-RC2 FreeBSD = 10.1-RC2 #0 r272876: Fri Oct 10 01:12:21 UTC 2014 root@releng1.nyi.free= bsd.org:/usr/obj/usr/src/sys/GENERIC amd64 [ 30.179] Build Date: 01 October 2014 01:19:43PM [ 30.179] =20 [ 30.179] Current version of pixman: 0.32.4 [ 30.179] Before reporting problems, check http://wiki.x.org to make sure that you have the latest version. [ 30.179] Markers: (--) probed, (**) from config file, (=3D=3D) default = setting, (++) from command line, (!!) notice, (II) informational, (WW) warning, (EE) error, (NI) not implemented, (??) unknown. [ 30.179] (=3D=3D) Log file: "/var/log/Xorg.0.log", Time: Tue Oct 14 11:= 33:00 2014 [ 30.221] (=3D=3D) Using config file: "/etc/X11/xorg.conf" [ 30.229] (=3D=3D) No Layout section. Using the first Screen section. [ 30.229] (**) |-->Screen "Screen0" (0) [ 30.229] (**) | |-->Monitor "Monitor0" [ 30.229] (**) | |-->Device "Device0" [ 30.229] (**) Option "DontZap" "false" [ 30.229] (=3D=3D) Not automatically adding devices [ 30.229] (=3D=3D) Not automatically enabling devices [ 30.368] (**) FontPath set to: /usr/local/lib/X11/fonts/TTF/, /usr/local/lib/X11/fonts/misc/, /usr/local/lib/X11/fonts/TTF/, /usr/local/lib/X11/fonts/OTF/, /usr/local/lib/X11/fonts/Type1/, /usr/local/lib/X11/fonts/100dpi/, /usr/local/lib/X11/fonts/75dpi/ [ 30.368] (=3D=3D) ModulePath set to "/usr/local/lib/xorg/modules" [ 30.368] (=3D=3D) |-->Input Device "Mouse0" [ 30.369] (=3D=3D) |-->Input Device "Keyboard0" [ 30.369] (=3D=3D) No Layout section. Using the first mouse device. [ 30.369] (=3D=3D) No Layout section. Using the first keyboard device. [ 30.369] (II) Loader magic: 0x7bf370 [ 30.369] (II) Module ABI versions: [ 30.369] X.Org ANSI C Emulation: 0.4 [ 30.369] X.Org Video Driver: 12.1 [ 30.369] X.Org XInput driver : 16.0 [ 30.369] X.Org Server Extension : 6.0 [ 30.369] (--) PCI:*(0:15:0:0) 10de:0659:10de:063a rev 161, Mem @ 0xe200= 0000/16777216, 0xd0000000/268435456, 0xe0000000/33554432, I/O @ 0x0000e000/= 128, BIOS @ 0x????????/65536 [ 30.369] (II) LoadModule: "extmod" [ 30.409] (II) Loading /usr/local/lib/xorg/modules/extensions/libextmod.= so [ 30.416] (II) Module extmod: vendor=3D"X.Org Foundation" [ 30.416] compiled for 1.12.4, module version =3D 1.0.0 [ 30.416] Module class: X.Org Server Extension [ 30.416] ABI class: X.Org Server Extension, version 6.0 [ 30.416] (II) Loading extension MIT-SCREEN-SAVER [ 30.416] (II) Loading extension XFree86-VidModeExtension [ 30.416] (II) Loading extension XFree86-DGA [ 30.416] (II) Loading extension DPMS [ 30.416] (II) Loading extension XVideo [ 30.416] (II) Loading extension XVideo-MotionCompensation [ 30.416] (II) Loading extension X-Resource [ 30.416] (II) LoadModule: "dbe" [ 30.416] (II) Loading /usr/local/lib/xorg/modules/extensions/libdbe.so [ 30.421] (II) Module dbe: vendor=3D"X.Org Foundation" [ 30.421] compiled for 1.12.4, module version =3D 1.0.0 [ 30.421] Module class: X.Org Server Extension [ 30.421] ABI class: X.Org Server Extension, version 6.0 [ 30.421] (II) Loading extension DOUBLE-BUFFER [ 30.421] (II) LoadModule: "glx" [ 30.422] (II) Loading /usr/local/lib/xorg/modules/extensions/libglx.so [ 31.136] (II) Module glx: vendor=3D"NVIDIA Corporation" [ 31.136] compiled for 4.0.2, module version =3D 1.0.0 [ 31.136] Module class: X.Org Server Extension [ 31.136] (II) NVIDIA GLX Module 331.67 Fri Apr 4 14:44:14 PDT 2014 [ 31.136] (II) Loading extension GLX [ 31.136] (II) LoadModule: "record" [ 31.136] (II) Loading /usr/local/lib/xorg/modules/extensions/librecord.= so [ 31.171] (II) Module record: vendor=3D"X.Org Foundation" [ 31.171] compiled for 1.12.4, module version =3D 1.13.0 [ 31.171] Module class: X.Org Server Extension [ 31.171] ABI class: X.Org Server Extension, version 6.0 [ 31.171] (II) Loading extension RECORD [ 31.171] (II) LoadModule: "dri" [ 31.172] (II) Loading /usr/local/lib/xorg/modules/extensions/libdri.so [ 31.197] (II) Module dri: vendor=3D"X.Org Foundation" [ 31.197] compiled for 1.12.4, module version =3D 1.0.0 [ 31.197] ABI class: X.Org Server Extension, version 6.0 [ 31.197] (II) Loading extension XFree86-DRI [ 31.197] (II) LoadModule: "dri2" [ 31.197] (II) Loading /usr/local/lib/xorg/modules/extensions/libdri2.so [ 31.197] (II) Module dri2: vendor=3D"X.Org Foundation" [ 31.198] compiled for 1.12.4, module version =3D 1.2.0 [ 31.198] ABI class: X.Org Server Extension, version 6.0 [ 31.198] (II) Loading extension DRI2 [ 31.198] (II) LoadModule: "nvidia" [ 31.228] (II) Loading /usr/local/lib/xorg/modules/drivers/nvidia_drv.so [ 31.309] (II) Module nvidia: vendor=3D"NVIDIA Corporation" [ 31.309] compiled for 4.0.2, module version =3D 1.0.0 [ 31.309] Module class: X.Org Video Driver [ 31.323] (II) LoadModule: "mouse" [ 31.323] (II) Loading /usr/local/lib/xorg/modules/input/mouse_drv.so [ 31.338] (II) Module mouse: vendor=3D"X.Org Foundation" [ 31.338] compiled for 1.12.4, module version =3D 1.9.0 [ 31.339] Module class: X.Org XInput Driver [ 31.339] ABI class: X.Org XInput driver, version 16.0 [ 31.339] (II) LoadModule: "kbd" [ 31.339] (II) Loading /usr/local/lib/xorg/modules/input/kbd_drv.so [ 31.346] (II) Module kbd: vendor=3D"X.Org Foundation" [ 31.346] compiled for 1.12.4, module version =3D 1.8.0 [ 31.346] Module class: X.Org XInput Driver [ 31.346] ABI class: X.Org XInput driver, version 16.0 [ 31.346] (II) NVIDIA dlloader X Driver 331.67 Fri Apr 4 14:26:21 PDT= 2014 [ 31.346] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs [ 31.346] (--) Using syscons driver with X support (version 2.0) [ 31.346] (--) using VT number 9 [ 31.359] (II) Loading sub module "fb" [ 31.359] (II) LoadModule: "fb" [ 31.359] (II) Loading /usr/local/lib/xorg/modules/libfb.so [ 31.380] (II) Module fb: vendor=3D"X.Org Foundation" [ 31.380] compiled for 1.12.4, module version =3D 1.0.0 [ 31.380] ABI class: X.Org ANSI C Emulation, version 0.4 [ 31.380] (II) Loading sub module "wfb" [ 31.380] (II) LoadModule: "wfb" [ 31.380] (II) Loading /usr/local/lib/xorg/modules/libwfb.so [ 31.390] (II) Module wfb: vendor=3D"X.Org Foundation" [ 31.390] compiled for 1.12.4, module version =3D 1.0.0 [ 31.390] ABI class: X.Org ANSI C Emulation, version 0.4 [ 31.390] (II) Loading sub module "ramdac" [ 31.390] (II) LoadModule: "ramdac" [ 31.390] (II) Module "ramdac" already built-in [ 31.391] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card su= pport [ 31.391] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32 [ 31.391] (=3D=3D) NVIDIA(0): RGB weight 888 [ 31.391] (=3D=3D) NVIDIA(0): Default visual is TrueColor [ 31.391] (=3D=3D) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) [ 31.392] (**) NVIDIA(0): Enabling 2D acceleration [ 32.844] (II) NVIDIA(0): NVIDIA GPU Quadro FX 580 (G96GL) at PCI:15:0:0= (GPU-0) [ 32.844] (--) NVIDIA(0): Memory: 524288 kBytes [ 32.844] (--) NVIDIA(0): VideoBIOS: 62.94.8b.00.05 [ 32.844] (II) NVIDIA(0): Detected PCI Express Link width: 16X [ 32.847] (--) NVIDIA(0): Valid display device(s) on Quadro FX 580 at PC= I:15:0:0 [ 32.847] (--) NVIDIA(0): CRT-0 [ 32.847] (--) NVIDIA(0): HP LA2205 (DFP-0) (boot, connected) [ 32.847] (--) NVIDIA(0): DFP-1 [ 32.847] (--) NVIDIA(0): DFP-2 [ 32.847] (--) NVIDIA(0): DFP-3 [ 32.847] (--) NVIDIA(0): DFP-4 [ 32.848] (--) NVIDIA(0): CRT-0: 400.0 MHz maximum pixel clock [ 32.848] (--) NVIDIA(0): HP LA2205 (DFP-0): Internal Dual Link TMDS [ 32.848] (--) NVIDIA(0): HP LA2205 (DFP-0): 330.0 MHz maximum pixel clo= ck [ 32.848] (--) NVIDIA(0): DFP-1: Internal Single Link TMDS [ 32.848] (--) NVIDIA(0): DFP-1: 165.0 MHz maximum pixel clock [ 32.848] (--) NVIDIA(0): DFP-2: Internal Single Link TMDS [ 32.848] (--) NVIDIA(0): DFP-2: 165.0 MHz maximum pixel clock [ 32.848] (--) NVIDIA(0): DFP-3: Internal DisplayPort [ 32.848] (--) NVIDIA(0): DFP-3: 480.0 MHz maximum pixel clock [ 32.848] (--) NVIDIA(0): DFP-4: Internal DisplayPort [ 32.848] (--) NVIDIA(0): DFP-4: 480.0 MHz maximum pixel clock [ 32.848] (**) NVIDIA(0): Using HorizSync/VertRefresh ranges from the ED= ID for display [ 32.848] (**) NVIDIA(0): device HP LA2205 (DFP-0) (Using EDID frequ= encies has been [ 32.848] (**) NVIDIA(0): enabled on all display devices.) [ 32.849] (=3D=3D) NVIDIA(0):=20 [ 32.849] (=3D=3D) NVIDIA(0): No modes were requested; the default mode = "nvidia-auto-select" [ 32.849] (=3D=3D) NVIDIA(0): will be used as the requested mode. [ 32.849] (=3D=3D) NVIDIA(0):=20 [ 32.849] (II) NVIDIA(0): Validated MetaModes: [ 32.849] (II) NVIDIA(0): "DFP-0:nvidia-auto-select" [ 32.849] (II) NVIDIA(0): Virtual screen size determined to be 1680 x 10= 50 [ 32.881] (--) NVIDIA(0): DPI set to (90, 88); computed from "UseEdidDpi= " X config [ 32.881] (--) NVIDIA(0): option [ 32.881] (WW) NVIDIA(0): UBB is incompatible with the Composite extensi= on. Disabling [ 32.881] (WW) NVIDIA(0): UBB. [ 32.881] (--) Depth 24 pixmap format is 32 bpp [ 32.881] (II) NVIDIA: Reserving 768.00 MB of virtual memory for indirec= t memory [ 32.881] (II) NVIDIA: access. [ 32.887] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select" [ 32.930] (II) Loading extension NV-GLX [ 32.945] (=3D=3D) NVIDIA(0): Disabling shared memory pixmaps [ 32.945] (=3D=3D) NVIDIA(0): Backing store disabled [ 32.945] (=3D=3D) NVIDIA(0): Silken mouse enabled [ 32.945] (**) NVIDIA(0): DPMS enabled [ 32.945] (II) Loading extension NV-CONTROL [ 32.946] (II) Loading extension XINERAMA [ 32.946] (II) Loading sub module "dri2" [ 32.946] (II) LoadModule: "dri2" [ 32.946] (II) Loading /usr/local/lib/xorg/modules/extensions/libdri2.so [ 32.946] (II) Module dri2: vendor=3D"X.Org Foundation" [ 32.946] compiled for 1.12.4, module version =3D 1.2.0 [ 32.946] ABI class: X.Org Server Extension, version 6.0 [ 32.946] (II) NVIDIA(0): [DRI2] Setup complete [ 32.946] (II) NVIDIA(0): [DRI2] VDPAU driver: nvidia [ 32.946] (--) RandR disabled [ 32.946] (II) Initializing built-in extension Generic Event Extension [ 32.946] (II) Initializing built-in extension SHAPE [ 32.946] (II) Initializing built-in extension MIT-SHM [ 32.946] (II) Initializing built-in extension XInputExtension [ 32.946] (II) Initializing built-in extension XTEST [ 32.946] (II) Initializing built-in extension BIG-REQUESTS [ 32.946] (II) Initializing built-in extension SYNC [ 32.946] (II) Initializing built-in extension XKEYBOARD [ 32.946] (II) Initializing built-in extension XC-MISC [ 32.946] (II) Initializing built-in extension XINERAMA [ 32.946] (II) Initializing built-in extension XFIXES [ 32.946] (II) Initializing built-in extension RENDER [ 32.946] (II) Initializing built-in extension RANDR [ 32.946] (II) Initializing built-in extension COMPOSITE [ 32.946] (II) Initializing built-in extension DAMAGE [ 32.949] (II) Initializing extension GLX [ 33.413] (II) Using input driver 'mouse' for 'Mouse0' [ 33.413] (**) Option "CorePointer" "on" [ 33.414] (**) Mouse0: always reports core events [ 33.414] (**) Option "Protocol" "auto" [ 33.414] (**) Option "Device" "/dev/sysmouse" [ 33.414] (**) Mouse0: Protocol: "auto" [ 33.414] (**) Mouse0: always reports core events [ 33.414] (=3D=3D) Mouse0: Emulate3Buttons, Emulate3Timeout: 50 [ 33.414] (**) Option "ZAxisMapping" "4 5 6 7" [ 33.414] (**) Mouse0: ZAxisMapping: buttons 4, 5, 6 and 7 [ 33.414] (**) Mouse0: Buttons: 7 [ 33.414] (II) XINPUT: Adding extended input device "Mouse0" (type: MOUS= E, id 6) [ 33.414] (**) Mouse0: (accel) keeping acceleration scheme 1 [ 33.414] (**) Mouse0: (accel) acceleration profile 0 [ 33.414] (**) Mouse0: (accel) acceleration factor: 2.000 [ 33.414] (**) Mouse0: (accel) acceleration threshold: 4 [ 33.414] (II) Mouse0: SetupAuto: hw.iftype is 4, hw.model is 0 [ 33.414] (II) Mouse0: SetupAuto: protocol is SysMouse [ 33.414] (II) Using input driver 'kbd' for 'Keyboard0' [ 33.414] (**) Option "CoreKeyboard" "on" [ 33.414] (**) Keyboard0: always reports core events [ 33.414] (**) Keyboard0: always reports core events [ 33.414] (**) Option "Protocol" "standard" [ 33.414] (**) Option "XkbRules" "base" [ 33.414] (**) Option "XkbModel" "pc105" [ 33.414] (**) Option "XkbLayout" "fr" [ 33.414] (**) Option "XkbOptions" "terminate:ctrl_alt_bksp" [ 33.414] (II) XINPUT: Adding extended input device "Keyboard0" (type: K= EYBOARD, id 7) [ 41.979] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select" This is new xorg without hal/devd support, here is xorg.conf: Section "ServerFlags" Option "DontZap" "false" EndSection Section "Files" FontPath "/usr/local/lib/X11/fonts/TTF/" EndSection Section "InputClass" Identifier "Keyboard Defaults" MatchIsKeyboard "yes" Option "XkbOptions" "terminate:ctrl_alt_bksp" Option "XkbLayout" "fr" EndSection Section "InputDevice" Identifier "Keyboard0" Driver "kbd" Option "XkbLayout" "fr" Option "XkbOptions" "terminate:ctrl_alt_bksp" EndSection Section "InputDevice" Identifier "Mouse0" Driver "mouse" Option "Protocol" "auto" Option "Device" "/dev/sysmouse" Option "ZAxisMapping" "4 5 6 7" EndSection Section "Monitor" Identifier "Monitor0" VendorName "HP" ModelName "HP Compaq LA2205wg" HorizSync 28.0 - 33.0 VertRefresh 43.0 - 72.0 Option "DPMS" EndSection Section "Device" Identifier "Device0" Driver "nvidia" EndSection Section "Screen" Identifier "Screen0" Device "Device0" Monitor "Monitor0" DefaultDepth 24 SubSection "Display" Viewport 0 0 Depth 8 EndSubSection SubSection "Display" Viewport 0 0 Depth 15 EndSubSection SubSection "Display" Viewport 0 0 Depth 16 EndSubSection SubSection "Display" Viewport 0 0 Depth 24 EndSubSection SubSection "Display" Viewport 0 0 Depth 32 EndSubSection EndSection Kernel modules? Id Refs Address Size Name 1 32 0xffffffff80200000 17533c0 kernel 2 1 0xffffffff81954000 267b48 zfs.ko 3 2 0xffffffff81bbc000 6778 opensolaris.ko 4 1 0xffffffff81bc3000 1a120 fuse.ko 5 1 0xffffffff81bde000 e58648 nvidia.ko 6 1 0xffffffff82c11000 52b2 fdescfs.ko 7 1 0xffffffff82c17000 9d2c linprocfs.ko 8 1 0xffffffff82c21000 43baa linux.ko 9 1 0xffffffff82c65000 357a ums.ko --=20 Matthieu Volat --Sig_/8vuO.1KU8wyOZpmp+ff02nT Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQ9fr8ACgkQ+ENDeYKZi37dgACgt3bOFy5RU+t/9vcJKJJPfQX/ NZUAniMGsMNnTKaza0ZLlL5JVeHNDKi3 =+LtI -----END PGP SIGNATURE----- --Sig_/8vuO.1KU8wyOZpmp+ff02nT-- From owner-freebsd-stable@FreeBSD.ORG Tue Oct 14 21:59:11 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7C12CED0; Tue, 14 Oct 2014 21:59:11 +0000 (UTC) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F8BEC83; Tue, 14 Oct 2014 21:59:10 +0000 (UTC) X-AuditID: 1209190f-f79aa6d000005b45-77-543d9ca69b7d Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id 60.B8.23365.6AC9D345; Tue, 14 Oct 2014 17:59:02 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id s9ELx1EE026774; Tue, 14 Oct 2014 17:59:02 -0400 Received: from localhost (mass-toolpike.mit.edu [18.9.64.17]) (authenticated bits=0) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s9ELx0Zp008940; Tue, 14 Oct 2014 17:59:01 -0400 Date: Tue, 14 Oct 2014 17:59:00 -0400 (EDT) From: Benjamin Kaduk X-X-Sender: kaduk@mass-toolpike.mit.edu To: freebsd-hackers@FreeBSD.org Subject: FreeBSD Quarterly Status Report - Third Quarter 2014 Message-ID: User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprAKsWRmVeSWpSXmKPExsUixCmqrbtsjm2Iwc1NJha7rp1mt5jz5gOT xfbN/xgtDjcLObB4zPg0nyWAMYrLJiU1J7MstUjfLoEro3nHRLaC/1+YK+ZO+8bcwHi9i7mL kZNDQsBEounLF0YIW0ziwr31bF2MXBxCArOZJHb9usQC4WxklPi1ZyoThPOJUWLnze1sIC0s AtoSM3vfgo1iE1CTWL/iGtRYdYnP8xeCjRURkJfY1/SeHcRmFrCWaFn5grWLkYNDWMBW4t6F dJAwr4C7xMyWX2DlogK6Eof+/WGDiAtKnJz5hAWi1V/i+bzN7BMY+WchSc1CkoKwdSSmTFzB CGFrS9y/2ca2gJFlFaNsSm6Vbm5iZk5xarJucXJiXl5qka6JXm5miV5qSukmRnDQSvLvYPx2 UOkQowAHoxIPb0GkTYgQa2JZcWXuIUZJDiYlUd7ZfbYhQnxJ+SmVGYnFGfFFpTmpxYcYJTiY lUR47ZOBcrwpiZVVqUX5MClpDhYlcd5NP/hChATSE0tSs1NTC1KLYLIyHBxKErx5s4EaBYtS 01Mr0jJzShDSTBycIMN5gIbHg9TwFhck5hZnpkPkTzHacxzreNnLxNHzGUR++AUiH+z+M4FJ iCUvPy9VSpy3DKRNAKQtozQPbjIsIb1iFAd6VJjXDKSKB5jM4Ga/AlrLBLR2YijY2pJEhJRU A6PPtK43un9vl1n/3fzbvCp0YWa47geFXY1KLMJbl53Pk7lx6Z2D5mFrXoMP1z9HXwhrvMta d3j1klmm2osZnWx9THeGdU7f+uFYrn3GxrCj2i8vey2tWNgiIrdqFYP4fsU01j0VZ5++YfJ9 yBMRwhPLGGHT1Rw177saj5yq18zLC9+e82bRMFRiKc5INNRiLipOBAD90mepIwMAAA== Content-Type: TEXT/PLAIN; charset=ISO-8859-2 Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-current@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Oct 2014 21:59:11 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 FreeBSD Project Quarterly Status Report: July - September 2014 This report covers FreeBSD-related projects between July and September 2014. This is the third of four reports planned for 2014. The third quarter of 2014 was another productive quarter for the FreeBSD project. A lot of work has been done on various ARM platforms, with the goal of bringing them to Tier 1 status in FreeBSD 11. The various ports teams have also worked hard to improve the state of FreeBSD as a desktop operating system. As usual, performance improvements feature in several places in this report and many of these can benefit from user benchmarking to validate our results. Thanks to all the reporters for the excellent work! The deadline for submissions covering the period from October to December 2014 is January 7th, 2015. __________________________________________________________________ FreeBSD Team Reports * FreeBSD Cluster Administration Team * FreeBSD Release Engineering Team * Ports Collection * The FreeBSD Core Team Projects * Address Space Layout Randomization (ASLR) * amd64 Xen Paravirtualization * bhyve * Chelsio iSCSI Offload Support * Debian GNU/kFreeBSD * FreeBSD Preseed Installation (PXE) * Jenkins Continuous Integration for FreeBSD * New Automounter * QEMU bsd-user-Enabled Ports Building * VMWare VAAI and Microsoft ODX Acceleration in CTL * ZFSguru Kernel * Intel GPU Driver Update * SDIO Driver * UEFI Boot * Updated vt(4) System Console * Updating OpenCrypto Architectures * FreeBSD on Newer ARM Boards * FreeBSD/arm64 Userland Programs * LLDB Debugger Port * LLVM Address Sanitizer (Asan) * SSE Variants of libc Routines for amd64 Ports * FreeBSD Python Ports * GNOME/FreeBSD * KDE on FreeBSD * The Graphics Stack on FreeBSD * Xfce Documentation * Handbook ezjail Section * Michael Lucas Books * ZFS Chapter of the Handbook Miscellaneous * The FreeBSD Foundation __________________________________________________________________ FreeBSD Cluster Administration Team Contact: FreeBSD Cluster Administration Team The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on: * Implemented a central, FreeBSD cluster-specific package building node using ports-mgmt/poudriere-devel, providing a single consistent set of third-party binary packages throughout the FreeBSD cluster from a common, known-working configuration. * Converted all machines running in the FreeBSD cluster from individual (and sometimes different) userland and kernel configurations to a single configuration for the base system. This enabled the implementation of a FreeBSD.org-specific binary update mechanism currently deployed throughout the cluster. This project is sponsored by The FreeBSD Foundation . __________________________________________________________________ FreeBSD Release Engineering Team URL: http://www.freebsd.org/releases/10.1R/schedule.html URL: http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things. In mid-July, FreeBSD 9.3-RELEASE was released without delay in release cycle. In late August, the FreeBSD 10.1-RELEASE cycle began, and as of this writing, is expected to stay on schedule. Work to produce virtual machine images as part of the release cycle has continued, supporting various cloud services such as Microsoft Azure, Amazon EC2, and Google Compute Engine. This project is sponsored by The FreeBSD Foundation . __________________________________________________________________ Ports Collection URL: http://www.FreeBSD.org/ports/ URL: http://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing-ports/ URL: http://portsmon.freebsd.org/index.html URL: http://portscout.freebsd.org/ URL: http://www.freebsd.org/portmgr/index.html URL: http://blogs.freebsdish.org/portmgr/ URL: http://www.twitter.com/freebsd_portmgr/ URL: http://www.facebook.com/portmgr URL: http://plus.google.com/communities/108335846196454338383 Contact: Frederic Culot Contact: Port Management Team As of the end of Q3, the ports tree holds a bit more than 24,000 ports, and the PR count is below 1,400. Despite the summer holidays the tree saw sustained activity with more than 9,000 commits and almost 2,000 ports PRs closed! In Q3, five new developers were granted a ports commit bit. None were taken in for safekeeping. On the management side, tabthorpe@ decided to step down from his portmgr duties in July. No other changes were made to the team during Q3. This quarter also saw the release of the third quarterly branch, namely 2014Q3. On the QA side, 34 exp-runs were performed to validate sensitive updates or cleanups. Last, the 20th anniversary of the ports tree was commemorated during Q3 and a video was published for this event. Open tasks: 1. Tremendous work was done on the PR front in Q3 and we would be very pleased to see committers dedicate themselves to closing as many as possible in Q4 as well. __________________________________________________________________ The FreeBSD Core Team Contact: FreeBSD Core Team The FreeBSD Core Team constitutes the project's "Board of Directors", responsible for deciding the project's overall goals and direction as well as managing specific areas of the FreeBSD project landscape. The third quarter of this year was a relatively quiet time in terms of Core Team activity. No major policy changes were decided, but the criterea for awarding commit bits were reviewed and the existing requirements were clarified and documented at http://www.freebsd.org/interna=B3proposing-committers.html. Other items dealt with by core during this period: * Confirmed with Microsoft that it is permissible to include DCTCP in FreeBSD. * Promoted git-beta.freebsd.org out of beta-test to an official service, consequently renaming it to git.freebsd.org. * Mediated between groups of contributors and committers adopting different positions on the on-going work to introduce ASLR and associated technologies. * Worked with the FreeBSD Foundation to obtain a license for XenForo, and with the forum administrators on plans to migrate the FreeBSD forums onto the FreeBSD cluster and to switch to XenForo. During this period, three commit bits were granted, and two commit bits were taken in for safe keeping. __________________________________________________________________ Address Space Layout Randomization (ASLR) URL: http://hardenedbsd.org/ URL: https://reviews.freebsd.org/D473 URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D193940 URL: https://wiki.freebsd.org/201409DevSummit/ASLR URL: https://wiki.freebsd.org/AddressSpaceLayoutRandomization Contact: Shawn Webb Contact: Oliver Pinter Address Space Layout Randomization (ASLR) is a computer security technique that aids in mitigating low-level vulnerabilities such as buffer overflows. ASLR randomizes the memory layout of running applications, to prevent an attacker from knowing where a given exploitable vulnerability lies in memory. A lot has happened in the last few months. Shawn Webb gave presentations at both BSDCan 2014 and EuroBSDCon 2014. The presentations were met with a lot of support and backing. At the end of EuroBSDCon Ilya Bakulin fixed the known bug with ASLR on ARM systems. Shawn Webb and Oliver Pinter have submitted our patch to the Phabricator code review system. Shawn Webb added an API for allowing a debugger to disable ASLR to support deterministic debugging. Oliver Pinter enhanced the performance of our ASLR implementation. A package building exp-run was ran and came out favorably in terms of performance. Shawn Webb bumped up the maximum number of bits allowed to be randomized to 20 and set the default to 14. Shawn Webb and Oliver Pinter founded The HardenedBSD project to serve as a staging area for their work on security-related projects for FreeBSD. This project is sponsored by SoldierX. Open tasks: 1. Get more people testing and reviewing our patch 2. Run more performance tests 3. Figure out why the two ports failed in the EXP-RUN. Involve the port maintainers. 4. Test on different architectures (we need help with this) __________________________________________________________________ amd64 Xen Paravirtualization URL: https://wiki.freebsd.org/FreeBSD/Xen Contact: Cherry Mathew This project aims to add support to the FreeBSD kernel for running in Xen Paravirtualised mode on amd64 systems. This project has finally reached a "Proof of Concept" stage on the branch projects/amd64_xen_pv. Testing and bug reports on various configurations is encouraged! The author is also seeking bounties to help complete the effort and assess potential interest. Please send email if interested. PV kernels are still supported by most cloud providers for a range of configurations, although they are expected to be phased out in the future. This project is sponsored by Spectralogic Corporation (2012-2013) . Open tasks: 1. Large page support 2. SMP support 3. Debug and cleanup 4. Security vetting 5. Performance tweaks __________________________________________________________________ bhyve URL: http://www.bhyve.org URL: http://www.youtube.com/watch?v=3DlTOiSyu0-MA Contact: Peter Grehan Contact: Neel Natu Contact: John Baldwin Contact: Tycho Nightingale Contact: Allan Jude bhyve is a hypervisor that runs on the FreeBSD/amd64 platform. At present, it runs FreeBSD (8.x or later), Linux i386/x64, OpenBSD i386/amd64, and NetBSD/amd64 guests. Current development is focused on enabling additional guest operating systems and implementing features found in other hypervisors. A significant amount of progress has been made since the last status report. Most importantly, all of this work has been MFCed to the 10-STABLE branch and will be included in the 10.1 release. Support for AMD processors is being developed in the bhyve_svm SVN project branch. The branch is almost at feature-parity with mainline Intel VT-x support, and will be committed into -CURRENT in the near future. New features added this quarter: * Guest support for recent Linux i386/x64, OpenBSD i386/amd64, and NetBSD amd64. * Force guest reset and poweroff with bhyvectl * Allow the SMBIOS UUID to be set from the command line * PCI MMIO extended config space access * Improved AHCI error handling, legacy interrupt mode * Additional instruction emulation required by a number of guests * Legacy x86 task switching to support double-faults in FreeBSD/i386 * Legacy PCI interrupts, operation without an APIC (OpenBSD install) * Guest memory not included by default in core dumps * Allow guest vCPUs to be pinned to individual host CPUs * Virtio RNG device emulation * Chapter about bhyve added to FreeBSD Handbook Open tasks: 1. Improve documentation 2. CSM BIOS boot support for non UEFI-aware guests 3. Add support for virtio-scsi 4. Improve virtio-net, add offload features, support multiple queues 5. Implement Intel 82580 and e1000 NIC emulation 6. Netmap support 7. Flexible networking backend: wanproxy, vhost-net 8. Move to a single process model, instead of bhyveload and bhyve 9. Support running bhyve as non-root 10. Add filters for popular VM file formats (VMDK, VHD, QCOW2) 11. Implement an abstraction layer for video (no X11 or SDL in base system) 12. Support for VNC as a video output 13. Suspend/resume support 14. Live Migration 15. Nested VT-x support (bhyve in bhyve) 16. Support for other architectures (ARM, MIPS, PPC) __________________________________________________________________ Chelsio iSCSI Offload Support Contact: Sreenivasa Honnur Contact: Edward Tomasz Napiera=B3a Building on the new in-kernel iSCSI target and initiator stack released in FreeBSD 10.0, Chelsio Communications has begun developing an offload interface to take advantage of the hardware offload capabilities of Chelsio T4 and T5 10 and 40 gigabit Ethernet adapters. The code implements hardware PDU offload for both target and initiator. The iSCSI stack has been modified to provide a hardware-independent offload API, allowing offload drivers to be loaded as kernel modules, and to provide mechanisms for the system administrator to configure this feature. The project is entering a testing phase. The code will be released under the BSD license and is expected to be completed later in the year and ship in FreeBSD 10.2-RELEASE. This project is sponsored by Chelsio Communications , and The FreeBSD Foundation . Open tasks: 1. Complete testing __________________________________________________________________ Debian GNU/kFreeBSD URL: https://wiki.debian.org/Debian_GNU/kFreeBSD URL: https://www.debian.org/ports/kfreebsd-gnu/ Contact: Debian GNU/kFreeBSD Maintainers Debian GNU/kFreeBSD is a software distribution produced by Debian, based on the kernel of FreeBSD (instead of Linux) and GNU libc. Around 90% of Debian's software archive has now been ported to it, for amd64 and i386 architectures. It was first released with Debian "squeeze" as a development preview in 2011, featured again in the "wheezy" release, and hopes to be part of the official Debian "jessie" release in early 2015. In 2003 there were several attempts to bootstrap a minimal Debian system upon FreeBSD or NetBSD kernels, some also trying to use the native BSD libc. The most successful and longest-lived of these was a "GNU/FreeBSD" chroot bootstrapped by Robert Millan with the GNU libc that most of Debian's core packages were designed to work with. The "k" was later added to the name to reflect that it takes just the kernel from FreeBSD, with most everything else from the Debian archive. We do also package some FreeBSD utilities as needed to boot it and take advantage of certain features. FreeBSD support within GNU libc is now mostly maintained by Petr Salinger, who recently converted it from an older threading implementation based on LinuxThreads to NPTL, which is much more compatible with the software we run. We have the GNU compiler toolchain as well as Clang 3.4; Perl, Python and Ruby; and OpenJDK 7, based the on work done in FreeBSD's own ports collection. We use linprocfs for /proc because much of Debian GNU software expects this. The Linuxulator is not needed at all, but could make for interesting future uses. Porting work mostly focuses now on individual packages' build systems, on preprocessor #ifdefs that do not clearly distinguish between kernel and libc, or fixing testsuites' presumptions of Linux-specific behaviour. In the course of this, we even found the odd FreeBSD kernel bug, including EN-14:06 / CVE-2014-3880. GNU/kFreeBSD has already seen production use, mostly on webservers, email servers and file servers; one such machine has 475 days' uptime receiving around 10,000 emails per day. It has become increasingly practical for desktop/laptop uses thanks largely to new features coming in from FreeBSD 10.1. KMS graphics mean that 3D gaming and high-definition video playback perform brilliantly. We have great support for Intel graphics chipsets, but only an older nvidia Xorg driver. For radeonkms, Robert Millan was able to add firmware-loading support so that non-free binary blobs can be packaged separately, outside of Debian's main archive. Proprietary drivers are not useful to us as they would need to be rebuilt from source to port them. vt(4) was necessary for KMS to not break VT switching. But it has also improved the console's handling of non-ASCII character sets and we do look forward to having console fonts for non-Latin scripts. We have supported ZFS for some time, even as a root/boot filesystem (using GRUB 2; Robert Millan added the ZFS support which now FreeBSD itself is able to benefit from). Enhancements coming from OpenZFS, especially LZ4 compression, in combination with better memory management and GEOM improvements, mean that "jessie" should see a noticeable performance boost. debian-installer already allows for pre-seeded, unattended installs and there are PXE-bootable install images available. virtio drivers are new to the "jessie" release, enabling support for some public clouds. We are now compiling Xen domU and PVHVM support into our standard kernel builds. We already have userland tools to configure the PF firewall. As an experiment, we are compiling in IPSEC support by default for the upcoming release, and would like to see it put to good use against present-day privacy and security threats. We try to support the use of Debian GNU/kFreeBSD inside a jail on a FreeBSD host system, and hopefully vice-versa. Some of the jail utilities are not yet packaged, but we have documentation on the Debian Wiki on how to set up jails on "wheezy", which are fully functional. The init system we currently use is a parallel System V-style init, although Debian GNU/Linux will be switching away from that to systemd. For the next release we may switch to OpenRC, which is mostly ported already. Not having systemd or udev means that we will be unable to support GNOME 3.14 in the upcoming release. We have very good support for XFCE, also have KDE, LXDE and the recently-packaged MATE desktop environment. The Debian software archive provides many alternative window managers for Xorg such as IceWM, dozens of terminal emulators, and so on. As we approach the freeze of the Debian "jessie" release, we would love for anyone to test GNU/kFreeBSD, try to use it for whatever would be useful to you, and let us know what issues you run into. Ask for help on our project mailing list or IRC channel, and let us know of any bugs you find. We still have time to fix problems before release, and we would be happy to improve our documentation at any time. __________________________________________________________________ FreeBSD Preseed Installation (PXE) URL: https://wiki.freebsd.org/SummerOfCode2014/FreeBSD_PXE_preseed Contact: Kamil Czekirda This is a Google Summer of code project to provide a noninteractive FreeBSD installation process from the network. In the first part, an implementation was added for scripted bsdinstall(8). It supports variables such as KEYMAP, HOSTNAME, MIRROR, RELEASE, TIMEZONE, DAEMONS, ROOTPWHASH, and USERS. Network configuration, ZFS options, and others are also included. The second part of the project is about booting the fai (Fully Automatic Installer) from the network by PXE. An installer distro was created based on mfsBSD. After boot, fai looks for the "bootfile-name" parameter from the DHCP server. This parameter tells fai where the bsdinstall script is located. fai supports MAC-based configuration, or a default if a MAC-based configuration file does not exist. This project is sponsored by Google Summer of Code 2014 . Open tasks: 1. Documentation, including a HOWTO and handbook 2. More tests in different configurations 3. Support for more than one network interface is planned __________________________________________________________________ Jenkins Continuous Integration for FreeBSD URL: https://jenkins.freebsd.org URL: https://wiki.freebsd.org/201405DevSummit/Jenkins URL: http://www.bsdcan.org/2014/schedule/events/445.en.html URL: http://clang-analyzer.llvm.org/scan-build.html URL: http://scan.freebsd.org URL: http://www.bsdnow.tv/episodes/2014_07_02-base_iso_100 URL: http://www.slideshare.net/CraigRodrigues1/libvirt-bhyve URL: https://github.com/jmmv/kyua URL: https://issues.jenkins-ci.org/browse/JENKINS-24521 URL: https://github.com/jenkinsci/jenkins/pul=B31387 URL: https://github.com/jenkinsci/jenkins/pul=B31410 URL: http://jenkins.mouf.net/job/pkg/ URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D193499 URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D193246 URL: https://github.com/rodrigc/kyua/wiki/Quickstart-Guide Contact: Craig Rodrigues Contact: Jenkins Administrators Contact: FreeBSD Testing In May, Craig Rodrigues led a working group Continuous Integration and Testing with Jenkins for FreeBSD at the FreeBSD Devsummit in Ottawa. Craig Rodrigues also gave a Jenkins presentation at BSDCan. At BSDCan, Craig Rodrigues had some discussions with Julio Merino about how to better integrate FreeBSD testing efforts with Jenkins and kyua. As a result of this discussion, Julio Merino enhanced the kyua testing framework with a kyua report-junit command. This command takes kyua test results and outputs a test report in JUnit XML format. Jenkins can directly import JUnit XML test results and display them nicely. In June, Craig Rodrigues was interviewed for episode 44 of BSD Now. The interview covered Jenkins and Continuous Integration in the FreeBSD project. In July, Craig Rodrigues gave a presentation to the Bay Area FreeBSD Users Group (BAFUG), Libvirt and bhyve, based on experience he had with those technologies when used with Jenkins. Li-Wen Hsu set up a Jenkins job to run the LLVM scan-build tool to perform static analysis of the FreeBSD code, and make the results availalble at scan.freebsd.org. Steve Wills modified the Jenkins job which builds pkg(8) to use the kyua report-junit command to integrate pkg(8) test results in Jenkins. Anthony Williams reported that the version of the Java Native Access (JNA) library bundled with Jenkins has problems on FreeBSD. This causes problems with Jenkins using libpam and other plugins that use JNA. Craig filed JENKINS-24521 against Jenkins. Craig submitted patches to Jenkins to update Jenkins to use JNA 4.1.0, which has fixes for FreeBSD. Craig Rodrigues worked on automatically running the tests in the FreeBSD /usr/tests directory under Jenkins using the kyua test framework. Craig Rodrigues provided feedback to Julio Merino about kyua and Julio Merino incorporated some of the feedback as bugfixes and feature enhancements to kyua. Craig Rodrigues committed some fixes to FreeBSD to eliminate some test failures. One of the tests still results in a crash in byacc. This is being tracked in PR 193499. Thomas E. Dickey (byacc maintainer) submitted a patch to fix the problem, and this has been committed to FreeBSD. Craig Rodrigues analyzed the cause of some startup errors in Jenkins when opening a multicast socket. He had some discussion with Bruce M. Simpson captured in PR 193246. The Java JDK depends on functionality in Solaris where it is possible to open an IPv4 multicast socket, but with an IPv4 multicast address mapped to an IPv6 address. On Linux, there are modifications to the JDK itself to work around this. Bruce M. Simpson said that the work to make the FreeBSD IPv6 stack behave like Solaris would require some work. Craig Rodrigues started writing a Kyua Quickstart Guide. This guide is meant to help new kyua users who want to write tests and run them under kyua. Craig Rodrigues is seeking feedback to help improve this guide. Open tasks: 1. Upstream more fixes to Jenkins for FreeBSD, such as JNA fixes. 2. Automate the build of /usr/tests. 3. Set up more builds based on examples from the existing FreeBSD Tinderbox. 4. Get feedback for improving the Kyua Quickstart Guide. 5. We need more people to join us on freebsd-testing@FreeBSD.org and help out. We especially need people with devops and scripting experience to help us set up more builds and tests. We would also like to integrate with other parts of the project, such as Phabricator. __________________________________________________________________ New Automounter URL: https://wiki.freebsd.org/Automounter URL: http://people.freebsd.org/~trasz/autofs.pdf Contact: Edward Tomasz Napiera=B3a Limitations of the current automounter, amd(8), are a recurring problem reported by many FreeBSD users. A new automounter is being developed to address these concerns. The automounter is a cleanroom implementation of functionality available in most other Unix systems, using proper kernel support implemented via an autofs filesystem. The automounter supports a standard map format, and will integrate with the Lightweight Directory Access Protocol (LDAP) service. The FreeBSD Foundation worked with enterprise and university users to test the new automounter in existing LDAP-based environments, including some with thousands of map entries. The code is now ready to use. It has been committed to 11-CURRENT and 10-STABLE, and will ship as part of 10.1-RELEASE. There is ongoing work on improving performance and fixing possible bugs. This project is sponsored by The FreeBSD Foundation . __________________________________________________________________ QEMU bsd-user-Enabled Ports Building URL: https://wiki.freebsd.org/QemuUserModeHowTo URL: http://chips.ysv.freebsd.org/packages URL: https://github.com/seanbruno/qemu-bsd-user Contact: Sean Bruno Contact: Juergen Lock Contact: Stacey Son FreeBSD packages for the Tier-1 i386 and amd64 CPU architectures are built by a single very high-performance machine. Other architectures lack equivalent hardware, and we began experimenting with QEMU's user-mode emulation to cross-build packages from an amd64 builder. We have moved from just being able to produce packages to providing a stable repo of packages for ARMv6. ports-mgmt/poudriere-devel is still the current method for building packages. See the previous status report for explanations and details on methods. __________________________________________________________________ VMWare VAAI and Microsoft ODX Acceleration in CTL Contact: Alexander Motin The CAM Target Layer (CTL), used as base for the kernel iSCSI server, got support for VMWare VAAI and Microsoft ODX storage acceleration. It permits avoiding network bottlenecks and improves storage efficiency on sets of large operations, such as virtual machine (or large file) creation, initialization to zeros, copy, delete, etc.. VMWare VAAI includes support for these primitives/SCSI commands: Atomic Test and Set (ATS) -- COMPARE AND WRITE command; Extended Copy (Clone) -- SPC-3 subset of XCOPY commands; Write Same (Zero) -- set of WRITE SAME commands; and Dead Space Reclamation (Delete) -- UNMAP command. Microsoft ODX includes support for these SCSI commands: POPULATE TOKEN/WRITE USING TOKEN (SPC-4 extensions of XCOPY), WRITE SAME and UNMAP. All XCOPY operations are currently limited to one storage host. ODX operations are currently limited only to iSCSI disks. Accelerated inter-host copying or copying to/from files on Samba shares is not implemented and is handled by initiators in the legacy way. The code is committed to FreeBSD head and stable/10 branches, and will be present in FreeBSD 10.1 and FreeNAS 9.2.1.8 / 9.3 releases. This project is sponsored by iXsystems, Inc. . Open tasks: 1. Full support for thin provisioning, including capacity usage reporting and threshold notifications. 2. Inter-host XCOPY operations. __________________________________________________________________ ZFSguru URL: http://zfsguru.com URL: http://zfsguru.com/news/stateoftheproject/2014 URL: http://zfsguru.com/forum/zfsgurudevelopment/876 Contact: Jason Edwards ZFSguru is a multifunctional server appliance with a strong emphasis on storage. ZFSguru began as simple web-interface frontend to ZFS, but has since grown into a FreeBSD derivative with its own infrastructure. The scope of the project has also grown with the inclusion of add-on packages that add functionality beyond the traditional NAS functionality found in similar product like FreeNAS and NAS4Free. ZFSguru aims to be a true multifunctional server appliance that is extremely easy to set up and can unite both novice and more experienced users in a single user interface. The modular nature of the project combats the danger of bloat, whilst still allowing extended functionality to be easily deployed. The development work in Q3 focused heavily on the new build infrastructure. This allows the ZFSguru project to release new system images together with addon services at much higher frequency and with much less manual intervention. This should free up a lot of development time to be spent on the core of the project: the web interface. Furthermore, a new website and forum is being worked on, replacing the old-fashioned website that offers only limited functionality. The new website will be linked to the server database, providing real-time updates about the project. In addition, a new platform for collaborated development is in the works. A service addon has been created for the GitLab project, which is a drop-in replacement of the popular GitHub website. The choice was made to host our own solution and not rely on GitHub itself. In retrospect this appears to have been a good decision. The recent development where GitHub removed projects after DCMA-takedowns being sent is incompatible with the philosophy of free-flow-of-information, of which the ZFSguru project is a strong proponent. By hosting our own solution, we have avoided any dependency on third party projects. The next task will be to introduce a new remote database structure, dubbed GuruDB. This will speed up the web-interface as well as introduce Service Bulletins which address important notifications to our users, as well as announce new releases. After GuruDB, the Migration Manager is one of the last remaining features still missing in the web-interface. This functionality provides an easy way to upgrade the current system by performing a new clean installation, but migrate all relevant configuration to the new installation. It also allows users to 'backup' all system configuration in a single file to be stored on a different machine should things go awry. A longer version of the 2014 development progress of the ZFSguru project and information specific to the newly-released 10.1-002 system image can be found in the Links section. __________________________________________________________________ Intel GPU Driver Update URL: https://lists.freebsd.org/pipermai=B3freebsd-current/2014-October/052532 .html Contact: Konstantin Belousov The project to update the Intel graphics chipset driver (i915kms) to a recent snapshot of the Linux upstream code continues. A patch with a large chunk of updates has been made available to test for regressions against current functionality, but is not yet expected to provide working new functions. The GEM I/O ioctl code path has been modified to more closely resemble the Linux code structure (easing future imports). This project is sponsored by FreeBSD Foundation . Open tasks: 1. Fix any bugs reported against the latest versions of the patch. 2. Make Haswell graphics work with Mesa. __________________________________________________________________ SDIO Driver URL: https://wiki.freebsd.org/SDIO URL: https://github.com/kibab/freebsd/tree/mmccam URL: https://reviews.freebsd.org/D862 Contact: Ilya Bakulin SDIO is an interface designed as an extension of the existing SD card standard, allowing the connection of different peripherals to a host with a standard SD controller. Peripherals currently sold in the general market include WLAN/BT modules, cameras, fingerprint readers, and barcode scanners. SDIO is also used to connect some peripherals in products like Chromebooks and Wandboard. The current main focus of the project is to reimplement the existing MMC/SD stack using the CAM framework. This will allow utilizing the well-tested CAM locking model and debug features. The first version of the code was uploaded on Phabricator for review. The new stack is able to attach to the SD card and bring it to an operational state. The only supported SD controller driver is ti_sdhci which is used by the BeagleBone Black. Modifying other SDHCI-compliant drivers should not be a hard task. Open tasks: 1. At this point, feedback from kernel developers is really needed. This may be done in the form of code review. If the chosen way of implementing the CAM-aware MMC stack is considered correct, then adding code for interacting with SD cards (for example, setting the optimal transfer rates) will be the next task. 2. Write a CAM peripheral driver that implements an interface to the FreeBSD disk(9). It will send MMC I/O commands using the MMC XPT layer. 3. Extending camcontrol(8) to make it possible to send MMC-specific commands directly to the MMC/SD card using pass(4) will greatly assist in developing new features for the stack. 4. Modify the sdhci(4) driver to work with the new stack. This is required to work on the new stack using PC hardware, not only the BeagleBone Black. __________________________________________________________________ UEFI Boot URL: https://wiki.freebsd.org/UEFI URL: http://www.freebsd.org/snapshots/ Contact: Ed Maste Contact: Nathan Whitehorn The Unified Extensible Firmware Interface, or UEFI, provides boot- and run-time services for x86 and other computers. For the x86 architecture it replaces the legacy BIOS. This project will adapt the FreeBSD loader and kernel boot process for compatibility with UEFI firmware, found on contemporary servers, desktops, and laptops. Over the last three months Ed and others refined the existing UEFI support and merged it to the stable/10 branch for the upcoming FreeBSD 10.1 release. To avoid the risk of a regression, the standard FreeBSD 10.1 install images continue to use the existing partitioning scheme and support only legacy BIOS boot. Separate UEFI-enabled installer images will be included with 10.1. This project is sponsored by The FreeBSD Foundation. Open tasks: 1. Document manual installation, including dual-boot configurations. 2. Implement boot1.efi for ZFS file systems. 3. Add support for UEFI variables stored in non-volatile memory (NVRAM). 4. Debug boot failures with certain UEFI firmware implementations. 5. Support secure boot. __________________________________________________________________ Updated vt(4) System Console URL: https://wiki.freebsd.org/Newcons Contact: Aleksandr Rybalko Contact: Ed Maste Contact: Ed Schouten Contact: Jean-S=E9bastien P=E9dron Contact: Warren Block The vt(4) (aka Newcons) project provides a replacement for the legacy syscons system console. It brings a number of improvements, including better integration with graphics modes and broader character set support. A large number of improvements were committed to vt(4) over the last three months. Jean-S=E9bastien P=E9dron fixed significant performance regressions observed with vt_vga, particularly noticeable on virtual machines. Stefan Esser converted and cleaned up all of the keyboard map files for use with vt(4). The EFI framebuffer driver and the ofwfb driver now work with the xf86-video-scfb X11 video driver, supporting native-resolution (albeit unaccelerated) X. The fixes and improvements have all been merged and will be available in the upcoming FreeBSD 10.1 release. Open tasks: 1. Implement the remaining features supported by vidcontrol(1). 2. Write manual pages for vt(4) drivers and kernel interfaces. 3. Support direct handling of keyboards by the kbd device (without kbdmux(4)). 4. CJK fonts. This is in progress. 5. Switch to vt(4) by default. 6. Implement compatibility mode to be able to use single-byte charsets/key-codes in vt(4). __________________________________________________________________ Updating OpenCrypto URL: https://p4db.freebsd.org/depotTreeBrowser.cgi?FSPC=3D//depot/projects/op= e ncrypto URL: http://freebsdfoundation.blogspot.com/2014/08/freebsd-foundation-announ ces-ipsec.html Contact: John-Mark Gurney The project adds support for the AES-GCM and AES-CTR cryptography modes to the OpenCrypto framework. Both software and AES-NI accelerated versions are now functional and working. Ermal Lu=E7i (eri@) is working on adding support for these additional modes to IPsec. This project is sponsored by The FreeBSD Foundation , and Netgate . Open tasks: 1. Create a test suite for the most common modes. __________________________________________________________________ FreeBSD on Newer ARM Boards URL: https://github.com/tsgan/qualcomm Contact: Ganbold Tsagaankhuu Work on initial support of the IFC6410 board, which was stopped due to a bricked bootloader, has been started again. This board has the Qualcomm Snapdragon S4 SoC, featuring the Krait CPU. This CPU is considered a "platform" for use in smartphones, tablets, and smartbook devices. Krait has many similarities with the ARM Cortex-A15 CPU and is also based on the ARMv7 instruction set. These peripherals are working: * Qualcomm High Speed UART driver for MSM 7000/8000 series chips (included in src tree) * Krait Timer Open tasks: 1. Get the MMC driver working. May need more help from experts. __________________________________________________________________ FreeBSD/arm64 URL: http://svnweb.freebsd.org/base/projects/arm64/ Contact: Andrew Turner Until recently, all ARM CPU designs were 32-bit only. With the introduction of the ARMv8 architecture, ARM has added a new 64-bit mode. This new mode has been named AArch64. Arm64 is the name of the in-progress port of FreeBSD to ARMv8 CPUs when in AArch64 mode. Since the last status report, FreeBSD has started to execute userland instructions. This includes implementing more of the needed kernel functions to handle creation of processes. Using clang to compile userland has found a few issues with the version in the base system. These issues are expected to be resolved when clang 3.5 is imported. Initial support for device drivers has been added. This includes the start of the bus_space functions and interrupt handling. This allowed the existing timer and interrupt controller drivers from armv6 to be used as these devices are similar. The FDT data is now being passed from the loader to the kernel using the standard mechanism. The pmap implementation has been changed to be based on the amd64 code. This fixes a number of issues with the old implementation. Open tasks: 1. Boot to multi-user mode 2. Get dynamic libraries working 3. Test on real hardware __________________________________________________________________ LLDB Debugger Port URL: https://wiki.FreeBSD.org/lldb Contact: Ed Maste LLDB is the debugger project associated with Clang/LLVM. It supports the Mac OS X, Linux, and FreeBSD platforms, with Windows support under development. It builds on existing components in the larger LLVM project, for example using Clang's expression parser and LLVM's disassembler. Work over the last three months consisted mainly of maintenance, ensuring that the upstream FreeBSD port continues to build and that testsuite failures are promptly addressed. I plan to import a new LLDB snapshot after the base system Clang is updated to 3.5. Some upstream improvements that will be in that import include: * Ability to specify a count to thread step-* operations. * Ongoing AArch64 development. * Significant progress on the lldb-gdbserver debug stub. * I/O handling improvements. * A much faster C++ name demangler for most symbols. A proof-of-concept implementation of kernel debugging support for amd64 was completed as part of Google Summer of Code. It is not ready to be committed, but will form the basis for upcoming kernel debugging support. This project is sponsored by DARP/AFRL, SRI International, and University of Cambridge. Open tasks: 1. Port remote debug stub (lldb-gdbserver) from Linux to FreeBSD. 2. Add support for local and core file kernel debugging. 3. Implement, fix or test support on all non-amd64 architectures. 4. Verify cross-debugging. 5. Investigate and fix test suite failures. 6. Package LLDB as a port. 7. Enable by default in the base system for working architectures. __________________________________________________________________ LLVM Address Sanitizer (Asan) URL: https://code.google.com/p/address-sanitizer/ URL: http://clang.llvm.org/docs/AddressSanitizer.html URL: http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd URL: http://lists.freebsd.org/pipermai=B3svn-src-head/2014-July/060270.html URL: http://lists.freebsd.org/pipermai=B3svn-src-head/2014-July/060504.html Contact: Viktor Kuzutov The LLVM address sanitizer (Asan) is a fast memory error detector that can detect use-after-free errors and buffer overflows. It has been ported to FreeBSD. The mainline version of LLVM is known to pass all of the tests in the LLVM and Asan test suites without unexpected failures on FreeBSD 10.0. A buildbot running sanitizer tests under FreeBSD stable/10 has been established. See the Links section. To make it possible to run programs with sanitizer checks enabled on FreeBSD, a new sysctl named kern.proc_vmmap_skip_resident_count has been added. See the Links section. Running the address sanitizer checks on stable/10 requires this sysctl to be set to 1. A similar work is in progress to add FreeBSD support to the thread sanitizer (Tsan), which detects data races in parallel programs. __________________________________________________________________ SSE Variants of libc Routines for amd64 URL: http://trac.baldwin.cx:8080/freebsd/wiki/LibCSSE Contact: John Baldwin I have written SSE/AVX-optimized versions of a few libc routines for amd64. So far the list includes memcpy, memset, and strlen. For each routine I have written a simple regression test as well as performed some simple microbenchmarks on various AMD and Intel CPUs. The simplest routine is strlen which appears to be a general win in microbenchmarks. memcpy and memset have proven trickier as different variants can behave quite differently on different CPUs. At present, I do not yet have a patch relative to libc. Once I do, this will be suitable for more testing. I would like to see some real-world benchmarks that show measurable improvement before pushing any of this up into the tree. Open tasks: 1. Create a branch that holds a modified libc and is suitable for testing __________________________________________________________________ FreeBSD Python Ports URL: https://wiki.FreeBSD.org/Python URL: irc://freebsd-python@irc.freenode.net Contact: FreeBSD Python Team The FreeBSD Python team continued to improve the overall experience with Python-based software on FreeBSD. During the last quarter, the bsd.python.mk bits of the ports infrastructure were converted to the more modern USES format. Several options, such as support for easy_install, were deprecated or removed to make the infrastructure easier to maintain and less complex for port maintainers. The Python ports were refactored and simplified to improve maintainability and to get rid of long-standing issues due to the previously complex and error-prone build process. The Python 2 branch was updated to Python 2.7.8 and setuptools to 5.5.1. With the availability of pkg 1.3, installing Python packages and modules for different Python versions is now supported in the package management infrastructure. This allows us to remove the previously required port duplicates for Python 2 and Python 3. Open tasks: 1. Retire the Python 3 specific port duplicates 2. Convert ports to the new USES syntax 3. More tasks can be found on the team's wiki page (see Links). 4. To get involved, interested people can say hello on IRC and let us know their areas of interest! __________________________________________________________________ GNOME/FreeBSD URL: http://www.freebsd.org/gnome URL: https://wiki.gnome.org/Projects/Jhbuild/FreeBSD URL: http://marcuscom.com/downloads/marcusmerge Contact: FreeBSD GNOME Team GNOME is a desktop environment and graphical user interface that runs on top of a computer operating system. GNOME is part of the GNU Project and can be used with various Unix-like operating systems, including FreeBSD. MATE is a fork of GNOME 2. The MATE ports were updated to the 1.8 versions. Cairo, the vector graphics library used by GNOME, has been updated to 1.12. This allowed the merge of GNOME 3 to begin. We are currently doing test builds to find ports broken by the update and pruning ports that do not build any more because of incompatible updates. Gustau Perez started preliminary work on the next development version of GNOME in MC, to be ready for GNOME 3.15. We will skip 3.14 entirely. Open tasks: 1. Finish the GNOME 3.12 merge, and start tracking GNOME 3.15 (development series). __________________________________________________________________ KDE on FreeBSD URL: https://freebsd.kde.org/ URL: https://freebsd.kde.org/area51.php URL: https://wiki.freebsd.org/KDE URL: https://mail.kde.org/mailman/listinfo/kde-freebsd URL: http://portscout.freebsd.org/kde@freebsd.org.html Contact: KDE on FreeBSD team The KDE on FreeBSD team focuses on packaging and making sure the experience of KDE and Qt on FreeBSD is as good as possible. First of all, we are happy to announce that Alonso Schaich, longtime contributor to our experimental area51 repositories, has become a ports committer, mentored by KDE on FreeBSD members Raphael Kubo da Costa (rakuco@) and Max Brazhnikov (makc@). During this quarter, the team has kept most of the KDE and Qt ports up-to-date, working on the following releases: * CMake 3.0.1 and 3.0.2 * PyQt 4.11.1, SIP 4.16.2, QScintilla 2.8.3 * DigiKam 4.2.0 (in area51) * KDE 4.13.3, 4.14.0 and 4.14.1 (in area51) * KDE Telepathy 0.8.0 (in area51) Additionally, work on updating the Qt5 ports to the 5.3 series has begun, and we intend to commit the updated ports in our experimental area51 repository to the ports tree in Q4. Open tasks: 1. Updating out-of-date ports, see the Links Portscout entry for a list. 2. Committing all the updated ports we have been accumulating in our experimental repositories into the ports tree. __________________________________________________________________ The Graphics Stack on FreeBSD URL: https://wiki.freebsd.org/Graphics URL: http://trillian.chruetertee.ch/ports/browser/trunk URL: http://lists.freebsd.org/pipermai=B3freebsd-ports-announce/2014-October/ 000096.html URL: http://wiki.x.org/wiki/Events/XDC2014/ Contact: FreeBSD Graphics team The newest graphics stack (that is, the set of ports conditional on the WITH_NEW_XORG knob) was enabled on all architectures. The only regression is for users of Intel GPUs and FreeBSD 8.X or 9.0. Those releases lack the required kernel driver and therefore xf86-video-intel will not work (the last UMS-aware version does not work with xserver 1.12). Users can still use xf86-video-vesa if they cannot or do not want to update their FreeBSD workstation. Owners of Radeon GPUs can use xf86-video-ati-ums 6.14.6 with xserver 1.12 if the KMS driver is not available (that is, before FreeBSD 9.3). The old graphics stack will be removed with the next update to these ports. See the announcement in the Links section. Hardware context support was added to the i915 driver in both HEAD and 10.1-RELEASE. This will allow us to update libglapi, libGL, dri, libEGL and libglesv2 ports to a newer version of Mesa. The latest version is already available from our development ports tree (see the links section). Cairo was updated to 1.12. This will allow the FreeBSD GNOME team to upgrade pango and Gtk+ 3. Unfortunately, the update also revealed that xf86-video-intel 2.7.1 was in a much worse state than previously assumed. We will attend XDC 2014 (X.Org Developer's Conference) from October 8th through 10th in Bordeaux, France. The goal is to reconnect with graphics stack developers, who are mostly working with Linux these days. We will give a presentation on the current state of this stack on FreeBSD. See the XDC website in the Links section for the program and live streaming. Open tasks: 1. See the "Graphics" wiki page for up-to-date information. __________________________________________________________________ Xfce URL: https://wiki.freebsd.org/Xfce URL: http://www.redports.org/browser/olivierd/xfce4 URL: https://people.freebsd.org/~olivierd/xfce4-4.11-screencast.webm Contact: FreeBSD Xfce Team Xfce is a free software desktop environment for Unix and Unix-like platforms including FreeBSD. It aims to be fast and lightweight while still being visually appealing and easy to use. The Xfce team continues to keep each piece of the Xfce Desktop up to date. That is why we are working on the next stable release (no date scheduled). There were no major updates in the ports tree except for cosmetic changes this quarter. Major upcoming changes include: * Switch to the USES framework * Support both GTK2 and GTK3, with GTK2 being the default. * A GNOME-like default icons theme * Enhanced documentation (handbooks, FAQ) Below is a list of current ports in the devel repository (see link). * deskutils/xfce4-xkb-plugin 0.7.0 * deve=B3xfce4-dev-tools 4.11.0 * misc/xfce4-appfinder 4.11.0 * multimedia/xfce4-parole 0.6.1 and 0.7.0 * sysutils/garcon 0.3.0 * sysutils/xfce4-settings 4.11.3 * x11/libxfce4menu 4.11.1 * x11/libxfce4util 4.11.0 * x11-wm/xfce4-desktop 4.11.8 * x11-wm/xfce4-panel 4.11.1 * x11-wm/xfce4-session 4.11.0 There are also two new ports * deskutils/xfce4-volumed-pulse 0.2.0 * x11/xfce4-dashboard 0.2.3 and 0.3.2 For more details, please see our wiki page in the Links section. Open tasks: 1. Finish patching the ACPI helper (xfce4-power-manager). 2. Continue to work on documentation, especially the Porter's Handbook, and creata a FAQ). __________________________________________________________________ Handbook ezjail Section URL: https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-ezjail .html Contact: Warren Block ezjail is a very popular jails management utility, but was only mentioned in passing in the Handbook. This new section describes basic setup and usage. An in-depth example shows how to create and configure a jail. It also serves as an example of how to run a simple caching-only BIND in a jail. __________________________________________________________________ Michael Lucas Books URL: http://blather.michaelwlucas.com/archives/2088 URL: http://blather.michaelwlucas.com/archives/2119 URL: https://www.tiltedwindmillpress.com/?product=3Dfreebsd-mastery-storage-e= s sentials Contact: Michael Lucas Lucas is working on a series of small FreeBSD books. The first one, FreeBSD Mastery: Storage Essentials, is underway, and covers GEOM, gpart, MBR, UFS, GELI, GBDE, disk sector alignment, and more. You can pre-order the book at a discount from his web site, or wait for it to hit print and all major ebook retailers. Get status updates on his blog, or check @mwlauthor on Twitter. Open tasks: 1. Lucas needs to write faster. __________________________________________________________________ ZFS Chapter of the Handbook URL: https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs.html Contact: Allan Jude Contact: Benedict Reuschling Contact: Warren Block ZFS is one of the premier features of FreeBSD, and the quality of the documentation should match that of other important features. Much of the original documentation from Sun and Oracle has disappeared, moved, or is about the proprietary version of ZFS. The OpenZFS project does not provide much documentation and instead provides users with a few links, including the FreeBSD Handbook. New users have many questions about ZFS and yet there exists a great deal more bad advice about ZFS than proper documentation. After over a year of work, a new ZFS chapter has been added to the FreeBSD Handbook. Over 20,000 words describe the basics of creating, managing, and maintaining a ZFS pool. Advanced features like compression, deduplication, and delegation are covered. The chapter also contains a glossary of terms, explaining a number of the concepts unique to ZFS, and documents some of the many sysctl variables that can be used for tuning. The remaining work to be done is in the FAQ section, which aims to help users address the most common questions or problems they might face with ZFS. We would like to hear experiences, questions, misconceptions, gotchas, stumbling blocks and suggestions for the FAQ section from other users. A use cases section that highlights some of the cases where ZFS provides advantages over traditional file systems is also planned. Please send suggestions to the docs mailing list. This project is sponsored by ScaleEngine Inc.. Open tasks: 1. Technical review by Matt Ahrens (co-creator of ZFS) 2. Improve the delegation section 3. Improve the tuning section, and cover recently added sysctls 4. Add a section on jails and the jailed property 5. Add an FAQ section 6. Add a Use Cases section 7. General editing and review __________________________________________________________________ The FreeBSD Foundation URL: http://www.FreeBSDFoundation.org/ URL: http://freebsdjournal.com/ Contact: Deb Goodkin The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Most of the funding is used to support FreeBSD development projects, conferences, and developer summits, purchase equipment to grow and improve the FreeBSD infrastructure, and provide legal support for the Project. We published our fourth issue of the FreeBSD Journal. We have over 4500 subscriptions to date. Work continued on adding support for the Dynamic Edition which will be available soon. The fifth issue is also due out soon. Foundation staff member Konstantin Belousov wrapped up the PostgreSQL performance investigation project. Kostik reran the benchmarks as a configuration error may have affected earlier results. Improvements arising from the investigation are merged to the FreeBSD 10 development branch and will be in the 10.1 release. Kostik also committed a variety of virtual memory and file system bug fixes and improvements. Over the quarter, Foundation staff member Edward Napiera=B3a refined the new autofs-based automounter and incorporated feedback from testers in enterprise and university contexts. The automounter is available in the development branch of FreeBSD and will be included in FreeBSD 10.1. Edward also supported engineers at Chelsio in preparing iSCSI offload support for Chelsio's 10- and 40-gigabit per second Ethernet adapters. Ed Maste, our project manager, tested and integrated UEFI system boot and new vt(4) console work into the release branch for the upcoming FreeBSD 10.1 release. He committed a number of small toolchain and build infrastructure improvements. He also wrote an article on LLDB for the FreeBSD Journal and presented the LLDB work at EuroBSDCon. FreeBSD Foundation Systems Administrator and Release Engineer Glen Barber continued work on finalizing the 9.3-RELEASE process, followed by starting the 10.1-RELEASE process. Work has continued on producing regularly-updated FreeBSD development snapshot builds for the various supported architectures, which include amd64, i386, ia64, powerpc, powerpc64, sparc64, and arm. In addition, work has been committed to a project branch which allows FreeBSD virtual machine disk images to be produced by default as part of the release process. A plan has been put together for the upcoming Secure Boot work. More hardware has been purchased to support FreeBSD infrastructure at NYI and Sentex. We announced a collaboration between the Foundation and Cavium to deliver a FreeBSD ARMv8 based implementation. We signed a license agreement with Oracle to get access to the TCKs for Java 7 and 8. Robert Watson ran and organized the Cambridge Developer Summit. We provided a travel grant to a Google Summer of Code student to attend the summit. We provided a travel grant to a developer who organized and ran BSDDay in Argentina. We were a Gold Sponsor for EuroBSDCon 2014 and sponsored the Developer Summit. We provided 4 travel grants to assist FreeBSD contributors with their travel expenses to attend the conference. We also had 6 board/staff members attend the conference and some gave talks, tutorials, and chaired some sessions. We held our Fall Fundraising campaign there and raised over $2,000 in donations from attendees. We organized the Silicon Valley Vendor/Developer Summit that is happening November 3 and 4. Kirk McKusick, Robert Watson, and George Neville-Neil published the second edition of "The Design and Implementation of the FreeBSD Operating System." Kirk McKusick presented a 2-day tutorial on the FreeBSD kernel and gave a talk on the implementation of ZFS at EuroBSDCon. Dru Lavigne attended Fossetcon: September 11-13 (fossetcon.org). We created new recruiting fliers for upcoming events, including the Grace Hopper conference. We started sending out Foundation monthly update emails to keep the FreeBSD community informed on some of the activities we did the previous month to support FreeBSD. __________________________________________________________________ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGgBAEBCgAGBQJUPZwDAAoJECjZpvNk63USKbkMH0Q4dV1ZPR5PwkJXLW9HV8Ap wllhTQ9OG3kjq73nMkRpTI0xoGvldwAWbIXfAw1g0LfooewgImx4eLNBX858mYHA IQtafoDDrH2retKtUsDiobsoTAAbMjtRJDZJ0sd+sMWr9y59CkdvaLhnNLeQni7w jwIlDviZQYPyY0upTgXitb0sDN0K3W0ys1AmQoVYpClfn8N4qGKuCzWX95linj4R ykB3SSSeN1rI6dZhSR8H1Uc6aCv3D3gokP3sbwADYG6A321TAYCEtj7uQzTCO+V7 uZNn1LrhFQ8dT4Es8VU2FCbW4qdenFy2I5/86o/PF46kNoEnuOoCLxTcUa7UCtHj OtA4g/k9J3FKjOAiufMaAP44xy+fPvFPJWI88kwaSv/npX3VyhwNR3MN2RNUNEiJ HMm/ImuYIWCuFTqvVzAiWWZxxXxrzMAbdZCNiRmFhmv5KJznhnWkUvUI8CRqtMVF ZfANAYYgcZ8qqmeFpo2K/sDhsDbYu5IU9YY60MOJfOBcnQg=3D =3D3AH2 -----END PGP SIGNATURE----- From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 04:52:15 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 16B65F4F; Wed, 15 Oct 2014 04:52:15 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id A21B6AEB; Wed, 15 Oct 2014 04:52:14 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id 4FD4B20E7088B; Wed, 15 Oct 2014 04:52:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 985EE20E70885; Wed, 15 Oct 2014 04:52:10 +0000 (UTC) Message-ID: <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Wed, 15 Oct 2014 05:52:08 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 04:52:15 -0000 ----- Original Message ----- From: "Steven Hartland" To: "Mark Martinec" ; ; Sent: Tuesday, October 14, 2014 12:40 PM Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] > ----- Original Message ----- > From: "Mark Martinec" > > >> On 10/14/2014 13:19, Steven Hartland wrote: >>> Well interesting issue I left this pool alone this morning literally doing >>> nothing, and its now out of space. >>> zpool list >>> NAME SIZE ALLOC FREE FRAG EXPANDSZ CAP DEDUP HEALTH >>> ALTROOT >>> sys1boot 3.97G 3.97G 190K 0% - 99% 1.00x ONLINE - >>> sys1copy 3.97G 3.97G 8K 0% - 99% 1.00x ONLINE - >>> >>> There's something very wrong here as nothing has been accessing the pool. >>> >>> pool: zfs >>> state: ONLINE >>> status: One or more devices are faulted in response to IO failures. >>> action: Make sure the affected devices are connected, then run 'zpool >>> clear'. >>> see: http://illumos.org/msg/ZFS-8000-HC >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> zfs ONLINE 0 2 0 >>> md1 ONLINE 0 0 0 >>> >>> I tried destroying the pool and ever that failed, presumably because >>> the pool has suspended IO. >> >> That's exactly how trouble started here. Got the >> "One or more devices are faulted in response to IO failures" >> on all three small cloned boot pools one day, out of the blue. >> There was no activity there, except for periodic snapshoting >> every 10 minutes. > > Yer this isn't fragmentation, this is something else. I've started a > thread on the openzfs list to discuss this as theres something quite > odd going on. After bisecting the kernel versions in stable/10 the problem commit appears to be: https://svnweb.freebsd.org/base?view=revision&revision=268650 Removing it or using a pool without async_destory enabled prevents the leak. More debugging tomorrow. Regards steve From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 09:40:54 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 177918FA for ; Wed, 15 Oct 2014 09:40:54 +0000 (UTC) Received: from haggis.mythic-beasts.com (haggis.mythic-beasts.com [IPv6:2a00:1098:0:86:1000:0:2:1]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D2005B21 for ; Wed, 15 Oct 2014 09:40:53 +0000 (UTC) Received: from [213.205.236.17] (port=57319 helo=[172.20.10.2]) by haggis.mythic-beasts.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1XeL4c-0002TD-Nn; Wed, 15 Oct 2014 10:40:51 +0100 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: System hang on shutdown when running freebsd-update From: Colin Perkins In-Reply-To: Date: Wed, 15 Oct 2014 10:40:40 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> References: To: Kevin Oberman X-Mailer: Apple Mail (2.1878.6) X-BlackCat-Spam-Score: -28 X-Mythic-Debug: Threshold = On = X-Spam-Status: No, score=-2.9 Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 09:40:54 -0000 On 14 Oct 2014, at 18:09, Kevin Oberman wrote: > I thought that this was just a fluke, but it has now happened three = times, > so I guess it's now out of the "fluke" class. >=20 > I have upgraded several times recently to each 10.1 BETA and RC. After = the > first install pass t install the kernel and modules, the system = shutdown > freezes at the very end. I see the buffers synced to the disks and get = the > "All buffers synced" message. Then it just hangs. The disks are not = marked > as clean and are fscked after a reset and boot. >=20 > There is not much between the "All buffers synced" message and the = call to > vfs_unmountall(), so I suspect it is hanging in that call. I admit = that I > am pretty much lost whenever I look at the VFS code and I have not put = a > lot of effort going further. Just hoping that someone familiar with it > might have an idea. >=20 > I have tried several reboots and all run normally. The problem only = seems > to appear when upgrading the OS. It happened repeatedly when I tried = to > reboot before doing the second "install" pass of freebsd-update, but = not > after, so the kernel and world are not in sync. I am baffled as to = what > could be going on, but it means I need to be at the system (a baby = server) > when I upgrade, but not every time I upgrade. I know it happened on = the > 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. >=20 > Has anyone else seen this? I=92m seeing the same behaviour, most recently when moving to 10.1-RC1 = (haven=92t gone to -RC2 yet). The system is: FreeBSD 10.1-RC1 #0 r272463: Fri Oct 3 01:47:10 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: AMD Opteron(TM) Processor 6274 (2200.05-MHz = K8-class CPU) Origin =3D "AuthenticAMD" Id =3D 0x600f12 Family =3D 0x15 Model =3D = 0x1 Stepping =3D 2 = Features=3D0x178bfbff = Features2=3D0x1e98220b AMD Features=3D0x2e500800 AMD = Features2=3D0x1c9bfff TSC: P-state invariant, performance statistics real memory =3D 549755813888 (524288 MB) avail memory =3D 534559084544 (509795 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <041112 APIC1739> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 4 package(s) x 16 core(s) =85 I do have IPMI loaded, unlike the other reports. --=20 Colin Perkins https://csperkins.org/ From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 14:04:26 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D3DB1BC for ; Wed, 15 Oct 2014 14:04:26 +0000 (UTC) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 16888BEE for ; Wed, 15 Oct 2014 14:04:25 +0000 (UTC) Received: from torb.pix.net (torb.pix.net [IPv6:2001:470:e254:10:12dd:b1ff:febf:eca9]) (authenticated bits=0) by hydra.pix.net (8.14.9/8.14.9) with ESMTP id s9FE4OjY076596; Wed, 15 Oct 2014 10:04:24 -0400 (EDT) (envelope-from lidl@pix.net) Message-ID: <543E7EE8.3000102@pix.net> Date: Wed, 15 Oct 2014 10:04:24 -0400 From: Kurt Lidl User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: Re: FreeBSD 10.1-RC2 Now Available References: <20141013165244.GA61249@hub.FreeBSD.org> In-Reply-To: <20141013165244.GA61249@hub.FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 14:04:26 -0000 The sparc64 FreeBSD-10.1-RC2-sparc64-disc1.iso image, when burned to a cdrom, does not successfully boot on my test sparc64 machine. This same machine had no problems booting the iso image for the -BETA2 and -BETA3 release images. I used that same machine for tracking down the zfsloader regression on the sparc. When I boot the cdrom image, it probes the devices, and then hangs after the "Trying to mount root..." message. Console session follows: Executing last command: boot cdrom Boot device: /pci@1f,0/pci@1,1/ide@d/cdrom@0,0:f File and args: >> FreeBSD/sparc64 boot block Boot path: /pci@1f,0/pci@1,1/ide@d/cdrom@0,0:f Boot loader: /boot/loader Consoles: Open Firmware console FreeBSD/sparc64 bootstrap loader, Revision 1.0 (root@releng1.nyi.freebsd.org, Fri Oct 10 05:21:46 UTC 2014) bootpath="/pci@1f,0/pci@1,1/ide@d/cdrom@0,0:a" Loading /boot/defaults/loader.conf /boot/kernel/kernel data=0xc08140+0xe6710 syms=[0x8+0xc8b38+0x8+0xbb2c4] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... jumping to kernel entry at 0xc00a0000. Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.1-RC2 #0 r272876: Fri Oct 10 05:25:28 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/sparc64.sparc64/usr/src/sys/GENERIC sparc64 gcc version 4.2.1 20070831 patched [FreeBSD] real memory = 4294967296 (4096 MB) avail memory = 4177436672 (3983 MB) cpu0: Sun Microsystems UltraSparc-IIe Processor (648.00 MHz CPU) random device not loaded; using insecure entropy random: initialized kbd0 at kbdmux0 nexus0: pcib0: mem 0x1fe00000000-0x1fe0000ffff,0x1fe01000000-0x1fe010000ff irq 2032,2030,2031,2021 on nexus0 pcib0: Sabre, impl 0, version 0, IGN 0x1f, bus A, 66MHz pcib0: DVMA map: 0xc0000000 to 0xc3ffffff 8192 entries pcib0: [GIANT-LOCKED] pci0: on pcib0 pcib1: at device 1.1 on pci0 pci1: on pcib1 ebus0: mem 0xf0000000-0xf0ffffff,0xf1000000-0xf17fffff at device 12.0 on pci1 ebus0: : incomplete pcib2: at device 1.0 on pci0 pci2: on pcib2 ebus0: addr 0x1000000000-0x10000fffff (no driver attached) eeprom0: addr 0x1400000000-0x1400001fff on ebus0 eeprom0: model mk48t59 ebus0: addr 0x1400200000-0x1400200003 irq 42 (no driver attached) pci1: at device 3.0 (no driver attached) isab0: at device 7.0 on pci1 isa0: on isab0 gem0: mem 0xe0400000-0xe041ffff at device 12.1 on pci1 miibus0: on gem0 ukphy0: PHY 1 on miibus0 ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow gem0: 2kB RX FIFO, 2kB TX FIFO gem0: Ethernet address: 00:03:ba:2f:fc:50 ohci0: mem 0xe2000000-0xe2007fff at device 12.3 on pci1 usbus0 on ohci0 atapci0: port 0x400-0x407,0x418-0x41b,0x410-0x417,0x408-0x40b,0x420-0x42f at device 13.0 on pci1 atapci0: using PIO transfers above 137GB as workaround for 48bit DMA access bug, expect reduced performance ata2: at channel 0 on atapci0 ata3: at channel 1 on atapci0 gem1: mem 0xe0440000-0xe045ffff at device 5.1 on pci1 miibus1: on gem1 ukphy1: PHY 1 on miibus1 ukphy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow gem1: 2kB RX FIFO, 2kB TX FIFO gem1: Ethernet address: 00:03:ba:2f:fc:51 ohci1: mem 0xe5000000-0xe5007fff at device 5.3 on pci1 usbus1 on ohci1 sym0: <896> port 0xc00000-0xc000ff mem 0x2000-0x23ff,0x4000-0x5fff at device 8.0 on pci2 sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking sym1: <896> port 0xc00100-0xc001ff mem 0x6000-0x63ff,0x8000-0x9fff at device 8.1 on pci2 sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking nexus0: type unknown (no driver attached) uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 43 on isa0 uart0: console (9600,n,8,1) uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 43 on isa0 Timecounter "tick" frequency 648000000 Hz quality 1000 Event timer "tick" frequency 648000000 Hz quality 1000 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 uhub0: 4 ports with 4 removable, self powered uhub1: 4 ports with 4 removable, self powered da0 at sym0 bus 0 scbus2 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device random: unblocking device. da0: Serial Number AAL0P58055P6 da0: 80.000MB/s transfers (40.000MHz, offset 31, 16bit) da0: Command Queueing enabled da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) da1 at sym0 bus 0 scbus2 target 1 lun 0 da1: Fixed Direct Access SCSI-3 device da1: Serial Number DAL0P7906GC1 da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit) da1: Command Queueing enabled da1: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) cd0 at ata2 bus 0 scbus0 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: cd present [269440 x 2048 byte records] Trying to mount root from cd9660:/dev/iso9660/FREEBSD_INSTALL [ro]... I had really wanted to test this image to verify that the zfsloader changes in the -RC2 image were good. I didn't try the -RC1 image, mostly due to lack of time. -Kurt From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 15:56:46 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C026374; Wed, 15 Oct 2014 15:56:46 +0000 (UTC) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id E2AB6A79; Wed, 15 Oct 2014 15:56:45 +0000 (UTC) Received: from marvin.lab.vangyzen.net (c-24-125-214-90.hsd1.va.comcast.net [24.125.214.90]) by smtp.vangyzen.net (Postfix) with ESMTPSA id D352256436; Wed, 15 Oct 2014 10:56:44 -0500 (CDT) Message-ID: <543E993B.5050406@vangyzen.net> Date: Wed, 15 Oct 2014 11:56:43 -0400 From: Eric van Gyzen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org, re@FreeBSD.org Subject: No network interfaces with 10.1-RC2 kernel and 9.2 userland References: <20141013165244.GA61249@hub.FreeBSD.org> In-Reply-To: <20141013165244.GA61249@hub.FreeBSD.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 15:56:46 -0000 I used freebsd-update to update a 9.2-RELEASE-p10 i386 system to 10.1-RC2. After installing the kernel and rebooting, still using the 9.2 userland, ifconfig showed no interfaces. Is this expected? Do I need to bounce through 9.3 or 10.0 first? Thanks, Eric From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 16:18:50 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4044F7A1; Wed, 15 Oct 2014 16:18:50 +0000 (UTC) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E730C7C; Wed, 15 Oct 2014 16:18:49 +0000 (UTC) Received: from [10.10.1.198] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by webmail2.jnielsen.net (8.14.9/8.14.9) with ESMTP id s9FGIk8N025031 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 15 Oct 2014 10:18:48 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host office.betterlinux.com [199.58.199.60] claimed to be [10.10.1.198] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: No network interfaces with 10.1-RC2 kernel and 9.2 userland From: John Nielsen In-Reply-To: <543E993B.5050406@vangyzen.net> Date: Wed, 15 Oct 2014 10:18:45 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <4BF232A6-0B0E-4859-AFDB-03854612015C@jnielsen.net> References: <20141013165244.GA61249@hub.FreeBSD.org> <543E993B.5050406@vangyzen.net> To: Eric van Gyzen X-Mailer: Apple Mail (2.1878.6) Cc: re@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 16:18:50 -0000 On Oct 15, 2014, at 9:56 AM, Eric van Gyzen wrote: > I used freebsd-update to update a 9.2-RELEASE-p10 i386 system to > 10.1-RC2. After installing the kernel and rebooting, still using the > 9.2 userland, ifconfig showed no interfaces. Is this expected? Do I > need to bounce through 9.3 or 10.0 first? Are your interfaces recognized by the kernel? (check dmesg and/or = "pciconf -lv") Does "ifconfig -a" show your interfaces? If so, it just = means they aren't configured. Can you share the relevant portions of your /etc/rc.conf? JN From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 16:36:25 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from hub.FreeBSD.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 67EC8B88; Wed, 15 Oct 2014 16:36:24 +0000 (UTC) Date: Wed, 15 Oct 2014 16:36:19 +0000 From: Glen Barber To: Eric van Gyzen Subject: Re: No network interfaces with 10.1-RC2 kernel and 9.2 userland Message-ID: <20141015163619.GR36695@hub.FreeBSD.org> References: <20141013165244.GA61249@hub.FreeBSD.org> <543E993B.5050406@vangyzen.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="mgIE+9cwyCTt+85Z" Content-Disposition: inline In-Reply-To: <543E993B.5050406@vangyzen.net> X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event User-Agent: Mutt/1.5.23 (2014-03-12) Cc: re@FreeBSD.org, freebsd-stable@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 16:36:25 -0000 --mgIE+9cwyCTt+85Z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 15, 2014 at 11:56:43AM -0400, Eric van Gyzen wrote: > I used freebsd-update to update a 9.2-RELEASE-p10 i386 system to > 10.1-RC2. After installing the kernel and rebooting, still using the > 9.2 userland, ifconfig showed no interfaces. Is this expected? Do I > need to bounce through 9.3 or 10.0 first? >=20 What interfaces should be there? Please provide the contents of /var/run/dmesg.boot. Glen --mgIE+9cwyCTt+85Z Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJUPqKDAAoJEAMUWKVHj+KTxR0P/iyXXBfoxTUeeS7IFPM6LzmO f/n4LDo/gn7gReKBUDqVAhuSmGsx6AX+i00vioapcZB7qrEtZcLarR3zjfOsBSVy gJYeTiOXJrk9LWemGC9OkAb9rvPKNS7dhfniEf+3B9cg0cPxj0h5ggS1caa+OiwN gr2Y6FwI87okF12xf5vlTrH2UK2ceuHVNBnorIz/uSveTW3uilAgwIyyuLYpFk2R O0+PzESleh4/rZHVYGyzXKGvz5bl+IM4b/FfMDtjJR8jYE5QcdQLvnjqqVyLT/wM 5uRam31IsxSxNBvGB3rlt87M3UtNE/wa4QFgRiMaHsVdBgJTVvpOEFduWOrITBT4 dRP7MQrNaKq1XVj+Lo0Yux1cj2VElPKs2TrfX7aB1X/TKTEeHxnrEDxqio21tELG W2QsAIYTlgSmRPkCPeOV12ndQiM3iRHgc7boFj+XQcu/HHZLWWJbJcr6KRdIBG+3 1vVO4DNEktNSMVcJ4rNLzEbGBbC/Lm5KTF/gmXr77yMX1p9jj9TSdlVhCFUoz59o 3Gjd45HIVmSbKtCkBJQYPvC6nQQtZlAXLkwNb49joJzG3CSnQK14PpvG/1l84SxA fz+bsFO0fO5WGgh0G+QPmlkzzcxh7ev8Z7jUW5rvx2Fcw2BsoIWojbJmk+/R0ZA+ VR1mG7ZcLiG6PibQhe1O =twGK -----END PGP SIGNATURE----- --mgIE+9cwyCTt+85Z-- From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 16:54:15 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AEA4DBD; Wed, 15 Oct 2014 16:54:15 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B038DB; Wed, 15 Oct 2014 16:54:15 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s9FGsA2g068910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 15 Oct 2014 19:54:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s9FGsA2g068910 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s9FGsAtw068909; Wed, 15 Oct 2014 19:54:10 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 15 Oct 2014 19:54:09 +0300 From: Konstantin Belousov To: Glen Barber Subject: Re: No network interfaces with 10.1-RC2 kernel and 9.2 userland Message-ID: <20141015165409.GZ2153@kib.kiev.ua> References: <20141013165244.GA61249@hub.FreeBSD.org> <543E993B.5050406@vangyzen.net> <20141015163619.GR36695@hub.FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141015163619.GR36695@hub.FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-stable@FreeBSD.org, Eric van Gyzen , re@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 16:54:15 -0000 On Wed, Oct 15, 2014 at 04:36:19PM +0000, Glen Barber wrote: > On Wed, Oct 15, 2014 at 11:56:43AM -0400, Eric van Gyzen wrote: > > I used freebsd-update to update a 9.2-RELEASE-p10 i386 system to > > 10.1-RC2. After installing the kernel and rebooting, still using the > > 9.2 userland, ifconfig showed no interfaces. Is this expected? Do I > > need to bounce through 9.3 or 10.0 first? > > > > What interfaces should be there? > > Please provide the contents of /var/run/dmesg.boot. The interfaces used to manage networking (i.e. the kernel calls utilized by ifconfig(8), route(8) and others) are not kept stable between stable branches. Usually, the compatibility is broken. Simply put, you cannot use configuration utilities from stable/9 on stable/10 kernel. This is expected. You did not followed the instruction in UPDATING. From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 16:57:07 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 80217331; Wed, 15 Oct 2014 16:57:07 +0000 (UTC) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BCDB103; Wed, 15 Oct 2014 16:57:07 +0000 (UTC) Received: from [10.10.1.198] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by webmail2.jnielsen.net (8.14.9/8.14.9) with ESMTP id s9FGv3eU055316 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 15 Oct 2014 10:57:06 -0600 (MDT) (envelope-from john@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host office.betterlinux.com [199.58.199.60] claimed to be [10.10.1.198] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: No network interfaces with 10.1-RC2 kernel and 9.2 userland From: John Nielsen In-Reply-To: <20141015165409.GZ2153@kib.kiev.ua> Date: Wed, 15 Oct 2014 10:57:03 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <18F7164A-31D8-483E-B403-5142DAB20CE4@jnielsen.net> References: <20141013165244.GA61249@hub.FreeBSD.org> <543E993B.5050406@vangyzen.net> <20141015163619.GR36695@hub.FreeBSD.org> <20141015165409.GZ2153@kib.kiev.ua> To: Eric van Gyzen X-Mailer: Apple Mail (2.1878.6) Cc: Konstantin Belousov , Glen Barber , freebsd-stable@FreeBSD.org, re@FreeBSD.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 16:57:07 -0000 On Oct 15, 2014, at 10:54 AM, Konstantin Belousov = wrote: > On Wed, Oct 15, 2014 at 04:36:19PM +0000, Glen Barber wrote: >> On Wed, Oct 15, 2014 at 11:56:43AM -0400, Eric van Gyzen wrote: >>> I used freebsd-update to update a 9.2-RELEASE-p10 i386 system to >>> 10.1-RC2. After installing the kernel and rebooting, still using = the >>> 9.2 userland, ifconfig showed no interfaces. Is this expected? Do = I >>> need to bounce through 9.3 or 10.0 first? >>>=20 >>=20 >> What interfaces should be there? >>=20 >> Please provide the contents of /var/run/dmesg.boot. >=20 > The interfaces used to manage networking (i.e. the kernel calls = utilized > by ifconfig(8), route(8) and others) are not kept stable between = stable > branches. Usually, the compatibility is broken. >=20 > Simply put, you cannot use configuration utilities from stable/9 on > stable/10 kernel. This is expected. >=20 > You did not followed the instruction in UPDATING. Oh, I missed the bit about the old userland. Yes, that is expected. You = need to run "freebsd-update install" one or more times to install the = new userland and remove old libraries (once you've updated your = ports/packages). From owner-freebsd-stable@FreeBSD.ORG Wed Oct 15 17:23:33 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1815AA62 for ; Wed, 15 Oct 2014 17:23:33 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D87635EF for ; Wed, 15 Oct 2014 17:23:32 +0000 (UTC) Received: by mail-ig0-f181.google.com with SMTP id r10so1813885igi.14 for ; Wed, 15 Oct 2014 10:23:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=n6WDiD2s+8rUhAvdc72PFP0tX9yo3Z+ljeeYqlyhw04=; b=bQyKLoWXX/F1FXE3uqAsi4y3V/NsJ5ycMXkS9goDp1r9NQ3e2ydIoXLNjFkwOX9vE9 duwueU1uamUWuYS7RR4vErcLPZQfOyDEapc8EU5IhbZfaYmzSxPBXLPXUlwchk0teSWv PqD47J1dk/GRllXO6hKmFyk9bix4FSv1iB8xn3p22uzPjtpnflFkQFmRCCHGWEa1TeK2 J9Y15ws31hPApizyBD+asV7QA47hYXsfr0Zv1MBfg3dSKre2VaALva7YmmguSnjlZ3B/ kloE4oAvJ5Oq1MFCgIgBSeEFXhHdyUZa5heL5oysDBk6RbEvpQKil9WpUdhYIRgictcl SGyA== MIME-Version: 1.0 X-Received: by 10.42.26.14 with SMTP id d14mr12621945icc.5.1413393812184; Wed, 15 Oct 2014 10:23:32 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.11.141 with HTTP; Wed, 15 Oct 2014 10:23:32 -0700 (PDT) In-Reply-To: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> Date: Wed, 15 Oct 2014 10:23:32 -0700 X-Google-Sender-Auth: lAXbJzYikgAejE8T0uIWa0eILRE Message-ID: Subject: Re: System hang on shutdown when running freebsd-update From: Kevin Oberman To: Colin Perkins Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Oct 2014 17:23:33 -0000 On Wed, Oct 15, 2014 at 2:40 AM, Colin Perkins wrote: > On 14 Oct 2014, at 18:09, Kevin Oberman wrote: > > I thought that this was just a fluke, but it has now happened three > times, > > so I guess it's now out of the "fluke" class. > > > > I have upgraded several times recently to each 10.1 BETA and RC. After > the > > first install pass t install the kernel and modules, the system shutdow= n > > freezes at the very end. I see the buffers synced to the disks and get > the > > "All buffers synced" message. Then it just hangs. The disks are not > marked > > as clean and are fscked after a reset and boot. > > > > There is not much between the "All buffers synced" message and the call > to > > vfs_unmountall(), so I suspect it is hanging in that call. I admit that= I > > am pretty much lost whenever I look at the VFS code and I have not put = a > > lot of effort going further. Just hoping that someone familiar with it > > might have an idea. > > > > I have tried several reboots and all run normally. The problem only see= ms > > to appear when upgrading the OS. It happened repeatedly when I tried to > > reboot before doing the second "install" pass of freebsd-update, but no= t > > after, so the kernel and world are not in sync. I am baffled as to what > > could be going on, but it means I need to be at the system (a baby > server) > > when I upgrade, but not every time I upgrade. I know it happened on the > > 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. > > > > Has anyone else seen this? > > I=E2=80=99m seeing the same behaviour, most recently when moving to 10.1-= RC1 > (haven=E2=80=99t gone to -RC2 yet). The system is: > > FreeBSD 10.1-RC1 #0 r272463: Fri Oct 3 01:47:10 UTC 2014 > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 > CPU: AMD Opteron(TM) Processor 6274 (2200.05-MHz K8-clas= s > CPU) > Origin =3D "AuthenticAMD" Id =3D 0x600f12 Family =3D 0x15 Model =3D 0x= 1 > Stepping =3D 2 > > Features=3D0x178bfbff > > Features2=3D0x1e98220b > AMD Features=3D0x2e500800 > AMD > Features2=3D0x1c9bfff > TSC: P-state invariant, performance statistics > real memory =3D 549755813888 (524288 MB) > avail memory =3D 534559084544 (509795 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: <041112 APIC1739> > FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs > FreeBSD/SMP: 4 package(s) x 16 core(s) > =E2=80=A6 > > I do have IPMI loaded, unlike the other reports. > > -- > Colin Perkins > https://csperkins.org/ > Paul Koch replied privately with a pointer to a seemingly unrelated message he sent to stable last month. Take a look at the several paragraphs at the end starting with "On a side note". I'm suspicious that the generation of the large upgrade on /var during the "upgrade" pass is causing the delay. It fits pretty well and, in normal operation, my server would never see this issue at all. https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D326083+0+/usr/local/www/db/= text/2014/freebsd-stable/20140907.freebsd-stable Aside from fsyncing the files, I suspect just running "upgrade" waiting for a long time before doing reboot might prevent it from happening. I is likely relevant that the single partition on hte system is a 500GB SU+J UFS= . I need to research a bit on how freebsd does things as well as possible interaction with the large SU+J partition. I was already uncomfortable about the SU+J but went with it due to the time it would otherwise take to fsck the 500GB disk. -- R. Kevin Oberman, Network Engineer, Retired rkoberman@gmail.com From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 02:56:27 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 012AB40F; Thu, 16 Oct 2014 02:56:26 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id AC31090B; Thu, 16 Oct 2014 02:56:26 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id CF03620E7088B; Thu, 16 Oct 2014 02:56:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: * X-Spam-Status: No, score=2.0 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 52D6A20E70885; Thu, 16 Oct 2014 02:56:24 +0000 (UTC) Message-ID: <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> From: "Steven Hartland" To: "Mark Martinec" , , References: <54372173.1010100@ijs.si> <644FA8299BF848E599B82D2C2C298EA7@multiplay.co.uk> <54372EBA.1000908@ijs.si> <543731F3.8090701@ijs.si> <543AE740.7000808@ijs.si> <6E01BBEDA9984CCDA14F290D26A8E14D@multiplay.co.uk> <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@mu ltiplay.co.uk> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Thu, 16 Oct 2014 03:56:23 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 02:56:27 -0000 ----- Original Message ----- From: "Steven Hartland" > After bisecting the kernel versions in stable/10 the problem commit > appears to be: > https://svnweb.freebsd.org/base?view=revision&revision=268650 > > Removing it or using a pool without async_destory enabled prevents > the leak. > > More debugging tomorrow. Fix for this has now been committed: https://svnweb.freebsd.org/changeset/base/273158 I'm already talking with re@ to get this in to the 10.1 release. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 07:55:25 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CCA2E469; Thu, 16 Oct 2014 07:55:25 +0000 (UTC) Received: from odin.blazingdot.com (odin.blazingdot.com [204.109.60.170]) by mx1.freebsd.org (Postfix) with ESMTP id A9FC88B0; Thu, 16 Oct 2014 07:55:25 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 124AB130C8F; Thu, 16 Oct 2014 03:55:19 -0400 (EDT) Date: Thu, 16 Oct 2014 03:55:19 -0400 From: Marcus Reid To: Steven Hartland Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Message-ID: <20141016075518.GA14459@blazingdot.com> References: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Mark Martinec , freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 07:55:25 -0000 On Thu, Oct 16, 2014 at 03:56:23AM +0100, Steven Hartland wrote: > Fix for this has now been committed: > https://svnweb.freebsd.org/changeset/base/273158 > > I'm already talking with re@ to get this in to the 10.1 release. Thank you for that. I looked at your thread on the illumos zfs list, and from what I gather, if you aren't wedged into a state where you have to import read-only, you don't have to worry about leaked data in your pool, correct? I always have a small number of 'deferred free' blocks that's always somewhere between 8 and 10: 9 108K 15.5K 108K 12.0K 6.97 0.00 deferred free Also, if you run 'zdb -bb ' on a live pool, you can get a bunch of: leaked space: vdev 0, offset 0x16464c2000, size 1048576 ... and then: block traversal size 14586265600 != alloc 14667571200 (leaked 81305600) which I believe is normal and unrelated. Marcus From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 08:50:55 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D610CB2; Thu, 16 Oct 2014 08:50:55 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id 1BBC7F1A; Thu, 16 Oct 2014 08:50:54 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id C1F9F20E7088C; Thu, 16 Oct 2014 08:50:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 9238D20E70885; Thu, 16 Oct 2014 08:50:51 +0000 (UTC) Message-ID: <2307495EDFE14A4FB34374307B3EBC55@multiplay.co.uk> From: "Steven Hartland" To: "Marcus Reid" References: <14ADE02801754E028D9A0EAB4A16527E@multiplay.co.uk> <543C3C47.4010208@ijs.si> <138CF459AA0B41EB8CB4E11B3DE932CF@multiplay.co.uk> <543D0953.1070604@ijs.si> <8F4036C658724468B34B20CCBA658E43@multiplay.co.uk> <520ED11771DE40FBAEFCC066D2FF0441@multiplay.co.uk> <20141016075518.GA14459@blazingdot.com> Subject: Re: zpool import hangs when out of space - Was: zfs pool import hangs on [tx->tx_sync_done_cv] Date: Thu, 16 Oct 2014 09:50:48 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Mark Martinec , freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 08:50:55 -0000 ----- Original Message ----- From: "Marcus Reid" > On Thu, Oct 16, 2014 at 03:56:23AM +0100, Steven Hartland wrote: >> Fix for this has now been committed: >> https://svnweb.freebsd.org/changeset/base/273158 >> >> I'm already talking with re@ to get this in to the 10.1 release. > > Thank you for that. I looked at your thread on the illumos zfs list, > and from what I gather, if you aren't wedged into a state where you have > to import read-only, you don't have to worry about leaked data in your > pool, correct? > > I always have a small number of 'deferred free' blocks that's always > somewhere between 8 and 10: > > 9 108K 15.5K 108K 12.0K 6.97 0.00 deferred free > > Also, if you run 'zdb -bb ' on a live pool, you can get a bunch > of: > > leaked space: vdev 0, offset 0x16464c2000, size 1048576 > ... > > and then: > > block traversal size 14586265600 != alloc 14667571200 (leaked 81305600) > > which I believe is normal and unrelated. Yep thats normal. Regards Steve From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 10:26:54 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 473223DE for ; Thu, 16 Oct 2014 10:26:54 +0000 (UTC) Received: from balrog.mythic-beasts.com (balrog.mythic-beasts.com [IPv6:2a00:1098:0:82:1000:0:2:1]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF910D15 for ; Thu, 16 Oct 2014 10:26:53 +0000 (UTC) Received: from [130.209.247.112] (port=56036 helo=mangole.dcs.gla.ac.uk) by balrog.mythic-beasts.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1XeiGg-0006M5-9E; Thu, 16 Oct 2014 11:26:52 +0100 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: System hang on shutdown when running freebsd-update From: Colin Perkins In-Reply-To: Date: Thu, 16 Oct 2014 11:26:47 +0100 Message-Id: <1951041E-D133-4301-B07C-5D4B5A17C521@csperkins.org> References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> To: Kevin Oberman X-Mailer: Apple Mail (2.1878.6) X-BlackCat-Spam-Score: -28 X-Mythic-Debug: Threshold = On = X-Spam-Status: No, score=-2.9 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 10:26:54 -0000 On 15 Oct 2014, at 18:23, Kevin Oberman wrote: > On Wed, Oct 15, 2014 at 2:40 AM, Colin Perkins = wrote: > On 14 Oct 2014, at 18:09, Kevin Oberman wrote: > > I thought that this was just a fluke, but it has now happened three = times, > > so I guess it's now out of the "fluke" class. > > > > I have upgraded several times recently to each 10.1 BETA and RC. = After the > > first install pass t install the kernel and modules, the system = shutdown > > freezes at the very end. I see the buffers synced to the disks and = get the > > "All buffers synced" message. Then it just hangs. The disks are not = marked > > as clean and are fscked after a reset and boot. > > > > There is not much between the "All buffers synced" message and the = call to > > vfs_unmountall(), so I suspect it is hanging in that call. I admit = that I > > am pretty much lost whenever I look at the VFS code and I have not = put a > > lot of effort going further. Just hoping that someone familiar with = it > > might have an idea. > > > > I have tried several reboots and all run normally. The problem only = seems > > to appear when upgrading the OS. It happened repeatedly when I tried = to > > reboot before doing the second "install" pass of freebsd-update, but = not > > after, so the kernel and world are not in sync. I am baffled as to = what > > could be going on, but it means I need to be at the system (a baby = server) > > when I upgrade, but not every time I upgrade. I know it happened on = the > > 10.0-RELEAASE to 10.1-BETA1 and 10.1-RC1 to 10.1-RC2 upgrades. > > > > Has anyone else seen this? >=20 > I=92m seeing the same behaviour, most recently when moving to 10.1-RC1 = (haven=92t gone to -RC2 yet). The system is: >=20 > FreeBSD 10.1-RC1 #0 r272463: Fri Oct 3 01:47:10 UTC 2014 > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) = 20140512 > CPU: AMD Opteron(TM) Processor 6274 (2200.05-MHz = K8-class CPU) > Origin =3D "AuthenticAMD" Id =3D 0x600f12 Family =3D 0x15 Model =3D = 0x1 Stepping =3D 2 > = Features=3D0x178bfbff > = Features2=3D0x1e98220b > AMD Features=3D0x2e500800 > AMD = Features2=3D0x1c9bfff > TSC: P-state invariant, performance statistics > real memory =3D 549755813888 (524288 MB) > avail memory =3D 534559084544 (509795 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: <041112 APIC1739> > FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs > FreeBSD/SMP: 4 package(s) x 16 core(s) > =85 >=20 > I do have IPMI loaded, unlike the other reports. >=20 > -- > Colin Perkins > https://csperkins.org/ >=20 > Paul Koch replied privately with a pointer to a seemingly unrelated = message he sent to stable last month. Take a look at the several = paragraphs at the end starting with "On a side note". I'm suspicious = that the generation of the large upgrade on /var during the "upgrade" = pass is causing the delay. It fits pretty well and, in normal operation, = my server would never see this issue at all. >=20 > = https://docs.freebsd.org/cgi/getmsg.cgi?fetch=3D326083+0+/usr/local/www/db= /text/2014/freebsd-stable/20140907.freebsd-stable >=20 > Aside from fsyncing the files, I suspect just running "upgrade" = waiting for a long time before doing reboot might prevent it from = happening. I is likely relevant that the single partition on the system = is a 500GB SU+J UFS. The update to -RC2 worked without problem on my system. I did, however, = wait until vmstat showed the disks being idle after running = freebsd-update (this only took a few minutes). That said, when updating to -RC1 previously I left the box to sit for = maybe an hour at the =93All buffers synced=94 stage before giving up and = resetting it, so if it was just waiting for the disks to sync, it was = taking a very long time to do so.=20 > I need to research a bit on how freebsd does things as well as = possible interaction with the large SU+J partition. I was already = uncomfortable about the SU+J but went with it due to the time it would = otherwise take to fsck the 500GB disk. > -- > R. Kevin Oberman, Network Engineer, Retired > rkoberman@gmail.com --=20 Colin Perkins https://csperkins.org/ From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 10:32:20 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5F3C1570 for ; Thu, 16 Oct 2014 10:32:20 +0000 (UTC) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F0161DED for ; Thu, 16 Oct 2014 10:32:19 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id n3so4273066wiv.15 for ; Thu, 16 Oct 2014 03:32:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=lFvfCm+Ldv1lIfqTZqtDYbFfB2wiZWfXeN71hEpI8UY=; b=TsxFFDPRBfU7FB0Va5nfVXO0oVGPB+Vkl1rNgo5tnDxwMQJ6sQrjFCVsXXNDi1oUIp 6j5DuCCdXYf4dW3rK4b3kPFAZyRQSAQKhSyS8mOrv3SPJUmArtsGTgIO0lwuNjXtMcC8 gDKVxuxh7IL4+aOnf4hAKI+QMyb7qBr55Lj3qZ4dofZwYfUdp2/u38xyIqlVDxRwLhTa fNyD46MDZHusYhPtY2fgUW2QAu56psI7Bw18FTMidXT5AJQ+7sG0+cq/VQ3uFku99zoz cgXwAuBQCWwuuo0C7tsujAn1Q9QJP6Ty0wA8PC9BmTp0fqzY8/8HuGpNel+Ncna2N5m/ cnSg== MIME-Version: 1.0 X-Received: by 10.194.61.99 with SMTP id o3mr713545wjr.54.1413455536319; Thu, 16 Oct 2014 03:32:16 -0700 (PDT) Received: by 10.194.190.78 with HTTP; Thu, 16 Oct 2014 03:32:16 -0700 (PDT) Date: Thu, 16 Oct 2014 18:32:16 +0800 Message-ID: Subject: UEFI boot on Mac for 10.1 USB img? From: Ben Woods To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 10:32:20 -0000 Unfortunately, the FreeBSD 10.1-RC2 USB image does not boot properly on Mac. All loader messages are seen, but once it has loaded the kernel you don't see any more boot messages. The system has loaded (as evidenced by flashing lights / noises / ctrl+alt+dlt working), but the screen still shows the last kernel loading messages. Refer to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193745 How can we help testing? Is this targeted for fixing prior to 10.1-RELEASE? -- From: Benjamin Woods woodsb02@gmail.com From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 14:10:08 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F8A94A0 for ; Thu, 16 Oct 2014 14:10:08 +0000 (UTC) Received: from mail-lb0-x231.google.com (mail-lb0-x231.google.com [IPv6:2a00:1450:4010:c04::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09A0AA9D for ; Thu, 16 Oct 2014 14:10:07 +0000 (UTC) Received: by mail-lb0-f177.google.com with SMTP id w7so2892702lbi.36 for ; Thu, 16 Oct 2014 07:10:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=K5DYSVK6nwhgzedIAXDHqU+887o4TG1fD0P4iJMNKTU=; b=hnO2p793RjcR61ItxKHxZDdVq937MoD60//uuGZ3cWIOhBl2sJvCibT+zkDnOISbXy OPsVznl6CHEvFrFOhtgbabZ5BoazVizNmDmO1W/CZeQfjSCQmGYtY+wbe6wXjO+p1cjc d58CeD6f0A6iQLSWjnGtrho1VdpWOBkFmN67Nk/iT2F1WGi5PdX0OrCX3W/PGoTuYfGw 8u7jEAn8i6PKF/lMifRwhTctU56cC0bwzpLTpW2FuAcz10hF83jwk61Koe68fnC+Ai6r x9amXtAVwclOkrsrzgNpuV0k1Y2vaYS0j+Fwpaenrer5Cp/8+sgueT6mZiSJpak1152+ 9xvg== X-Received: by 10.152.21.39 with SMTP id s7mr2062742lae.0.1413468605974; Thu, 16 Oct 2014 07:10:05 -0700 (PDT) Received: from blazon-pc.rw.local ([78.84.244.14]) by mx.google.com with ESMTPSA id nb7sm7810570lbb.43.2014.10.16.07.10.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Oct 2014 07:10:04 -0700 (PDT) Message-ID: <543FD1BB.9010501@gmail.com> Date: Thu, 16 Oct 2014 17:10:03 +0300 From: Alnis Morics User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Subject: "msk0: watchdog timeout" still present in 10.1-RC2 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 14:10:08 -0000 Hello, Several months ago we were talking about Marvell NIC not working, and the problem remained unsolved. So far, I have experienced it only in FreeBSD 10 (no problem in 9.X-RELEASE or 11-CURRENT), and only in amd64. So I hoped in 10.1 it would be solved but no: suffice to scp a 5MB file through this interface, and the "watchdog timeout" is there. uname -a: FreeBSD myhost.mydomain.tld 10.1-RC2 FreeBSD 10.1-RC2 #0 r272876: Fri Oct 10 01:12:21 UTC 2014 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 pciconf -lv: [..] mskc0@pci0:9:0:0: class=0x020000 card=0xc072144d chip=0x435411ab rev=0x00 hdr=0x00 vendor = 'Marvell Technology Group Ltd.' device = '88E8040 PCI-E Fast Ethernet Controller' class = network subclass = ethernet _____ Alnis From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 15:29:48 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E9E64A0E for ; Thu, 16 Oct 2014 15:29:48 +0000 (UTC) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A4D05FF for ; Thu, 16 Oct 2014 15:29:48 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 66C6325D387C; Thu, 16 Oct 2014 15:29:38 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 6D15BC770AE; Thu, 16 Oct 2014 15:29:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id LdRdBf8NXZAY; Thu, 16 Oct 2014 15:29:36 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6] (orange-tun0-ula.sbone.de [IPv6:fde9:577b:c1a9:4420:cabc:c8ff:fe8b:4fe6]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 69503C77071; Thu, 16 Oct 2014 15:29:35 +0000 (UTC) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: System hang on shutdown when running freebsd-update From: "Bjoern A. Zeeb" In-Reply-To: Date: Thu, 16 Oct 2014 15:29:33 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> To: Kevin Oberman X-Mailer: Apple Mail (2.1878.6) Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 15:29:49 -0000 On 15 Oct 2014, at 17:23 , Kevin Oberman wrote: I have seen similar hangs on a ZFS-only system (even without = freebsd-update I think). I have often not been patient enough to wait = for the reboot to happen but I do remember that one time I walked away = and came back and the system was still stuck, got distracted and by the = time I got back again, it had rebooted. I wonder if anyone actually had been patient enough to wait an hour or = two, just to see if the system moves forward all by itself? =97=20 Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983 From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 15:39:51 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B709CE0F for ; Thu, 16 Oct 2014 15:39:51 +0000 (UTC) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) by mx1.freebsd.org (Postfix) with ESMTP id 899F478B for ; Thu, 16 Oct 2014 15:39:51 +0000 (UTC) Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id 7B3D15A9F25; Thu, 16 Oct 2014 15:39:50 +0000 (UTC) Date: Thu, 16 Oct 2014 15:39:50 +0000 From: Brooks Davis To: "Bjoern A. Zeeb" Subject: Re: System hang on shutdown when running freebsd-update Message-ID: <20141016153950.GK91488@spindle.one-eyed-alien.net> References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LYw3s/afESlflPpp" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Kevin Oberman , FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 15:39:51 -0000 --LYw3s/afESlflPpp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 16, 2014 at 03:29:33PM +0000, Bjoern A. Zeeb wrote: > On 15 Oct 2014, at 17:23 , Kevin Oberman wrote: >=20 > I have seen similar hangs on a ZFS-only system (even without freebsd-upda= te I think). I have often not been patient enough to wait for the reboot = to happen but I do remember that one time I walked away and came back and t= he system was still stuck, got distracted and by the time I got back again,= it had rebooted. >=20 > I wonder if anyone actually had been patient enough to wait an hour or tw= o, just to see if the system moves forward all by itself? I left a VM I'd upgraded from 10.0 to 10.1-RC1 for serveral hours by accident and it didn't make progress. -- Brooks --LYw3s/afESlflPpp Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlQ/5sUACgkQXY6L6fI4GtTKXQCaAwXRUQ6SThCJVdFHLdNwN1Je Pv0An2ybl90VhGHOB3CNge5VZoAvc4XO =d3kT -----END PGP SIGNATURE----- --LYw3s/afESlflPpp-- From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 16:43:55 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D8DACFD1 for ; Thu, 16 Oct 2014 16:43:55 +0000 (UTC) Received: from gwave1.banym.de (gwave1.banym.de [212.72.74.40]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9943BE78 for ; Thu, 16 Oct 2014 16:43:54 +0000 (UTC) Received: from [192.168.120.70] (pd907d022.dip0.t-ipconnect.de [217.7.208.34]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by gwave1.banym.de (Postfix) with ESMTP id BF71E15C37; Thu, 16 Oct 2014 18:23:13 +0200 (CEST) Message-ID: <543FF0FE.8020102@banym.de> Date: Thu, 16 Oct 2014 18:23:26 +0200 From: Dominik Zajac User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.2 MIME-Version: 1.0 To: Ben Woods , freebsd-stable@freebsd.org Subject: Re: UEFI boot on Mac for 10.1 USB img? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Banym-MailScanner-Information: Please contact the ISP for more information X-Banym-MailScanner-ID: BF71E15C37.A6258 X-Banym-MailScanner: Found to be clean X-Banym-MailScanner-From: banym@banym.de X-Spam-Status: No X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 16:43:55 -0000 I am facing same situation with a Mac Book Pro late 2008. Am 16.10.2014 um 12:32 schrieb Ben Woods: > Unfortunately, the FreeBSD 10.1-RC2 USB image does not boot properly on > Mac. All loader messages are seen, but once it has loaded the kernel you > don't see any more boot messages. The system has loaded (as evidenced by > flashing lights / noises / ctrl+alt+dlt working), but the screen still > shows the last kernel loading messages. > Refer to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193745 > > How can we help testing? Is this targeted for fixing prior to 10.1-RELEASE? > > -- > From: Benjamin Woods > woodsb02@gmail.com > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 17:33:36 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C4C99E4 for ; Thu, 16 Oct 2014 17:33:36 +0000 (UTC) Received: from list.sco.com (list.sco.com [69.36.163.214]) by mx1.freebsd.org (Postfix) with SMTP id D93E43DA for ; Thu, 16 Oct 2014 17:33:35 +0000 (UTC) Received: (qmail 25465 invoked from network); 16 Oct 2014 17:33:34 -0000 Received: from nimbus.nj.sco.com (69.36.163.214) by tasmania.ut.sco.com with SMTP; 16 Oct 2014 17:33:34 -0000 Received: from [127.0.0.1] (ip-132147089 [132.147.83.89]) by nimbus.nj.sco.com (8.12.9/UW7.1.3) with ESMTP id s9GHXWHH029693; Thu, 16 Oct 2014 13:33:33 -0400 (EDT) Message-ID: <5440016B.6030107@xinuos.com> Date: Thu, 16 Oct 2014 13:33:31 -0400 From: John Wolfe User-Agent: Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Dominik Zajac , Ben Woods , freebsd-stable@freebsd.org Subject: Re: UEFI boot on Mac for 10.1 USB img? References: <543FF0FE.8020102@banym.de> In-Reply-To: <543FF0FE.8020102@banym.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 17:33:36 -0000 I see the same thing on a Dell T320 when using the UEFI boot options for both the uefi-memstick and the uefi-dvd1. The same media will, of course boot when using the BIOS boot option on this hardware. -- John On 10/16/2014 12:23 PM, Dominik Zajac wrote: > I am facing same situation with a Mac Book Pro late 2008. > > > Am 16.10.2014 um 12:32 schrieb Ben Woods: >> Unfortunately, the FreeBSD 10.1-RC2 USB image does not boot properly on >> Mac. All loader messages are seen, but once it has loaded the kernel you >> don't see any more boot messages. The system has loaded (as evidenced by >> flashing lights / noises / ctrl+alt+dlt working), but the screen still >> shows the last kernel loading messages. >> Refer to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193745 >> >> How can we help testing? Is this targeted for fixing prior to >> 10.1-RELEASE? >> >> -- >> From: Benjamin Woods >> woodsb02@gmail.com >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> "freebsd-stable-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 17:36:54 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from hub.FreeBSD.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7694CB0F; Thu, 16 Oct 2014 17:36:53 +0000 (UTC) Date: Thu, 16 Oct 2014 17:36:49 +0000 From: Glen Barber To: Konstantin Belousov Subject: Re: Heads-up: Possible regression between 10.0-RELEASE and 10.1-BETA1 with '/ on ZFS' setup Message-ID: <20141016173649.GD36695@hub.FreeBSD.org> References: <20141004024011.GC1199@hub.FreeBSD.org> <20141004160052.GR26076@kib.kiev.ua> <20141004170137.GA1171@hub.FreeBSD.org> <1684676.g0E56GkSvf@overcee.wemm.org> <20141004174600.GU26076@kib.kiev.ua> <20141004211530.GB1171@hub.FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="io440R/5/oAEBxrM" Content-Disposition: inline In-Reply-To: <20141004211530.GB1171@hub.FreeBSD.org> X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event User-Agent: Mutt/1.5.23 (2014-03-12) Cc: FreeBSD Release Engineering Team , Steven Hartland , freebsd-stable@freebsd.org, Peter Wemm X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 17:36:54 -0000 --io440R/5/oAEBxrM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Oct 04, 2014 at 05:15:30PM -0400, Glen Barber wrote: > On Sat, Oct 04, 2014 at 08:46:00PM +0300, Konstantin Belousov wrote: > > On Sat, Oct 04, 2014 at 10:03:48AM -0700, Peter Wemm wrote: > > > > If we cannot increase KSTACK_PAGES by default, do we have any > > > > alternative solution outside of suggesting to avoid using ZFS on i3= 86 > > > > with more than one disk? > > >=20 > > > When zfs creates its kthreads it can specify how much stack it needs.= For=20 > > > i386 it could ask for more for the zfs threads. Its not a good optio= n but its=20 > > > better than more stack for everything when it's already easy to run o= ut=20 > > > without zfs. > >=20 > > This one probably happens in the init thread, not some of the zfs hord. > > Glen did not show the backtrace from ddb yet (I hope that ddb did not > > regressed and can step over double-fault boundary). > >=20 > > We could specifically increment the init thread stack size as well, but > > I have no idea if normal VFS calls into ZFS are affected and cause over= flow > > for the normal threads after the multitasking is fired. >=20 > As soon as I get the kernel built with debugging support, I'll be able > to get the backtrace. This, however, is proving to be non-trivial. >=20 I was finally able to get a VM configured in a way that would allow getting a ddb backtrace after panic. Screenshots of the panic are available here: https://people.freebsd.org/~gjb/i386-zfs/zfs-i386-1.png https://people.freebsd.org/~gjb/i386-zfs/zfs-i386-2.png https://people.freebsd.org/~gjb/i386-zfs/zfs-i386-3.png https://people.freebsd.org/~gjb/i386-zfs/zfs-i386-4.png I'll leave the VM in its current state in case more information is needed. I also want to point out that 10.0-RELEASE also crashes upon mounting the ZFS filesystems when the machine boots after an unclean shutdown, so although the behavior of "when" the machine crashes has changed, I am not certain this is a regression since 10.0-RELEASE. Glen --io440R/5/oAEBxrM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJUQAIxAAoJEAMUWKVHj+KTe74QAIOC5FrHY4AQgeeNXvpJTv1+ hRKRZbUjB60Fk/N1930+Zo7vi5Vbok2o8B+VoBK9oAlyFJPscelS1Dck4nG69vWI CX+QEqG/V0o8LAF376ov8exe2GLvur2cAV6PywG76kSmghASvcJ7JrEjnHg9wdBr K/vGmWf+QUSn62NtML/jpIPst0SRjI2aplPLTnYWMNidok8XojvkBl6iiu0qVL8c mIhjV0A7JJru0NOHsFgnBSUcPtxV8b3qA6liROCzwzQcgTjB212SeSKtNQmyrlp3 Vz+Do5lwR61LIpsipBhpvCY05lkmh3w+3/BFfKj5GFtO0xZF6T7XKmlV3uEr1mA5 gRqEIlLOBUAmXnx82DvClTSme+27Mh9nMx0JpyQ6fMgxdJdlR3wAp8Nwe2ytY16m PN8yfDP+VdA7PGymkph7ZlsMeYVbXfWmOwVHvS3wWZYmpbmluiyW8Mzz+wXteKJY Stlun6/K1ly2PAe+ZgyeVNdAEK7aN7gmFlK0PsFkYVqiXWSBnr0l66z5gwHG6+SJ BaOVGgTZ49YVHkIUdrXjtfauNZ83OsGHurwlWOrapAthJ6lHxAEx2ln9oGz/M6iN Ggq4Bvw502xt8dUm9WWH5RZJq6DVikzTrJqWJJp4cx50+RZU7h2AmBnYrV2ssqND jwKju9LvFo6fGUVCLqt+ =eQw9 -----END PGP SIGNATURE----- --io440R/5/oAEBxrM-- From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 18:06:36 2014 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from hub.FreeBSD.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 640714C2; Thu, 16 Oct 2014 18:06:36 +0000 (UTC) Date: Thu, 16 Oct 2014 18:06:31 +0000 From: Glen Barber To: freebsd-stable@FreeBSD.org Subject: Heads-up: ABI incompatibility fix in releng/10.1 Message-ID: <20141016180631.GE36695@hub.FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="TBbVzKyKt0Rtmm0o" Content-Disposition: inline X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event User-Agent: Mutt/1.5.23 (2014-03-12) Cc: FreeBSD Release Engineering Team X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 18:06:37 -0000 --TBbVzKyKt0Rtmm0o Content-Type: text/plain; charset=us-ascii Content-Disposition: inline It was brought to our attention a few days ago that there was an ABI incompatibility introduced in the stable/10 branch about two months ago (prior to the code freeze for the 10.1-RELEASE cycle). This incompatibility was reverted in releng/10.1 as of r273169. This change was not reverted in the stable/10 branch, pending a more long term solution for situations where we must bump a shared library version in a stable/ branch. After upgrading an existing 10.1 system after r273169, third-party applications that link against libopie.so.8 will need to be reinstalled to link against the corrected version, libopie.so.7. If using binary packages from the official FreeBSD pkg(8) repositories, no action should be necessary. This information will also be included in the 10.1-RC3 announcement email when the builds are ready in a few days. Thanks. Glen --TBbVzKyKt0Rtmm0o Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJUQAknAAoJEAMUWKVHj+KTb0cP/i06LXBxPifvq3dR2qQF2SgM KOGfRTcWWSvE7Zgcp5EppOqxlvbST162LZm5phjqvCIZYT9CpA6LrfYjRZWK+n9w K8Oia/6K/cUbjegskT5DusGqs3yCDtFK6Hx6vcv4UM8B0FMfq5wW+bjC2633/lAF A0b1APc6XhHZUrN6KZs+67d+2aIx06XSMBc38ONhq+KNnOiKm+nY9zagh9nd6FF+ vVomelIcUNIMNAgcYIqZw3nNJYgu2G5CYHcAh9UmEHOCZ9GH82zsCt7bIF+GUaaY nV2dQTKr0hm+REHmUoWIXra2TbFqeTDa3mxgxGFFA5VE6uNCQzR+2A2TJziEyRzV lB1ecVAQeP2e3WZSNBWKsYCWSI7vXH3rhmSIKgEKJtyYtpG5tutPPwIPI0kSVbTX hjKtklQJSmsKrgkA9JgPka/s/fxReMbAMgRUdPI7zJwkYmw8IuMg28Z6rOb44+L3 I6Ycjwm9+ilZk9gvbBeFQuJq4Drn9DJf3dDspVWLfBZK7hENC1mNrqDfDEO7VUWk +hjHlPVt19c8AnO6Jzl5yCSIrAnx2RMIu5uYZv7wkPLmi8VS5V8PIvB2UGFy20im rrPikHxTfmN/wpiCfG9i0dq7xIigetunXuc9yFJP2aiCPSc9f4Kw0lsQUEZcvp+C 74zdIJIOQudn6gW3jHts =2VZ0 -----END PGP SIGNATURE----- --TBbVzKyKt0Rtmm0o-- From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 19:37:34 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 61D995C1 for ; Thu, 16 Oct 2014 19:37:34 +0000 (UTC) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C359313 for ; Thu, 16 Oct 2014 19:37:34 +0000 (UTC) Received: by mail-oi0-f43.google.com with SMTP id u20so3235294oif.2 for ; Thu, 16 Oct 2014 12:37:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qnCIviO2T5gVgvHKukuE3+KPnEYGXwRZtLg51odv86M=; b=OJPLtiau26FMdjg2SFoSZzwlkTqJHkuaZqjPJlt52oMhd9IR1Y8GGuAe2ckhqqvA/s kROwf2l2wC3rtJRMvtv2SgOvYTeTSrTnaFA8Ahp7CaAfooIcwfiocko8nwWjrWb4wYCT Uoe+C/O8+uXL9ZczMvrhnpj05mHrLJ6Mm/6R9sFgS25e3rVhhcihIqBV/+swnFUxXHw/ OkUfBnrplxoL5smL0ccjwRhopMf1UQMeELAhkbBQ2jFZDQqLVhBSgyfF4g6e6ycw+x8u YoA8oy5OM3OW05Sd85NpMfsbipKg00SxHVvS1Su3yathS+yDHxPjHdjX7wzni3V7bUVZ KVrg== MIME-Version: 1.0 X-Received: by 10.182.125.165 with SMTP id mr5mr2702937obb.71.1413488253501; Thu, 16 Oct 2014 12:37:33 -0700 (PDT) Received: by 10.202.178.8 with HTTP; Thu, 16 Oct 2014 12:37:33 -0700 (PDT) In-Reply-To: References: Date: Thu, 16 Oct 2014 12:37:33 -0700 Message-ID: Subject: Re: nvd disk on nvme controller not detected at boot-time From: Jim Harris To: Maikel Verheijen Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 19:37:34 -0000 On Tue, Oct 14, 2014 at 1:47 AM, Maikel Verheijen < Maikel.Verheijen@redwood.com> wrote: > Hi list! > > We recently purchased a HP DL380e G8 server to serve as our backup server > with an Intel P3600 that uses the nvme interface. We added load_nvme=3D"Y= ES" > and load_nvd=3D"YES" to our loader.conf, and they both get loaded, howeve= r > the ssd disk device is not detected at boot. When we unload and reload th= e > nvd module the disk does get detected. Is there a way to see if the > load-order is correct? We added verbose_loading=3D"YES" to the loader.con= f, > but dmesg doesn't show me the actual loading. > > One thing I did see related to the nvme in the dmesg (full dmesg attached= ) > output is this: > > Hi Maikel, Which version of FreeBSD are you using? I believe both of these issues have been fixed on HEAD and merged to stable/9 and stable/10. 9.3 and 10.1-RC would have these fixes. 9.2 and 10.0 would not. > nvme0: mem 0xfbdf0000-0xfbdf3fff irq 16 at device > 0.0 on pci3 > =E2=80=A6 > nvme0: SET FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000080 cdw11:0000000= 0 > nvme0: INTERNAL DEVICE ERROR (00/06) sqid:0 cid:9 cdw0: > This was fixed in r263277. Incidentally, the error message here is harmless and not associated with the primary issue you have described above= . > > After unloading and reloading the nvd module the disk does get detected: > > nvd0: NVMe namespace > nvd0: 381554MB (781422768 512 byte sectors) > The load on boot issue was fixed in r263310. Best regards, -Jim > > Is there anyone that might give me some pointers on how to get the nvd0 > device loaded consistently at boot time? > Thanks in advance, > Kind regards, > Maikel Verheijen > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 20:19:21 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 072D055D; Thu, 16 Oct 2014 20:19:21 +0000 (UTC) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "smarthost.sentex.ca", Issuer "smarthost.sentex.ca" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9FAFD9C0; Thu, 16 Oct 2014 20:19:20 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.9/8.14.9) with ESMTP id s9GKJISV049744; Thu, 16 Oct 2014 16:19:18 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <5440284A.2020301@sentex.net> Date: Thu, 16 Oct 2014 16:19:22 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Rumen Telbizov , "Alexander V. Chernikov" Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table References: <542AAA3C.1080803@ipfw.ru> <542AE376.6000003@FreeBSD.org> <542AFAE3.9030705@FreeBSD.org> <20141001135124.GM73266@glebius.int.ru> <542C20D7.3070606@sentex.net> <20141001171646.GQ73266@glebius.int.ru> <542C5529.9030800@FreeBSD.org> <7AB35136-89AA-48F2-8B0E-1BA3DCD4A6BA@ipfw.ru> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.75 Cc: "Alexander V. Chernikov" , Gleb Smirnoff , brian@freebsd.org, "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 20:19:21 -0000 Hi, Have these two patches been merged to RELENG_10 ? ---Mike On 10/2/2014 6:25 PM, Rumen Telbizov wrote: > Hello everyone, > > Alexander, Gleb: good news guys. The problem seems to be resolved! Good > job.Here's what I have. > > 1. I upgraded the backup firewall to 10-STABLE (r272435), applied Gleb's > patch of pf_table.c and rebuilt. Just as reported by Gleb, the leak seemed > to be reduced but it was still present and leaked a little bit of memory > upon every PF reload. > > 2. In addition to 1) I applied the second patch from Alexander on radix.c > and rebuilt the kernel. After the reboot it seems like the amount of memory > consumed by the routetbl grows a little bit temporarily after reload and > then drops back to the previous, stable amount shortly after. > > Gleb, Alexander, thank you very much for your help. I appreciate your quick > response in providing patches for this problem. I hope those 2 patches get > tested by more people and get MFC'ed into 10-STABLE soon. -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 20:29:48 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 96B9E966; Thu, 16 Oct 2014 20:29:48 +0000 (UTC) Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F4AEAD0; Thu, 16 Oct 2014 20:29:48 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id x19so4268348ier.39 for ; Thu, 16 Oct 2014 13:29:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2LNKoO1UzLtHKTbaakdY3O6fi8QgtSjoBVLGoc07z7Y=; b=WltyKS6NxAgqWl3izyMhXbn05/OnV22i+mgNOlVhzebEBgTfaK+OLgpq5OIgx5s6hy TFuvQI3NkxwriL/ITWIQCGCYTO2jXNxeOLn2TPL5FGpydDK/DvROw09uDCN88Ika3KFT u+AxbqPXPOPh0aRFqIunJS2stpy1DuDTdU2z4R+ppgrtZOnnTbbjL5c5nkqJ//NFJmnH HsLHBCyAB12rXk5TJ4O5bgu+Cgpu8tL7vCQ9s3EW/dfThPu9oUx9XJcR75nY6f1DPm3G QtNeY1cgmpplCTxxAXUWU5oTGB3v+89PiMIp+z9izGS5d6Zrh1602K/7h6AwBo8fp01/ F4rA== MIME-Version: 1.0 X-Received: by 10.107.162.80 with SMTP id l77mr4267961ioe.21.1413491387579; Thu, 16 Oct 2014 13:29:47 -0700 (PDT) Received: by 10.50.87.130 with HTTP; Thu, 16 Oct 2014 13:29:47 -0700 (PDT) In-Reply-To: <5440284A.2020301@sentex.net> References: <542AAA3C.1080803@ipfw.ru> <542AE376.6000003@FreeBSD.org> <542AFAE3.9030705@FreeBSD.org> <20141001135124.GM73266@glebius.int.ru> <542C20D7.3070606@sentex.net> <20141001171646.GQ73266@glebius.int.ru> <542C5529.9030800@FreeBSD.org> <7AB35136-89AA-48F2-8B0E-1BA3DCD4A6BA@ipfw.ru> <5440284A.2020301@sentex.net> Date: Thu, 16 Oct 2014 13:29:47 -0700 Message-ID: Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table From: Rumen Telbizov To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "Alexander V. Chernikov" , Gleb Smirnoff , brian@freebsd.org, "freebsd-stable@freebsd.org" , "Alexander V. Chernikov" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 20:29:48 -0000 As far as I know they have not. I needed to apply them manually after svn up to the latest 10 stable. It would be great if they become part of 10.1-RELEASE. I have been running with them for 2 weeks now both my firewalls and everything has been rock solid. On Thu, Oct 16, 2014 at 1:19 PM, Mike Tancsa wrote: > Hi, > Have these two patches been merged to RELENG_10 ? > > ---Mike > > On 10/2/2014 6:25 PM, Rumen Telbizov wrote: > >> Hello everyone, >> >> Alexander, Gleb: good news guys. The problem seems to be resolved! Good >> job.Here's what I have. >> >> 1. I upgraded the backup firewall to 10-STABLE (r272435), applied Gleb's >> patch of pf_table.c and rebuilt. Just as reported by Gleb, the leak seemed >> to be reduced but it was still present and leaked a little bit of memory >> upon every PF reload. >> >> 2. In addition to 1) I applied the second patch from Alexander on radix.c >> and rebuilt the kernel. After the reboot it seems like the amount of >> memory >> consumed by the routetbl grows a little bit temporarily after reload and >> then drops back to the previous, stable amount shortly after. >> >> Gleb, Alexander, thank you very much for your help. I appreciate your >> quick >> response in providing patches for this problem. I hope those 2 patches get >> tested by more people and get MFC'ed into 10-STABLE soon. >> > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > -- Rumen Telbizov Unix Systems Administrator From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 20:46:38 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91FD1FE7; Thu, 16 Oct 2014 20:46:38 +0000 (UTC) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D930CB3; Thu, 16 Oct 2014 20:46:37 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.9/8.14.9) with ESMTP id s9GKkZRH088208 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 17 Oct 2014 00:46:35 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.9/8.14.9/Submit) id s9GKkZ5Z088207; Fri, 17 Oct 2014 00:46:35 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Fri, 17 Oct 2014 00:46:35 +0400 From: Gleb Smirnoff To: Rumen Telbizov Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table Message-ID: <20141016204635.GZ73266@glebius.int.ru> References: <20141001135124.GM73266@glebius.int.ru> <542C20D7.3070606@sentex.net> <20141001171646.GQ73266@glebius.int.ru> <542C5529.9030800@FreeBSD.org> <7AB35136-89AA-48F2-8B0E-1BA3DCD4A6BA@ipfw.ru> <5440284A.2020301@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "Alexander V. Chernikov" , brian@freebsd.org, "freebsd-stable@freebsd.org" , "Alexander V. Chernikov" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 20:46:38 -0000 On Thu, Oct 16, 2014 at 01:29:47PM -0700, Rumen Telbizov wrote: R> As far as I know they have not. I needed to apply them manually after svn R> up to the latest 10 stable. It would be great if they become part of R> 10.1-RELEASE. I have been running with them for 2 weeks now both my R> firewalls and everything has been rock solid. I just merged both patches to stable/10. Will do a request for merge into the release. -- Totus tuus, Glebius. From owner-freebsd-stable@FreeBSD.ORG Thu Oct 16 21:29:32 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AFF78156 for ; Thu, 16 Oct 2014 21:29:32 +0000 (UTC) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78807227 for ; Thu, 16 Oct 2014 21:29:32 +0000 (UTC) Received: by mail-oi0-f45.google.com with SMTP id i138so3372854oig.18 for ; Thu, 16 Oct 2014 14:29:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=l0Uw5kUiqW8oiRIEcV6uNuCgGYkX/XAT2gNfnjZE+Yg=; b=0j4xJnToKsTN8RQ0Z97PT/C+DOsK3BwX7NfqbN0p8hA9T5rOJwdozI2NqFsGLO97Ow NE/m84+JDoTgStLoQ8Meftv7a6gW4ZfbNad9MVb+ZGegCRsqPlS5zGg4OiBgCaeRqjHd 4WMhoy0x2a13sqq5eObefig5eLxZESlqGwJf9AZeXl1XF8GyEiJDgtcQBkWBYdmu0dYZ bsufZVxHzKBbZ/czZSro1D2K1gQkPZkqiLpHitV7XMb5zAAlJX2ovzET6cbbqHTiZIyj 4FkZ58U+a5mVG8GhSI3TLxVE0KYY+l11oFx1OV55vFYvR2Ii0+sD9a7wMYST2Ug2SX7U Ffyw== MIME-Version: 1.0 X-Received: by 10.182.246.40 with SMTP id xt8mr3185235obc.59.1413494971813; Thu, 16 Oct 2014 14:29:31 -0700 (PDT) Received: by 10.202.178.8 with HTTP; Thu, 16 Oct 2014 14:29:31 -0700 (PDT) In-Reply-To: References: Date: Thu, 16 Oct 2014 14:29:31 -0700 Message-ID: Subject: Re: nvd disk on nvme controller not detected at boot-time From: Jim Harris To: Maikel Verheijen Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 21:29:32 -0000 On Thu, Oct 16, 2014 at 1:43 PM, Maikel Verheijen < Maikel.Verheijen@redwood.com> wrote: > Hi Jim, > > I'm currently running FreeBSD 10.0 with updates from freebsd-update. > Would FreeBSD 10.1-RC be safe for a production environment or would it be > better to wait until the real release? The machine is not used for > production yet, but we do want to go live with it in the near future. I > currently use a =E2=80=9Cworkaround=E2=80=9D, I=E2=80=99m not loading ZFS= at boot time but use a > custom RC script to load the nvd module, test if the nvd0 disk is there, > then load the ZFS module. That seems to work for now, but having it work > =E2=80=9Cout of the box=E2=80=9D would be better, and a lot less prone to= manual mistakes > made by me :) > > I cannot recommend whether 10.1-RC would be safe for a production environment, but you could certainly use 10.1-RC to at least confirm it fixes the issues described. You could also apply these two patches to your 10.0 kernel tree and rebuild just nvme.ko. Best regards, -Jim From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 00:04:48 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6CA3FE32; Fri, 17 Oct 2014 00:04:48 +0000 (UTC) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.69.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cell.glebius.int.ru", Issuer "cell.glebius.int.ru" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DB966376; Fri, 17 Oct 2014 00:04:46 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.9/8.14.9) with ESMTP id s9H04ioG089490 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 17 Oct 2014 04:04:44 +0400 (MSK) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.9/8.14.9/Submit) id s9H04hwE089489; Fri, 17 Oct 2014 04:04:43 +0400 (MSK) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Fri, 17 Oct 2014 04:04:43 +0400 From: Gleb Smirnoff To: Rumen Telbizov Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table Message-ID: <20141017000443.GF73266@glebius.int.ru> References: <542C20D7.3070606@sentex.net> <20141001171646.GQ73266@glebius.int.ru> <542C5529.9030800@FreeBSD.org> <7AB35136-89AA-48F2-8B0E-1BA3DCD4A6BA@ipfw.ru> <5440284A.2020301@sentex.net> <20141016204635.GZ73266@glebius.int.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141016204635.GZ73266@glebius.int.ru> User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "Alexander V. Chernikov" , brian@freebsd.org, "freebsd-stable@freebsd.org" , "Alexander V. Chernikov" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 00:04:48 -0000 On Fri, Oct 17, 2014 at 12:46:35AM +0400, Gleb Smirnoff wrote: T> On Thu, Oct 16, 2014 at 01:29:47PM -0700, Rumen Telbizov wrote: T> R> As far as I know they have not. I needed to apply them manually after svn T> R> up to the latest 10 stable. It would be great if they become part of T> R> 10.1-RELEASE. I have been running with them for 2 weeks now both my T> R> firewalls and everything has been rock solid. T> T> I just merged both patches to stable/10. Will do a request for merge T> into the release. Merged to releng/10.1 -- Totus tuus, Glebius. From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 02:19:59 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EA5BFB3 for ; Fri, 17 Oct 2014 02:19:59 +0000 (UTC) Received: from mx12-out4.antispamcloud.com (mx12-out4.antispamcloud.com [46.165.232.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7B79CF for ; Fri, 17 Oct 2014 02:19:58 +0000 (UTC) Received: from net06.redwood.com ([91.233.6.246] helo=mail.redwood.com) by mx12.antispamcloud.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1XertA-0005F5-0P; Thu, 16 Oct 2014 22:43:15 +0200 Received: from exceu2.rwd.lan ([fe80::e42a:a959:5640:381a]) by exceu2.rwd.lan ([fe80::e42a:a959:5640:381a%10]) with mapi id 14.01.0438.000; Thu, 16 Oct 2014 22:43:11 +0200 From: Maikel Verheijen To: Jim Harris Subject: Re: nvd disk on nvme controller not detected at boot-time Thread-Topic: nvd disk on nvme controller not detected at boot-time Thread-Index: AQHP54ttNqAodkLuEEmDqzIhnS+RzpwzAKSAgAAz24A= Date: Thu, 16 Oct 2014 20:43:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, nl-NL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.5.141003 x-originating-ip: [10.31.1.9] MIME-Version: 1.0 X-Filter-ID: s0sct1PQhAABKnZB5plbIV6jvtGXzsRLyDxHGPnOSosTzNYXe+HZr4WowsEZlxIc1tZWiYINMnD4 agKXMg0NtBlCFCOA7kScelqCJFYKursKpTLhc3scs8lIU/v8u+rLonq0jcl6y1B4xrk1sOdNSWS2 BpcZp5MieEYXpn6TUALZI/Mr0BskU9hAYZ3Ymm+Y77VjOXfkZEVwPyGgPVOYnvLF3n+yP+L+jcBn IV6ubcZIn5ObN5P05LGWFQlP+VmPBIIZGAiIku3Cx523pDpt3x48vBcmKs9GYcu0FLW7EWPlgTl6 fJxyntEfhZCKje4Zx0bFDp3mGK8Y7ug+kgE2tol2UKe9D6SGnFl2eHdGD56roVjH0lzFpZqUycuK 16uwBmJ8Q0iagechgWqTL3vIA1Jh9nHh5xBVP9pvcu1pgYqSoC5VPlOdrUr8E6UB0EPg3VyTyNUA us5wK5dhtldfyDQttY3c9bWbvNE9580khkH/asDO0V6SUvEY03codo+QZd7VZNM714SJ3DaW+K4f VsdxRa5Rvm6JFujnJaOZwdtixEWhmbBXdtEvx2xJnvEp4oIKD0HBt4sWrEIlE8VJalYFvf25LVON YbYifH5OzZANAwxW2AvNktcRxKgEznEpH50YYHE3/09I4g/iRQecfi+orTeREDUbREcPuO+7veLj r9jD+xEtCAUWMNpEVddcVmUctY5jL1s3weViixsNRA== X-Report-Abuse-To: spam@mx99.antispamcloud.com X-Filter-Fingerprint: IFrWXGses7OKB5S5G8/dJbmLbQULIXlEhuVRudChxyrJUWjZ8+qhjyB23tbDuyLOYL8Ff78gYsez 4Rl08xudmXi4esCQ0R1MchVjt7wblGlvhFgW0MjUMRkF5sMCDfftTXNFDzN17hnrWeZYOJvLq0Ic WjZ+XcEjj/7Pkld0zkmvziDInX9WdMov2kn2yXjdwv61T+KDYyYtREgszdyFwv8IxCB3p/oCKvxr eyISh3JGb7OS5oVgiO+kDxZrVPLz3MmEGC2PrUKqLq5WmHK+Nw== X-Originating-IP: 91.233.6.246 X-Spampanel-Domain: redwood.com X-Spampanel-Username: sysadmin Authentication-Results: antispamcloud.com; auth=pass (login) smtp.auth=sysadmin X-Spampanel-Outgoing-Class: ham X-Spampanel-Outgoing-Evidence: Combined (0.07) X-Recommended-Action: accept Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 02:19:59 -0000 Hi Jim, On Tue, Oct 14, 2014 at 1:47 AM, Maikel Verheijen > wrote: Hi list! We recently purchased a HP DL380e G8 server to serve as our backup server w= ith an Intel P3600 that uses the nvme interface. We added load_nvme=3D"YES"= and load_nvd=3D"YES" to our loader.conf, and they both get loaded, however= the ssd disk device is not detected at boot. When we unload and reload the= nvd module the disk does get detected. Is there a way to see if the load-o= rder is correct? We added verbose_loading=3D"YES" to the loader.conf, but d= mesg doesn't show me the actual loading. One thing I did see related to the nvme in the dmesg (full dmesg attached) = output is this: Hi Maikel, Which version of FreeBSD are you using? I believe both of these issues hav= e been fixed on HEAD and merged to stable/9 and stable/10. 9.3 and 10.1-RC= would have these fixes. 9.2 and 10.0 would not. I'm currently running FreeBSD 10.0 with updates from freebsd-update. Would = FreeBSD 10.1-RC be safe for a production environment or would it be better = to wait until the real release? The machine is not used for production yet,= but we do want to go live with it in the near future. I currently use a = =93workaround=94, I=92m not loading ZFS at boot time but use a custom RC sc= ript to load the nvd module, test if the nvd0 disk is there, then load the = ZFS module. That seems to work for now, but having it work =93out of the bo= x=94 would be better, and a lot less prone to manual mistakes made by me :) nvme0: mem 0xfbdf0000-0xfbdf3fff irq 16 at device 0.0= on pica =85 nvme0: SET FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000080 cdw11:00000000 nvme0: INTERNAL DEVICE ERROR (00/06) sqid:0 cid:9 cdw0: This was fixed in r263277. Incidentally, the error message here is harmles= s and not associated with the primary issue you have described above. Thanks, this was the only thing I thought that might be relevant to the iss= ue, glad to hear it=92s harmless. After unloading and reloading the nvd module the disk does get detected: nvd0: NVMe namespace nvd0: 381554MB (781422768 512 byte sectors) The load on boot issue was fixed in r263310. That is good news, that means we can probably drop my rc script in the near= future and use the proper loading procedures. Best regards, -Jim Thanks again and warm regards, Maikel Verheijen From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 08:10:49 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF16DA9D; Fri, 17 Oct 2014 08:10:49 +0000 (UTC) Received: from mail-yk0-x22c.google.com (mail-yk0-x22c.google.com [IPv6:2607:f8b0:4002:c07::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8374934C; Fri, 17 Oct 2014 08:10:49 +0000 (UTC) Received: by mail-yk0-f172.google.com with SMTP id 19so131898ykq.31 for ; Fri, 17 Oct 2014 01:10:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=6pHgM4ewGGmMIsEJknaV3l3dKBM89/tegH9AL4LAu88=; b=w4TNzUZJdZnFNLvux770E+kfeXnHn2I6VrVMhzqwo+Ni/WDONZ9mFSxIYFMgZDE5hW +P13ZShd+g1MbS/FQ61e0L2kN42CidgsuKbvaPr3aE8fcBUkDdK+gm1w5mK90Oc6D9TT NlJyqyJiNZbbr9Nr3vg46sjqz1JQK9izT0nx/iLyEe292QYjVPcUW+Z/oFF48gUXAObv pImAU8sz1zzyAWNwgjM1N0NpdjgEJ+4qcHkg98lCripngDcKYo3LpBysKHrlLGYaFMjo O+vAQ4ihC7qNWrdDqWAmALvAYJOmel2Yj63YFRJUzlEQcoKO8VTRZ1fJDt53rmw528T3 lV/A== MIME-Version: 1.0 X-Received: by 10.236.16.230 with SMTP id h66mr2162342yhh.145.1413533448616; Fri, 17 Oct 2014 01:10:48 -0700 (PDT) Received: by 10.170.156.139 with HTTP; Fri, 17 Oct 2014 01:10:48 -0700 (PDT) In-Reply-To: <201410132047.s9DKlGxD030176@gw.catspoiler.org> References: <43D94A22FBD2477FBBDEFF16C0088DDA@multiplay.co.uk> <201410132047.s9DKlGxD030176@gw.catspoiler.org> Date: Fri, 17 Oct 2014 09:10:48 +0100 Message-ID: Subject: Re: getting to 4K disk blocks in ZFS From: krad To: Don Lewis Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-stable , Steven Hartland , fullermd@over-yonder.net X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 08:10:49 -0000 your also forgetting you can enable compression, which on mail files will give you a large ratio On 13 October 2014 21:47, Don Lewis wrote: > On 13 Oct, Steven Hartland wrote: > > ----- Original Message ----- > > From: "Matthew D. Fuller" > > > > > >> On Mon, Oct 13, 2014 at 11:48:27AM -0700 I heard the voice of > >> Darren Pilgrim, and lo! it spake thus: > >>> > >>> If the default is 4k and (for the limited time they're still common) > >>> you use true 512b disks, you can waste space. Sure, but how much > >>> space? > >> > >> The median file in /usr/ports is 408 bytes. Over 90% of the files are > >> under 2k, which means the wastage for them is over 100% (before > >> counting what gain compression might get). A little offhand mathery > >> says it's about 78% extra overhead on the whole. > >> > >> And that includes the almost hundred megs (over 22% of the total size > >> of the FS) for the INDEX.db, plus the ~90 megs of the flat INDEX files > >> (another 20%). If you pull those out, the overhead is 130%. > >> > >> > >> (To be sure, relatively few people have ports trees eating most of > >> their space, but still; it's pretty pathological. I for one did > >> decide some years back to always force 4k on any new FSen to make > >> future life simpler, accepting the bloat, but it's there.) > > > > And thats before you add the overhead if your running RAIDZ... > > > > A good read on this is > > http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/ > > This is a timely subject. I'm planning on moving my Cyrus imap mail > spool from a 4K/1K UFS filesystem to a three drive raidz1. It looks > like the UFS fragmentation overhead is about 2.4%. ZFS ashift=12 > increases that to about 17%. Combine that with raidz and now the > overhead is about 40%. Ouch! > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 08:37:59 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABFDF1B5 for ; Fri, 17 Oct 2014 08:37:59 +0000 (UTC) Received: from mx12-out4.antispamcloud.com (mx12-out4.antispamcloud.com [46.165.232.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3008C7FE for ; Fri, 17 Oct 2014 08:37:58 +0000 (UTC) Received: from net06.redwood.com ([91.233.6.246] helo=mail.redwood.com) by mx12.antispamcloud.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1Xf32n-0006TN-CB; Fri, 17 Oct 2014 10:37:56 +0200 Received: from exceu2.rwd.lan ([fe80::e42a:a959:5640:381a]) by exceu2.rwd.lan ([fe80::e42a:a959:5640:381a%10]) with mapi id 14.01.0438.000; Fri, 17 Oct 2014 10:37:35 +0200 From: Maikel Verheijen To: Jim Harris Subject: Re: nvd disk on nvme controller not detected at boot-time Thread-Topic: nvd disk on nvme controller not detected at boot-time Thread-Index: AQHP54ttNqAodkLuEEmDqzIhnS+RzpwzAKSAgAAz24D//+tugIAA3CsA Date: Fri, 17 Oct 2014 08:37:34 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, nl-NL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.5.141003 x-originating-ip: [10.31.1.9] MIME-Version: 1.0 X-Filter-ID: s0sct1PQhAABKnZB5plbIV6jvtGXzsRLyDxHGPnOSosTzNYXe+HZr4WowsEZlxIc1tZWiYINMnD4 agKXMg0NtBlCFCOA7kScelqCJFYKursKpTLhc3scs8lIU/v8u+rLonq0jcl6y1B4xrk1sOdNSWS2 BpcZp5MieEYXpn6TUALZI/Mr0BskU9hAYZ3Ymm+Y77VjOXfkZEVwPyGgPVOYnuwpsCmjI/A5aeNz 6rc5B79wU+ui+vam5szJFPwaR4DgBIIZGAiIku3Cx523pDpt3x48vBcmKs9GYcu0FLW7EWPlgTl6 fJxyntEfhZCKje4Zx0bFDp3mGK8Y7ug+kgE2tol2UKe9D6SGnFl2eHdGD56roVjH0lzFpZqUycuK 16uwBmJ8Q0iagechgWqTL3vIA1Jh9nHh5xBVP9pvcu1pgYqSoC5VPlOdrUr8E6UB0EPg3VyTyNUA us5wK5dhtldfyDQttY3c9bWbvNE9580khkH/asDO0V6SUvEY03codo+QZd7VZNM714SJ3DaW+K4f VsdxRa5Rvm6JFujnJaOZwdtixEWhmbBXdtEvx2xJnvEp4oIKD0HBt4sWrEIlE8VJalYFvf25LVON YbYifH5OzZANAwxW2AvNktcRxKgEznEpH50YYHE3/09I4g/iRQecfi+orTeREDUbREcPuO+7veLj r9jD+xEtCAUWMNpEVddcVmUctY5jL1s3weViixsNRA== X-Report-Abuse-To: spam@mx99.antispamcloud.com X-Filter-Fingerprint: IFrWXGses7OKB5S5G8/dJbmLbQULIXlEhuVRudChxyrJUWjZ8+qhjyB23tbDuyLOYL8Ff78gYsez 4Rl08xudmXi4esCQ0R1MchVjt7wblGlvhFgW0MjUMRkF5sMCDfftTXNFDzN17hnrWeZYOJvLq0Ic WjZ+XcEjj/7Pkld0zkmvziDInX9WdMov2kn2yXjdwv61T+KDYyYtREgszdyFwv8IxCB3p/oCKvxr eyISh3JGb7OS5oVgiO+kDxZrVPLz3MmEGC2PrUKqLq5WmHK+Nw== X-Originating-IP: 91.233.6.246 X-Spampanel-Domain: redwood.com X-Spampanel-Username: sysadmin Authentication-Results: antispamcloud.com; auth=pass (login) smtp.auth=sysadmin X-Spampanel-Outgoing-Class: ham X-Spampanel-Outgoing-Evidence: Combined (0.03) X-Recommended-Action: accept Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 08:37:59 -0000 Hi Jim, On Thu, Oct 16, 2014 at 1:43 PM, Maikel Verheijen > wrote: Hi Jim, I'm currently running FreeBSD 10.0 with updates from freebsd-update. Would = FreeBSD 10.1-RC be safe for a production environment or would it be better = to wait until the real release? The machine is not used for production yet,= but we do want to go live with it in the near future. I currently use a = =93workaround=94, I=92m not loading ZFS at boot time but use a custom RC sc= ript to load the nvd module, test if the nvd0 disk is there, then load the = ZFS module. That seems to work for now, but having it work =93out of the bo= x=94 would be better, and a lot less prone to manual mistakes made by me :) I cannot recommend whether 10.1-RC would be safe for a production environme= nt, but you could certainly use 10.1-RC to at least confirm it fixes the is= sues described. You could also apply these two patches to your 10.0 kernel tree and rebuild= just nvme.ko. Thank you for your quick reply. We=92ll test 10.1-rc, then wait for 10.1 re= lease. We prefer to use the freebsd-update tool to keep the system up to d= ate rather than to build our own kernel. Best regards, -Jim Best regards, Maikel From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 09:58:43 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 13825511 for ; Fri, 17 Oct 2014 09:58:43 +0000 (UTC) Received: from balrog.mythic-beasts.com (balrog.mythic-beasts.com [IPv6:2a00:1098:0:82:1000:0:2:1]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCC72F5B for ; Fri, 17 Oct 2014 09:58:42 +0000 (UTC) Received: from [130.209.247.112] (port=54919 helo=mangole.dcs.gla.ac.uk) by balrog.mythic-beasts.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1Xf4Iw-00051i-6K; Fri, 17 Oct 2014 10:58:38 +0100 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: System hang on shutdown when running freebsd-update From: Colin Perkins In-Reply-To: Date: Fri, 17 Oct 2014 10:58:27 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <64A14F51-E115-4929-8CE9-9F786A43C9A1@csperkins.org> References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> To: "Bjoern A. Zeeb" X-Mailer: Apple Mail (2.1878.6) X-BlackCat-Spam-Score: -28 X-Mythic-Debug: Threshold = On = X-Spam-Status: No, score=-2.9 Cc: Kevin Oberman , FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 09:58:43 -0000 On 16 Oct 2014, at 16:29, Bjoern A. Zeeb = wrote: > On 15 Oct 2014, at 17:23 , Kevin Oberman wrote: > I have seen similar hangs on a ZFS-only system (even without = freebsd-update I think). I have often not been patient enough to wait = for the reboot to happen but I do remember that one time I walked away = and came back and the system was still stuck, got distracted and by the = time I got back again, it had rebooted. >=20 > I wonder if anyone actually had been patient enough to wait an hour or = two, just to see if the system moves forward all by itself? I gave up waiting after about an hour with my system that showed the = problem. --=20 Colin Perkins https://csperkins.org/ From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 17:23:47 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A1D2FF3 for ; Fri, 17 Oct 2014 17:23:47 +0000 (UTC) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 14D363EC for ; Fri, 17 Oct 2014 17:23:47 +0000 (UTC) Received: by mail-ie0-f181.google.com with SMTP id at20so1137879iec.26 for ; Fri, 17 Oct 2014 10:23:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=uqiy4g9x1F9oG26dWGV9hwvolVyxn49upkg2Hg2MZmw=; b=IGmrSxL/QYTI2MqRtwJDecJNyZOj6/jZh17K/QpM+xeU22KETOahyBPPPgtF3NB0GI Jq5VPX7WZnWy0C+4lZWxkCmsQRze3VAP1aejQTl1+7lXuB8SlD53mYU+bnL28Dyy69KA q7jSYxK2k8irlWIB4NjZcBKpmFWne9tinHgSyGyivdKejb8SmJt7jHH5t9NEex6amuoQ 1dc26/qHX/x9unitFfQayw+NdEwyDfR3yJT1t5IC/nkL4Db+2bsvaT1CjEB/inY5eMV2 OauSDyrp2wjTE/oUb5bcEw+FhFZ6bM61uJNgLmf8JCLVvSHqkCCVtNjAfrusIv+juq0t ZgCQ== MIME-Version: 1.0 X-Received: by 10.107.162.16 with SMTP id l16mr4298345ioe.54.1413566625926; Fri, 17 Oct 2014 10:23:45 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.107.11.141 with HTTP; Fri, 17 Oct 2014 10:23:45 -0700 (PDT) In-Reply-To: <64A14F51-E115-4929-8CE9-9F786A43C9A1@csperkins.org> References: <7479DC25-4451-4940-AFE7-7C81D08206D4@csperkins.org> <64A14F51-E115-4929-8CE9-9F786A43C9A1@csperkins.org> Date: Fri, 17 Oct 2014 10:23:45 -0700 X-Google-Sender-Auth: P4twvRlIo5Uha1UyDu4wKDHowBs Message-ID: Subject: Re: System hang on shutdown when running freebsd-update From: Kevin Oberman To: Colin Perkins Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "Bjoern A. Zeeb" , FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 17:23:47 -0000 On Fri, Oct 17, 2014 at 2:58 AM, Colin Perkins wrote: > On 16 Oct 2014, at 16:29, Bjoern A. Zeeb > wrote: > > On 15 Oct 2014, at 17:23 , Kevin Oberman wrote: > > I have seen similar hangs on a ZFS-only system (even without > freebsd-update I think). I have often not been patient enough to wait for > the reboot to happen but I do remember that one time I walked away and came > back and the system was still stuck, got distracted and by the time I got > back again, it had rebooted. > > > > I wonder if anyone actually had been patient enough to wait an hour or > two, just to see if the system moves forward all by itself? > > I gave up waiting after about an hour with my system that showed the > problem. > My server is production, so waiting is not an option, nor is using it for testing. -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com