From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 18:22:17 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8705A1065670 for ; Sun, 13 Nov 2011 18:22:17 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id 212FC8FC0C for ; Sun, 13 Nov 2011 18:22:16 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 4C60F153434; Sun, 13 Nov 2011 19:22:15 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rcyd7ucllUHX; Sun, 13 Nov 2011 19:22:13 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id 29032153433; Sun, 13 Nov 2011 19:22:13 +0100 (CET) Message-ID: <4EC00ACF.4050803@digiware.nl> Date: Sun, 13 Nov 2011 19:22:07 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Joshua Boyd References: <4EBB97DF.3020803@digiware.nl> <20111110095041.GA73812@icarus.home.lan> <4EBBBACE.3020900@digiware.nl> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "stable@freebsd.org" , "Vogel, Jack" , Jeremy Chadwick Subject: Re: em0 watchdog timeout X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 18:22:17 -0000 On 2011-11-10 23:25, Joshua Boyd wrote: > On Thu, Nov 10, 2011 at 6:51 AM, Willem Jan Withagen > wrote: > > em0@pci0:0:25:0: class=0x020000 card=0x10bd15d9 > chip=0x10bd8086 rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xdf900000, size > 131072, enabled > bar [14] = type Memory, range 32, base 0xdf924000, size 4096, > enabled > bar [18] = type I/O Port, range 32, base 0x1820, size 32, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > cap 13[e0] = PCI Advanced Features: FLR TP > > > And note that this problem only raises it nasty head very few weeks... > > > I have had the same problem, as shown here: > > http://lists.freebsd.org/pipermail/freebsd-stable/2011-June/063092.html > > According to your pciconf output, your card either doesn't support > MSI-X, or you have MSI-X disabled. > > Check the hw.pci.enable_msix sysctl and make sure that it is set to 1. > Also check to make sure there aren't any BIOS settings blocking MSI-X. > > Apparently the older Intel gigabit cards don't support MSI-X, and as > such get starved. Upgraded to a new bios, but that does not help either. Now the trick question will be: IF I get a new servertype PCI-E ethernet card, would that get me an MSI-X ethernet device. --WjW From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 18:51:06 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3377106566C for ; Sun, 13 Nov 2011 18:51:06 +0000 (UTC) (envelope-from brett@lariat.net) Received: from lariat.net (lariat.net [66.62.230.51]) by mx1.freebsd.org (Postfix) with ESMTP id 79E5C8FC13 for ; Sun, 13 Nov 2011 18:51:06 +0000 (UTC) Received: from WildRover.lariat.net (IDENT:ppp1000.lariat.net@lariat.net [66.119.58.2] (may be forged)) by lariat.net (8.9.3/8.9.3) with ESMTP id LAA08702 for ; Sun, 13 Nov 2011 11:51:03 -0700 (MST) Message-Id: <201111131851.LAA08702@lariat.net> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 13 Nov 2011 11:51:00 -0700 To: stable@freebsd.org From: Brett Glass Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: Subject: Building servers this weekend. Recommendations? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 18:51:06 -0000 I need to build up some new servers this weekend, and my first choice for the OS would be FreeBSD 9.0-RC2 if it were available. Alas, it isn't, and there's no sign of when it's coming (neither the Release Engineering page or the Wiki is anywhere near up to date). RC1 has a few bugs that I'd like to avoid, as does 8.2-RELEASE. Recommendations? --Brett Glass From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 19:07:27 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C22FB106564A for ; Sun, 13 Nov 2011 19:07:27 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 5875B8FC17 for ; Sun, 13 Nov 2011 19:07:26 +0000 (UTC) Received: by wwg14 with SMTP id 14so5387830wwg.31 for ; Sun, 13 Nov 2011 11:07:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1X0jizIkaJdogzSmYyABtUd7EdJTBqhQ7x3h8zeb8ow=; b=NM4aNjXm6ZHDqIi6avonIyIqZ7KdMst1HdHDSQ7LN165Zylz4LTG5EmnRutSrYogiu Ixjv5Z63BkeCljqgAlnXq77edAVCxMDmucYPC0SHtx8neulmM3eD2iegSIIcfAsnAD7d tQZ32IGWtQNvH71hg8GoIvYtewAa/O06cDUM0= MIME-Version: 1.0 Received: by 10.180.109.106 with SMTP id hr10mr22697364wib.9.1321209611251; Sun, 13 Nov 2011 10:40:11 -0800 (PST) Received: by 10.180.95.229 with HTTP; Sun, 13 Nov 2011 10:40:11 -0800 (PST) In-Reply-To: <4EC00ACF.4050803@digiware.nl> References: <4EBB97DF.3020803@digiware.nl> <20111110095041.GA73812@icarus.home.lan> <4EBBBACE.3020900@digiware.nl> <4EC00ACF.4050803@digiware.nl> Date: Sun, 13 Nov 2011 10:40:11 -0800 Message-ID: From: Jack Vogel To: Willem Jan Withagen Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "stable@freebsd.org" , Joshua Boyd , "Vogel, Jack" , Jeremy Chadwick Subject: Re: em0 watchdog timeout X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 19:07:27 -0000 On Sun, Nov 13, 2011 at 10:22 AM, Willem Jan Withagen wrote: > On 2011-11-10 23:25, Joshua Boyd wrote: > >> On Thu, Nov 10, 2011 at 6:51 AM, Willem Jan Withagen > > wrote: >> >> em0@pci0:0:25:0: class=0x020000 card=0x10bd15d9 >> chip=0x10bd8086 rev=0x02 hdr=0x00 >> vendor = 'Intel Corporation' >> device = 'Intel 82566DM Gigabit Ethernet Adapter (82566DM)' >> class = network >> subclass = ethernet >> bar [10] = type Memory, range 32, base 0xdf900000, size >> 131072, enabled >> bar [14] = type Memory, range 32, base 0xdf924000, size 4096, >> enabled >> bar [18] = type I/O Port, range 32, base 0x1820, size 32, enabled >> cap 01[c8] = powerspec 2 supports D0 D3 current D0 >> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message >> cap 13[e0] = PCI Advanced Features: FLR TP >> >> >> And note that this problem only raises it nasty head very few weeks... >> >> >> I have had the same problem, as shown here: >> >> http://lists.freebsd.org/**pipermail/freebsd-stable/2011-** >> June/063092.html >> >> According to your pciconf output, your card either doesn't support >> MSI-X, or you have MSI-X disabled. >> >> Check the hw.pci.enable_msix sysctl and make sure that it is set to 1. >> Also check to make sure there aren't any BIOS settings blocking MSI-X. >> >> Apparently the older Intel gigabit cards don't support MSI-X, and as >> such get starved. >> > > Upgraded to a new bios, but that does not help either. > > Now the trick question will be: > IF I get a new servertype PCI-E ethernet card, would that get me > an MSI-X ethernet device. > > There is no 'trick' to it :) The only MSIX capable device that uses the em driver is 82574. But if you go with igb (82575 and beyond) they are all MSIX and multiqueue capable. Jack From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 19:25:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20463106566B for ; Sun, 13 Nov 2011 19:25:10 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3fd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id 812148FC0A for ; Sun, 13 Nov 2011 19:25:09 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [81.187.76.163]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id pADJP0uO027510 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Sun, 13 Nov 2011 19:25:01 GMT (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.4.1 smtp.infracaninophile.co.uk pADJP0uO027510 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1321212301; bh=SOB0IrVCo1/zK71RNWVbYHx4jsUAFxjYh3ZiH3ylzw0=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Cc; b=JxpzPuuKhcgNktEf67Iq8OQkD2zmj+Om+FZrv4X9k8CrIvJMKdzpUfwojpn2K4UlE RUkUKEDPdwgmV55vIWMJE75wWeE8/w5aFDDwNWckRbBUBVXMtsrEJxFAIys7PiV14H OxJCK+klOsYL0EHVGGmNwzhqqsYnSrUvdU1aQ1rI= Message-ID: <4EC01983.3000704@infracaninophile.co.uk> Date: Sun, 13 Nov 2011 19:24:51 +0000 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <201111131851.LAA08702@lariat.net> In-Reply-To: <201111131851.LAA08702@lariat.net> X-Enigmail-Version: 1.3.3 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6A836D08D3C9B12F81D54219" X-Virus-Scanned: clamav-milter 0.97.3 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,SPF_FAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: Building servers this weekend. Recommendations? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 19:25:10 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6A836D08D3C9B12F81D54219 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 13/11/2011 18:51, Brett Glass wrote: > I need to build up some new servers this weekend, and my first choice > for the OS would be FreeBSD 9.0-RC2 if it were available. Alas, it > isn't, and there's no sign of when it's coming (neither the Release > Engineering page or the Wiki is anywhere near up to date). RC1 has a fe= w > bugs that I'd like to avoid, as does 8.2-RELEASE. Recommendations? It's here already: maggot:~:% uname -a FreeBSD maggot.black-earth.co.uk 9.0-RC2 FreeBSD 9.0-RC2 #1: Sat Nov 12 01:06:59 GMT 2011 root@maggot.black-earth.co.uk:/usr/obj/usr/src/sys/MAGGOT amd64 Don't know about iso images or the like: this was upgraded from -RC1 by compiling sources. Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enig6A836D08D3C9B12F81D54219 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7AGYsACgkQ8Mjk52CukIx47wCfaGHUf9OrKKc80FWDmSIlEr0o hDMAnA0fncy44RneAD2iOlyJg3SJDRHd =lq4g -----END PGP SIGNATURE----- --------------enig6A836D08D3C9B12F81D54219-- From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 20:02:23 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7487E106567F for ; Sun, 13 Nov 2011 20:02:23 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 370E28FC1B for ; Sun, 13 Nov 2011 20:02:22 +0000 (UTC) Received: by iakl21 with SMTP id l21so7234703iak.13 for ; Sun, 13 Nov 2011 12:02:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to; bh=SvV8y+DKf8BdLkFtZtnT0+zttzK/EQyWXM0w0JWwwxk=; b=ItlaCem9jhFTbVgQXaPP8/yfMBvq/8GkOwK8szI6C+XCIfBCRrx/y+J7b++PAQhApy ctLi65U7fphvfzq2wVkm8WrXbEZkhQSMgc7omhqeljlVJPmQ+yRliR5KheJswgBJdrHP F9iczj+0qxUcP6O7btxLgi0IfEcA1XOtkNVWo= Received: by 10.50.173.74 with SMTP id bi10mr21824144igc.4.1321214541244; Sun, 13 Nov 2011 12:02:21 -0800 (PST) Received: from DataIX.net (adsl-99-181-143-66.dsl.klmzmi.sbcglobal.net. [99.181.143.66]) by mx.google.com with ESMTPS id fu10sm14925458igc.6.2011.11.13.12.02.19 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 13 Nov 2011 12:02:20 -0800 (PST) Sender: Jason Hellenthal Received: from DataIX.net (localhost.DataIX.local [127.0.0.1]) by DataIX.net (8.14.5/8.14.5) with ESMTP id pADK2HE2094910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 13 Nov 2011 15:02:17 -0500 (EST) (envelope-from jhell@DataIX.net) Received: (from jhell@localhost) by DataIX.net (8.14.5/8.14.5/Submit) id pADK2HIn094909; Sun, 13 Nov 2011 15:02:17 -0500 (EST) (envelope-from jhell@DataIX.net) Date: Sun, 13 Nov 2011 15:02:17 -0500 From: Jason Hellenthal To: Matthew Seaman Message-ID: <20111113200217.GB90789@DataIX.net> References: <201111131851.LAA08702@lariat.net> <4EC01983.3000704@infracaninophile.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EC01983.3000704@infracaninophile.co.uk> Cc: freebsd-stable@freebsd.org Subject: Re: Building servers this weekend. Recommendations? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 20:02:23 -0000 On Sun, Nov 13, 2011 at 07:24:51PM +0000, Matthew Seaman wrote: > On 13/11/2011 18:51, Brett Glass wrote: > > I need to build up some new servers this weekend, and my first choice > > for the OS would be FreeBSD 9.0-RC2 if it were available. Alas, it > > isn't, and there's no sign of when it's coming (neither the Release > > Engineering page or the Wiki is anywhere near up to date). RC1 has a few > > bugs that I'd like to avoid, as does 8.2-RELEASE. Recommendations? > > It's here already: > > maggot:~:% uname -a > FreeBSD maggot.black-earth.co.uk 9.0-RC2 FreeBSD 9.0-RC2 #1: Sat Nov 12 > 01:06:59 GMT 2011 > root@maggot.black-earth.co.uk:/usr/obj/usr/src/sys/MAGGOT amd64 > > Don't know about iso images or the like: this was upgraded from -RC1 by > compiling sources. > Don't know why you would go with 9.0 on a new server but ok. It's not that hard to build up your own ISO via the process laid out in the docs. make realease is pretty nice. Tweak as neccesary... #!/bin/sh trap 'exit 1' 2 VERSION=8.1-STABLE BRANCH="stable/8" SVNVER=$(svnversion /usr/src) RELNAME=${VERSION}-r${SVNVER} THISBUILD=${RELNAME} ;export THISBUILD MAKEOBJDIRPREFIX=/usr/obj/RELEASE ;export MAKEOBJDIRPREFIX KERNELS=SH4500 ;export KERNELS NUMCPUS=$(($(sysctl -n kern.smp.cpus)*2+(8))) cd /usr/src/release nice make release CHROOTDIR=/usr/obj/RELEASE BUILDNAME=${THISBUILD} \ SVNROOT=file:///exports/nsvn SVNBRANCH=${BRANCH} EXTSRCDIR=/usr/src \ NOPORTS=YES NODOCS=YES EXTPORTSDIR=/usr/ports EXTDOCDIR=/usr/doc \ MAKE_ISOS=YES From owner-freebsd-stable@FreeBSD.ORG Sun Nov 13 22:31:16 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BA76106564A for ; Sun, 13 Nov 2011 22:31:16 +0000 (UTC) (envelope-from brett@lariat.net) Received: from lariat.net (lariat.net [66.62.230.51]) by mx1.freebsd.org (Postfix) with ESMTP id 497028FC08 for ; Sun, 13 Nov 2011 22:31:15 +0000 (UTC) Received: from WildRover.lariat.net (IDENT:ppp1000.lariat.net@lariat.net [66.119.58.2] (may be forged)) by lariat.net (8.9.3/8.9.3) with ESMTP id PAA10188 for ; Sun, 13 Nov 2011 15:31:13 -0700 (MST) Message-Id: <201111132231.PAA10188@lariat.net> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 13 Nov 2011 15:30:46 -0700 To: stable@freebsd.org From: Brett Glass Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: Subject: Re: Building servers this weekend. Recommendations? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Nov 2011 22:31:16 -0000 > Don't know why you would go with 9.0 on a new server I understand what you're saying: normally, we wouldn't use a .0 release in production. But there's a problem. 8.2-RELEASE has serious networking bugs that would cause the servers to lock up. It also has a few other problems (e.g. in timekeeping) that we have seen to cause instability. A small number of these are fixed in 8.2-STABLE (but to use it we'd not only have to build our own unique snapshot but create a machine to build it on). Other problems have been fixed in 9.0-STABLE but have not been backported, perhaps due to the push to get 9.0 out the door. Some will likely never be backported, either because they represent ABI or architectural changes or because the developers' energies are focused on not one but two newer branches. And we need the servers now. See our dilemma? IMHO, this problem stems from moving between major version numbers too quickly rather than having at least 5 or 6 releases on each major branch, leaving the last one or two highly polished so that no one feels compelled to use a .0 release in production. While it probably isn't a good idea to do more than 5 or 6 releases on a branch, it does result in unparalleled stability. (Witness 4.11, which may have been the highest quality release in FreeBSD's history.) Using an official "RC2" build still wouldn't be ideal, because freebsd-update couldn't be used to do a binary upgrade to the release. (We'd still need to cvsup and "make world" on production machines, which we do not like to do.) But at least we wouldn't have to tweak configuration files, and would only have to rebuild those application binaries that were statically linked. So far, it seems like the best option, but I'd be interested in other suggestions. --Brett Glass From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 08:03:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88B65106566C for ; Mon, 14 Nov 2011 08:03:48 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 8E6E28FC12 for ; Mon, 14 Nov 2011 08:03:45 +0000 (UTC) Received: (qmail 83590 invoked by uid 907); 14 Nov 2011 08:03:42 -0000 Received: from jmmacpro.transactionware.com (HELO jmmacpro.transactionware.com) (192.168.1.33) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Mon, 14 Nov 2011 19:03:42 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Jan Mikkelsen In-Reply-To: <4EB9BD9B.8080604@unsane.co.uk> Date: Mon, 14 Nov 2011 19:03:42 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> References: <4EA9E0C3.5080306@unsane.co.uk> <992755CA-6479-4B9A-A3D5-DD5C1871089A@transactionware.com> <4EB1BA7A.2000307@unsane.co.uk> <201111081450.34686.jhb@freebsd.org> <4EB9AC0F.2040209@unsane.co.uk> <4EB9BD9B.8080604@unsane.co.uk> To: John Baldwin X-Mailer: Apple Mail (2.1251.1) Cc: FreeBSD Stable Mailing List , Jeremy Chadwick , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 08:03:48 -0000 Hi, I have just tested mfi on a machine that has just arrived. I'm seeing = the command timeout problem on boot with 9.0-RC1. The message is "mfi0: COMMAND TIMEOUT AFTER 59 seconds", and then = repeats every 30 seconds (with the time changed, obviously). I have tested the 9.0-RC1 ISO, a 9-stable kernel patched with the patch = from the PR I referenced below (also in FreeBSD cvs in revision 1.62 of = src/sys/dev/mfi/mfi.c), and 9-stable kernel with the patch at = www.freebsd.org/~jhb/patches/mfi.patch with the 'pci_alloc_msix' call = changed to 'pci_alloc_msi'. I'm setting up a test with the pci_alloc_msix call unchanged at the = moment, but I don't have anything in /boot/loader.conf, so that I'm not = expecting that to make a difference. This is on a Supermicro X8DTi-F, 48GB memory and an LSI MegaRAID = 9261-8i. 8.2-RELEASE boots fine, dmesg and some output from mfiutil below. Any suggestions gratefully received! Thanks, Jan Mikkelsen On 09/11/2011, at 10:39 AM, Vincent Hoffman wrote: > On 08/11/2011 22:24, Vincent Hoffman wrote: >> On 08/11/2011 19:50, John Baldwin wrote: >>> On Wednesday, November 02, 2011 5:47:38 pm Vincent Hoffman wrote: >>>> On 28/10/2011 04:14, Jan Mikkelsen wrote: >>>>> Hi, >>>>>=20 >>>>> There is a patch linked to from this PR, which seems very similar: >>>>>=20 >>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/140416 >>>>>=20 >>>>> = http://lists.freebsd.org/pipermail/freebsd-scsi/2011-March/004839.html >>>>>=20 >>>>> The problem is also consistent with running mfiutil clearing the = problem. >>>>>=20 >>>>> I'm about to deploy mfi controllers in a similar configuration, so = I'd be=20 >>> very curious about whether the patch fixes the problem for you. >>>> The patch you linked to seems to have removed the stalls, although = I >>>> have only had it running for a day. I'll post if it stalls again = though. >>>>=20 >>>> I did manage to scrounge the use of a Dell r410 with a >>>> LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) >>>> Badged as Dell PERC H700 Adapter >>>>=20 >>>> to test out the patch I originally found but had the same issue as = this post >>>>=20 >>>> = http://lists.freebsd.org/pipermail/freebsd-stable/2011-September/063821.ht= ml >>>>=20 >>>>=20 >>>> I couldnt get the dell to stall in the first place either though so = it >>>> could be a specific firmware version that the issue. >>>>=20 >>>> Anyway thanks for the pointers. >>> Hmm, did you try the patch I had posted from that earlier thread? = It had >>> two changes in it, one was similar to the patch in the PR, the = second added >>> MSI-X support. I've since tweaked it to make the MSI-X support off = by >>> default but possible to enable via loader.conf. Would you be = willing to >>> try the updated patch at www.freebsd.org/~jhb/patches/mfi.patch? >> Hi, >> yes I tried the patch you posted originally, unfortunately the dell >> never finished booting either. The Supermicro is now in production = but >> I'll take the dell up to 9-STABLE and try your updated patch. >>=20 > The patch didnt apply quite cleanly for 9-STABLE, 1 reject as it had > already been applied. >=20 > I have rebooted the dell and it seems happy with the new patch (msi > disabled.) > Booting with > hw.mfi.msix=3D1 in /boot/loader.conf causes the timeouts again and = stops > the boot from completing. > I can give root access to the machine if this would be helpful, I cant > give KVM access though unfortunately. > (but can look in from time to time if needed.) >=20 >=20 > Vince Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE #0: Thu May 26 14:33:45 EST 2011 = root@valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/w= ork/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/GEN= ERIC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (2409.70-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x206c2 Family =3D 6 Model =3D 2c = Stepping =3D 2 = Features=3D0xbfebfbff = Features2=3D0x9ee3fd AMD Features=3D0x2c100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 51543801856 (49156 MB) avail memory =3D 49664753664 (47364 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 16 cpu7 (AP): APIC ID: 17 cpu8 (AP): APIC ID: 18 cpu9 (AP): APIC ID: 19 cpu10 (AP): APIC ID: 20 cpu11 (AP): APIC ID: 21 cpu12 (AP): APIC ID: 32 cpu13 (AP): APIC ID: 33 cpu14 (AP): APIC ID: 34 cpu15 (AP): APIC ID: 35 cpu16 (AP): APIC ID: 36 cpu17 (AP): APIC ID: 37 cpu18 (AP): APIC ID: 48 cpu19 (AP): APIC ID: 49 cpu20 (AP): APIC ID: 50 cpu21 (AP): APIC ID: 51 cpu22 (AP): APIC ID: 52 cpu23 (AP): APIC ID: 53 ioapic1: Changing APIC ID to 7 ioapic2: Changing APIC ID to 8 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 400, 100 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0x4C, should be 0x49 = (20101013/tbutils-354) cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 igb0: port = 0xec00-0xec1f mem = 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 = at device 0.0 on pci1 igb0: Using MSIX interrupts with 9 vectors igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: [ITHREAD] igb0: Ethernet address: 00:25:90:62:20:3e igb1: port = 0xe800-0xe81f mem = 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf98000-0xfaf9bfff irq 40 = at device 0.1 on pci1 igb1: Using MSIX interrupts with 9 vectors igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: [ITHREAD] igb1: Ethernet address: 00:25:90:62:20:3f pcib2: at device 3.0 on pci0 pci2: on pcib2 pcib3: at device 5.0 on pci0 pci3: on pcib3 pcib4: at device 7.0 on pci0 pci4: on pcib4 pcib5: at device 9.0 on pci0 pci5: on pcib5 pci0: at device 20.0 (no driver = attached) pci0: at device 20.1 (no driver = attached) pci0: at device 20.2 (no driver = attached) pci0: at device 20.3 (no driver = attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) pci0: at device 22.2 (no driver attached) pci0: at device 22.3 (no driver attached) pci0: at device 22.4 (no driver attached) pci0: at device 22.5 (no driver attached) pci0: at device 22.6 (no driver attached) pci0: at device 22.7 (no driver attached) uhci0: port 0xcf80-0xcf9f = irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup =3D 0x2f00 usbus0: on uhci0 uhci1: port 0xcf40-0xcf5f = irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup =3D 0x2f00 usbus1: on uhci1 uhci2: port 0xcf20-0xcf3f = irq 19 at device 26.2 on pci0 uhci2: [ITHREAD] uhci2: LegSup =3D 0x2f00 usbus2: on uhci2 ehci0: mem = 0xfaefc000-0xfaefc3ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 uhci3: port 0xcf00-0xcf1f = irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] uhci3: LegSup =3D 0x2f00 usbus4: on uhci3 uhci4: port 0xcec0-0xcedf = irq 19 at device 29.1 on pci0 uhci4: [ITHREAD] uhci4: LegSup =3D 0x2f00 usbus5: on uhci4 uhci5: port 0xcea0-0xcebf = irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] uhci5: LegSup =3D 0x2f00 usbus6: on uhci5 ehci1: mem = 0xfaeda000-0xfaeda3ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus7: EHCI version 1.0 usbus7: on ehci1 pcib6: at device 30.0 on pci0 pci6: on pcib6 vgapci0: mem = 0xf9000000-0xf9ffffff,0xfbefc000-0xfbefffff,0xfb000000-0xfb7fffff irq 16 = at device 4.0 on pci6 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port = 0xcff0-0xcff7,0xcf7c-0xcf7f,0xcfe0-0xcfe7,0xcef4-0xcef7,0xcfa0-0xcfaf,0xcf= 60-0xcf6f irq 19 at device 31.2 on pci0 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci1: port = 0xcee0-0xcee7,0xcef0-0xcef3,0xce70-0xce77,0xce7c-0xce7f,0xce60-0xce6f,0xce= 40-0xce4f irq 19 at device 31.5 on pci0 atapci1: [ITHREAD] ata4: on atapci1 ata4: [ITHREAD] ata5: on atapci1 ata5: [ITHREAD] pcib7: on acpi0 pci128: on pcib7 pcib8: at device 0.0 on pci128 pci129: on pcib8 pcib9: at device 1.0 on pci128 pci130: on pcib9 pcib10: at device 3.0 on pci128 pci131: on pcib10 pcib11: at device 5.0 on pci128 pci132: on pcib11 mfi0: port 0xf800-0xf8ff mem = 0xf8f9c000-0xf8f9ffff,0xf8fc0000-0xf8ffffff irq 50 at device 0.0 on = pci132 mfi0: Megaraid SAS driver Ver 3.00=20 mfi0: 2061 (374583936s/0x0020/info) - Shutdown command received from = host mfi0: 2062 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) mfi0: 2063 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 mfi0: 2064 (boot + 9s/0x0008/info) - Battery Present mfi0: 2065 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 mfi0: 2066 (boot + 9s/0x0020/info) - Board Revision 15A mfi0: 2067 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) mfi0: 2068 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 mfi0: 2069 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) mfi0: 2070 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 mfi0: 2071 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) mfi0: 2072 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 mfi0: 2073 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) mfi0: 2074 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 mfi0: 2075 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) mfi0: 2076 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221104000000,0000000000000000 mfi0: 2077 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) mfi0: 2078 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221105000000,0000000000000000 mfi0: 2079 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) mfi0: 2080 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 mfi0: 2081 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) mfi0: 2082 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 mfi0: 2083 (374610308s/0x0020/info) - Time established as 11/14/11 = 18:25:08; (51 seconds since power on) mfi0: 2084 (374610326s/0x0008/info) - Battery temperature is normal mfi0: 2085 (374610362s/0x0020/info) - Patrol Read resumed mfi0: 2086 (374610392s/0x0020/info) - Host driver is loaded and = operational mfi0: 2089 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) mfi0: 2090 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 mfi0: 2091 (boot + 9s/0x0008/info) - Battery Present mfi0: 2092 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 mfi0: 2093 (boot + 9s/0x0020/info) - Board Revision 15A mfi0: 2094 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0xff/s3) mfi0: 2095 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 mfi0: 2096 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0xff/s0) mfi0: 2097 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 mfi0: 2098 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0xff/s1) mfi0: 2099 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 mfi0: 2100 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0xff/s2) mfi0: 2101 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 mfi0: 2102 (boot + 28s/0x0002/info) - Inserted: PD 10(e0xff/s7) mfi0: 2103 (boot + 28s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221104000000,0000000000000000 mfi0: 2104 (boot + 28s/0x0002/info) - Inserted: PD 11(e0xff/s6) mfi0: 2105 (boot + 28s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221105000000,0000000000000000 mfi0: 2106 (boot + 28s/0x0002/info) - Inserted: PD 12(e0xff/s4) mfi0: 2107 (boot + 28s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 mfi0: 2108 (boot + 28s/0x0002/info) - Inserted: PD 13(e0xff/s5) mfi0: 2109 (boot + 28s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 mfi0: 2110 (374610516s/0x0020/info) - Time established as 11/14/11 = 18:28:36; (35 seconds since power on) mfi0: 2111 (374610550s/0x0008/info) - Battery temperature is normal mfi0: 2112 (374610570s/0x0020/info) - Patrol Read resumed mfi0: 2115 (374610602s/0x0020/info) - Host driver is loaded and = operational mfi0: 2116 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) mfi0: 2117 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 mfi0: 2118 (boot + 9s/0x0008/info) - Battery Present mfi0: 2119 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 mfi0: 2120 (boot + 9s/0x0020/info) - Board Revision 15A mfi0: 2121 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) mfi0: 2122 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 mfi0: 2123 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) mfi0: 2124 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 mfi0: 2125 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) mfi0: 2126 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 mfi0: 2127 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) mfi0: 2128 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 mfi0: 2129 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) mfi0: 2130 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221104000000,0000000000000000 mfi0: 2131 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) mfi0: 2132 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221105000000,0000000000000000 mfi0: 2133 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) mfi0: 2134 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 mfi0: 2135 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) mfi0: 2136 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 mfi0: 2137 (374610911s/0x0020/info) - Time established as 11/14/11 = 18:35:11; (52 seconds since power on) mfi0: 2138 (374610928s/0x0008/info) - Battery temperature is normal mfi0: 2139 (374610964s/0x0020/info) - Patrol Read resumed mfi0: [ITHREAD] pcib12: at device 7.0 on pci128 pci133: on pcib12 pcib13: at device 9.0 on pci128 pci134: on pcib13 pci128: at device 20.0 (no = driver attached) pci128: at device 20.1 (no = driver attached) pci128: at device 20.2 (no = driver attached) pci128: at device 20.3 (no = driver attached) pci128: at device 22.0 (no driver attached) pci128: at device 22.1 (no driver attached) pci128: at device 22.2 (no driver attached) pci128: at device 22.3 (no driver attached) pci128: at device 22.4 (no driver attached) pci128: at device 22.5 (no driver attached) pci128: at device 22.6 (no driver attached) pci128: at device 22.7 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] acpi_hpet0: iomem 0xfed00000-0xfed003ff on = acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 qpi0: on motherboard pcib14: pcibus 255 on qpi0 pci255: on pcib14 pcib15: pcibus 254 on qpi0 pci254: on pcib15 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 est12: on cpu12 p4tcc12: on cpu12 est13: on cpu13 p4tcc13: on cpu13 est14: on cpu14 p4tcc14: on cpu14 est15: on cpu15 p4tcc15: on cpu15 est16: on cpu16 p4tcc16: on cpu16 est17: on cpu17 p4tcc17: on cpu17 est18: on cpu18 p4tcc18: on cpu18 est19: on cpu19 p4tcc19: on cpu19 est20: on cpu20 p4tcc20: on cpu20 est21: on cpu21 p4tcc21: on cpu21 est22: on cpu22 p4tcc22: on cpu22 est23: on cpu23 p4tcc23: on cpu23 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered acd0: DVDR at ata2-master UDMA100 SATA 1.5Gb/s mfi0: 2142 (374610993s/0x0020/info) - Host driver is loaded and = operational mfid0: on mfi0 mfid0: 511999MB (1048575744 sectors) RAID volume '' is optimal SMP: AP CPU #1 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #17 Launched! uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Trying to mount root from cd9660:/dev/acd0 ugen4.2: at usbus4 ukbd0: = on usbus4 kbd2 at ukbd0 ugen1.2: at usbus1 ukbd1: on usbus1 kbd3 at ukbd1 ums0: on usbus1 igb0: link state changed to DOWN igb0: link state changed to UP mfiutil show adapter: mfi0 Adapter: Product Name: LSI MegaRAID SAS 9261-8i Serial Number: SV13605705 Firmware: 12.12.0-0073 RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50 Battery Backup: present NVRAM: 32K Onboard Memory: 512M Minimum Stripe: 8K Maximum Stripe: 1M mfiutil show firmware: mfi0 Firmware Package Version: 12.12.0-0073 mfi0 Firmware Images: Name Version Date Time Status APP 2.120.183-1415 Oct 05 2011 12:32:12 active BIOS 3.23.00_4.11.05.00_0x05080000 9/15/2011 9/15/2011 active PCLI 04.04-018:#%00008 Aug 05 2011 18:14:00 active BCON 6.0-45-e_40-Rel Sep 26 2011 09:55:15 active NVDT 2.09.03-0024 Oct 05 2011 00:43:21 active BTBL 2.02.00.00-0000 Sep 16 2009 21:37:06 active BOOT 09.250.01.219 4/4/2011 15:58:38 active From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 08:16:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7ED6106566B for ; Mon, 14 Nov 2011 08:16:08 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 6EFCF8FC1A for ; Mon, 14 Nov 2011 08:16:06 +0000 (UTC) Received: (qmail 83861 invoked by uid 907); 14 Nov 2011 08:16:05 -0000 Received: from jmmacpro.transactionware.com (HELO jmmacpro.transactionware.com) (192.168.1.33) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Mon, 14 Nov 2011 19:16:05 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Jan Mikkelsen In-Reply-To: <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> Date: Mon, 14 Nov 2011 19:16:04 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <20EDA65E-0AFF-4D89-A15A-E3813476F8B6@transactionware.com> References: <4EA9E0C3.5080306@unsane.co.uk> <992755CA-6479-4B9A-A3D5-DD5C1871089A@transactionware.com> <4EB1BA7A.2000307@unsane.co.uk> <201111081450.34686.jhb@freebsd.org> <4EB9AC0F.2040209@unsane.co.uk> <4EB9BD9B.8080604@unsane.co.uk> <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> To: John Baldwin X-Mailer: Apple Mail (2.1251.1) Cc: FreeBSD Stable Mailing List , Jeremy Chadwick , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 08:16:09 -0000 Following up ... Booting the 9-stable kernel with the patch from = http://www.freebsd.org/~jhb/patches/mfi.patch, modified to use = pci_alloc_msi with hw.mfi.msix=3D1 boots OK. Haven't put any load on it = yet. Will try the plain patch without the pci_alloc_msi change. On 14/11/2011, at 7:03 PM, Jan Mikkelsen wrote: > Hi, >=20 > I have just tested mfi on a machine that has just arrived. I'm seeing = the command timeout problem on boot with 9.0-RC1. >=20 > The message is "mfi0: COMMAND TIMEOUT AFTER 59 seconds", and = then repeats every 30 seconds (with the time changed, obviously). >=20 > I have tested the 9.0-RC1 ISO, a 9-stable kernel patched with the = patch from the PR I referenced below (also in FreeBSD cvs in revision = 1.62 of src/sys/dev/mfi/mfi.c), and 9-stable kernel with the patch at = www.freebsd.org/~jhb/patches/mfi.patch with the 'pci_alloc_msix' call = changed to 'pci_alloc_msi'. >=20 > I'm setting up a test with the pci_alloc_msix call unchanged at the = moment, but I don't have anything in /boot/loader.conf, so that I'm not = expecting that to make a difference. >=20 > This is on a Supermicro X8DTi-F, 48GB memory and an LSI MegaRAID = 9261-8i. >=20 > 8.2-RELEASE boots fine, dmesg and some output from mfiutil below. >=20 > Any suggestions gratefully received! >=20 > Thanks, >=20 > Jan Mikkelsen >=20 >=20 > On 09/11/2011, at 10:39 AM, Vincent Hoffman wrote: >=20 >> On 08/11/2011 22:24, Vincent Hoffman wrote: >>> On 08/11/2011 19:50, John Baldwin wrote: >>>> On Wednesday, November 02, 2011 5:47:38 pm Vincent Hoffman wrote: >>>>> On 28/10/2011 04:14, Jan Mikkelsen wrote: >>>>>> Hi, >>>>>>=20 >>>>>> There is a patch linked to from this PR, which seems very = similar: >>>>>>=20 >>>>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/140416 >>>>>>=20 >>>>>> = http://lists.freebsd.org/pipermail/freebsd-scsi/2011-March/004839.html >>>>>>=20 >>>>>> The problem is also consistent with running mfiutil clearing the = problem. >>>>>>=20 >>>>>> I'm about to deploy mfi controllers in a similar configuration, = so I'd be=20 >>>> very curious about whether the patch fixes the problem for you. >>>>> The patch you linked to seems to have removed the stalls, although = I >>>>> have only had it running for a day. I'll post if it stalls again = though. >>>>>=20 >>>>> I did manage to scrounge the use of a Dell r410 with a >>>>> LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) >>>>> Badged as Dell PERC H700 Adapter >>>>>=20 >>>>> to test out the patch I originally found but had the same issue as = this post >>>>>=20 >>>>> = http://lists.freebsd.org/pipermail/freebsd-stable/2011-September/063821.ht= ml >>>>>=20 >>>>>=20 >>>>> I couldnt get the dell to stall in the first place either though = so it >>>>> could be a specific firmware version that the issue. >>>>>=20 >>>>> Anyway thanks for the pointers. >>>> Hmm, did you try the patch I had posted from that earlier thread? = It had >>>> two changes in it, one was similar to the patch in the PR, the = second added >>>> MSI-X support. I've since tweaked it to make the MSI-X support off = by >>>> default but possible to enable via loader.conf. Would you be = willing to >>>> try the updated patch at www.freebsd.org/~jhb/patches/mfi.patch? >>> Hi, >>> yes I tried the patch you posted originally, unfortunately the dell >>> never finished booting either. The Supermicro is now in production = but >>> I'll take the dell up to 9-STABLE and try your updated patch. >>>=20 >> The patch didnt apply quite cleanly for 9-STABLE, 1 reject as it had >> already been applied. >>=20 >> I have rebooted the dell and it seems happy with the new patch (msi >> disabled.) >> Booting with >> hw.mfi.msix=3D1 in /boot/loader.conf causes the timeouts again and = stops >> the boot from completing. >> I can give root access to the machine if this would be helpful, I = cant >> give KVM access though unfortunately. >> (but can look in from time to time if needed.) >>=20 >>=20 >> Vince >=20 > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-RELEASE #0: Thu May 26 14:33:45 EST 2011 > = root@valhalla.transactionware.com:/home/janm/p4/freebsd-image-std-2008.2/w= ork/base-freebsd/home/janm/p4/freebsd-image-std-2008.2/FreeBSD/src/sys/GEN= ERIC amd64 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (2409.70-MHz = K8-class CPU) > Origin =3D "GenuineIntel" Id =3D 0x206c2 Family =3D 6 Model =3D 2c = Stepping =3D 2 > = Features=3D0xbfebfbff > = Features2=3D0x9ee3fd > AMD Features=3D0x2c100800 > AMD Features2=3D0x1 > TSC: P-state invariant > real memory =3D 51543801856 (49156 MB) > avail memory =3D 49664753664 (47364 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs > FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 16 > cpu7 (AP): APIC ID: 17 > cpu8 (AP): APIC ID: 18 > cpu9 (AP): APIC ID: 19 > cpu10 (AP): APIC ID: 20 > cpu11 (AP): APIC ID: 21 > cpu12 (AP): APIC ID: 32 > cpu13 (AP): APIC ID: 33 > cpu14 (AP): APIC ID: 34 > cpu15 (AP): APIC ID: 35 > cpu16 (AP): APIC ID: 36 > cpu17 (AP): APIC ID: 37 > cpu18 (AP): APIC ID: 48 > cpu19 (AP): APIC ID: 49 > cpu20 (AP): APIC ID: 50 > cpu21 (AP): APIC ID: 51 > cpu22 (AP): APIC ID: 52 > cpu23 (AP): APIC ID: 53 > ioapic1: Changing APIC ID to 7 > ioapic2: Changing APIC ID to 8 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 24-47 on motherboard > ioapic2 irqs 48-71 on motherboard > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 400, 100 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0x4C, should be = 0x49 (20101013/tbutils-354) > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > cpu12: on acpi0 > cpu13: on acpi0 > cpu14: on acpi0 > cpu15: on acpi0 > cpu16: on acpi0 > cpu17: on acpi0 > cpu18: on acpi0 > cpu19: on acpi0 > cpu20: on acpi0 > cpu21: on acpi0 > cpu22: on acpi0 > cpu23: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 1.0 on pci0 > pci1: on pcib1 > igb0: port = 0xec00-0xec1f mem = 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 = at device 0.0 on pci1 > igb0: Using MSIX interrupts with 9 vectors > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: [ITHREAD] > igb0: Ethernet address: 00:25:90:62:20:3e > igb1: port = 0xe800-0xe81f mem = 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf98000-0xfaf9bfff irq 40 = at device 0.1 on pci1 > igb1: Using MSIX interrupts with 9 vectors > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: [ITHREAD] > igb1: Ethernet address: 00:25:90:62:20:3f > pcib2: at device 3.0 on pci0 > pci2: on pcib2 > pcib3: at device 5.0 on pci0 > pci3: on pcib3 > pcib4: at device 7.0 on pci0 > pci4: on pcib4 > pcib5: at device 9.0 on pci0 > pci5: on pcib5 > pci0: at device 20.0 (no = driver attached) > pci0: at device 20.1 (no = driver attached) > pci0: at device 20.2 (no = driver attached) > pci0: at device 20.3 (no = driver attached) > pci0: at device 22.0 (no driver attached) > pci0: at device 22.1 (no driver attached) > pci0: at device 22.2 (no driver attached) > pci0: at device 22.3 (no driver attached) > pci0: at device 22.4 (no driver attached) > pci0: at device 22.5 (no driver attached) > pci0: at device 22.6 (no driver attached) > pci0: at device 22.7 (no driver attached) > uhci0: port 0xcf80-0xcf9f = irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup =3D 0x2f00 > usbus0: on uhci0 > uhci1: port 0xcf40-0xcf5f = irq 21 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup =3D 0x2f00 > usbus1: on uhci1 > uhci2: port 0xcf20-0xcf3f = irq 19 at device 26.2 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup =3D 0x2f00 > usbus2: on uhci2 > ehci0: mem = 0xfaefc000-0xfaefc3ff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus3: EHCI version 1.0 > usbus3: on ehci0 > uhci3: port 0xcf00-0xcf1f = irq 23 at device 29.0 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup =3D 0x2f00 > usbus4: on uhci3 > uhci4: port 0xcec0-0xcedf = irq 19 at device 29.1 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup =3D 0x2f00 > usbus5: on uhci4 > uhci5: port 0xcea0-0xcebf = irq 18 at device 29.2 on pci0 > uhci5: [ITHREAD] > uhci5: LegSup =3D 0x2f00 > usbus6: on uhci5 > ehci1: mem = 0xfaeda000-0xfaeda3ff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus7: EHCI version 1.0 > usbus7: on ehci1 > pcib6: at device 30.0 on pci0 > pci6: on pcib6 > vgapci0: mem = 0xf9000000-0xf9ffffff,0xfbefc000-0xfbefffff,0xfb000000-0xfb7fffff irq 16 = at device 4.0 on pci6 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port = 0xcff0-0xcff7,0xcf7c-0xcf7f,0xcfe0-0xcfe7,0xcef4-0xcef7,0xcfa0-0xcfaf,0xcf= 60-0xcf6f irq 19 at device 31.2 on pci0 > atapci0: [ITHREAD] > ata2: on atapci0 > ata2: [ITHREAD] > ata3: on atapci0 > ata3: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atapci1: port = 0xcee0-0xcee7,0xcef0-0xcef3,0xce70-0xce77,0xce7c-0xce7f,0xce60-0xce6f,0xce= 40-0xce4f irq 19 at device 31.5 on pci0 > atapci1: [ITHREAD] > ata4: on atapci1 > ata4: [ITHREAD] > ata5: on atapci1 > ata5: [ITHREAD] > pcib7: on acpi0 > pci128: on pcib7 > pcib8: at device 0.0 on pci128 > pci129: on pcib8 > pcib9: at device 1.0 on pci128 > pci130: on pcib9 > pcib10: at device 3.0 on pci128 > pci131: on pcib10 > pcib11: at device 5.0 on pci128 > pci132: on pcib11 > mfi0: port 0xf800-0xf8ff mem = 0xf8f9c000-0xf8f9ffff,0xf8fc0000-0xf8ffffff irq 50 at device 0.0 on = pci132 > mfi0: Megaraid SAS driver Ver 3.00=20 > mfi0: 2061 (374583936s/0x0020/info) - Shutdown command received from = host > mfi0: 2062 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) > mfi0: 2063 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 > mfi0: 2064 (boot + 9s/0x0008/info) - Battery Present > mfi0: 2065 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 > mfi0: 2066 (boot + 9s/0x0020/info) - Board Revision 15A > mfi0: 2067 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) > mfi0: 2068 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 > mfi0: 2069 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) > mfi0: 2070 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 > mfi0: 2071 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) > mfi0: 2072 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 > mfi0: 2073 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) > mfi0: 2074 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 > mfi0: 2075 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) > mfi0: 2076 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221104000000,0000000000000000 > mfi0: 2077 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) > mfi0: 2078 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221105000000,0000000000000000 > mfi0: 2079 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) > mfi0: 2080 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 > mfi0: 2081 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) > mfi0: 2082 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 > mfi0: 2083 (374610308s/0x0020/info) - Time established as 11/14/11 = 18:25:08; (51 seconds since power on) > mfi0: 2084 (374610326s/0x0008/info) - Battery temperature is normal > mfi0: 2085 (374610362s/0x0020/info) - Patrol Read resumed > mfi0: 2086 (374610392s/0x0020/info) - Host driver is loaded and = operational > mfi0: 2089 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) > mfi0: 2090 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 > mfi0: 2091 (boot + 9s/0x0008/info) - Battery Present > mfi0: 2092 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 > mfi0: 2093 (boot + 9s/0x0020/info) - Board Revision 15A > mfi0: 2094 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0xff/s3) > mfi0: 2095 (boot + 28s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 > mfi0: 2096 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0xff/s0) > mfi0: 2097 (boot + 28s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 > mfi0: 2098 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0xff/s1) > mfi0: 2099 (boot + 28s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 > mfi0: 2100 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0xff/s2) > mfi0: 2101 (boot + 28s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 > mfi0: 2102 (boot + 28s/0x0002/info) - Inserted: PD 10(e0xff/s7) > mfi0: 2103 (boot + 28s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221104000000,0000000000000000 > mfi0: 2104 (boot + 28s/0x0002/info) - Inserted: PD 11(e0xff/s6) > mfi0: 2105 (boot + 28s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221105000000,0000000000000000 > mfi0: 2106 (boot + 28s/0x0002/info) - Inserted: PD 12(e0xff/s4) > mfi0: 2107 (boot + 28s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 > mfi0: 2108 (boot + 28s/0x0002/info) - Inserted: PD 13(e0xff/s5) > mfi0: 2109 (boot + 28s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 > mfi0: 2110 (374610516s/0x0020/info) - Time established as 11/14/11 = 18:28:36; (35 seconds since power on) > mfi0: 2111 (374610550s/0x0008/info) - Battery temperature is normal > mfi0: 2112 (374610570s/0x0020/info) - Patrol Read resumed > mfi0: 2115 (374610602s/0x0020/info) - Host driver is loaded and = operational > mfi0: 2116 (boot + 4s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9263/1000) > mfi0: 2117 (boot + 4s/0x0020/info) - Firmware version 2.120.183-1415 > mfi0: 2118 (boot + 9s/0x0008/info) - Battery Present > mfi0: 2119 (boot + 9s/0x0020/info) - Package version 12.12.0-0073 > mfi0: 2120 (boot + 9s/0x0020/info) - Board Revision 15A > mfi0: 2121 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) > mfi0: 2122 (boot + 44s/0x0002/info) - Inserted: PD 0c(e0xff/s3) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D00, = sasAddr=3D4433221100000000,0000000000000000 > mfi0: 2123 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) > mfi0: 2124 (boot + 44s/0x0002/info) - Inserted: PD 0d(e0xff/s0) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D01, = sasAddr=3D4433221103000000,0000000000000000 > mfi0: 2125 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) > mfi0: 2126 (boot + 44s/0x0002/info) - Inserted: PD 0e(e0xff/s1) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D02, = sasAddr=3D4433221102000000,0000000000000000 > mfi0: 2127 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) > mfi0: 2128 (boot + 44s/0x0002/info) - Inserted: PD 0f(e0xff/s2) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D03, = sasAddr=3D4433221101000000,0000000000000000 > mfi0: 2129 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) > mfi0: 2130 (boot + 44s/0x0002/info) - Inserted: PD 10(e0xff/s7) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D05, = sasAddr=3D4433221104000000,0000000000000000 > mfi0: 2131 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) > mfi0: 2132 (boot + 44s/0x0002/info) - Inserted: PD 11(e0xff/s6) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D04, = sasAddr=3D4433221105000000,0000000000000000 > mfi0: 2133 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) > mfi0: 2134 (boot + 44s/0x0002/info) - Inserted: PD 12(e0xff/s4) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D07, = sasAddr=3D4433221107000000,0000000000000000 > mfi0: 2135 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) > mfi0: 2136 (boot + 44s/0x0002/info) - Inserted: PD 13(e0xff/s5) Info: = enclPd=3Dffff, scsiType=3D0, portMap=3D06, = sasAddr=3D4433221106000000,0000000000000000 > mfi0: 2137 (374610911s/0x0020/info) - Time established as 11/14/11 = 18:35:11; (52 seconds since power on) > mfi0: 2138 (374610928s/0x0008/info) - Battery temperature is normal > mfi0: 2139 (374610964s/0x0020/info) - Patrol Read resumed > mfi0: [ITHREAD] > pcib12: at device 7.0 on pci128 > pci133: on pcib12 > pcib13: at device 9.0 on pci128 > pci134: on pcib13 > pci128: at device 20.0 (no = driver attached) > pci128: at device 20.1 (no = driver attached) > pci128: at device 20.2 (no = driver attached) > pci128: at device 20.3 (no = driver attached) > pci128: at device 22.0 (no driver attached) > pci128: at device 22.1 (no driver attached) > pci128: at device 22.2 (no driver attached) > pci128: at device 22.3 (no driver attached) > pci128: at device 22.4 (no driver attached) > pci128: at device 22.5 (no driver attached) > pci128: at device 22.6 (no driver attached) > pci128: at device 22.7 (no driver attached) > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on = acpi0 > uart0: [FILTER] > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > uart1: [FILTER] > acpi_hpet0: iomem 0xfed00000-0xfed003ff = on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > qpi0: on motherboard > pcib14: pcibus 255 on qpi0 > pci255: on pcib14 > pcib15: pcibus 254 on qpi0 > pci254: on pcib15 > orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on = isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=3D0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > ppc0: cannot reserve I/O port range > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > est2: on cpu2 > p4tcc2: on cpu2 > est3: on cpu3 > p4tcc3: on cpu3 > est4: on cpu4 > p4tcc4: on cpu4 > est5: on cpu5 > p4tcc5: on cpu5 > est6: on cpu6 > p4tcc6: on cpu6 > est7: on cpu7 > p4tcc7: on cpu7 > est8: on cpu8 > p4tcc8: on cpu8 > est9: on cpu9 > p4tcc9: on cpu9 > est10: on cpu10 > p4tcc10: on cpu10 > est11: on cpu11 > p4tcc11: on cpu11 > est12: on cpu12 > p4tcc12: on cpu12 > est13: on cpu13 > p4tcc13: on cpu13 > est14: on cpu14 > p4tcc14: on cpu14 > est15: on cpu15 > p4tcc15: on cpu15 > est16: on cpu16 > p4tcc16: on cpu16 > est17: on cpu17 > p4tcc17: on cpu17 > est18: on cpu18 > p4tcc18: on cpu18 > est19: on cpu19 > p4tcc19: on cpu19 > est20: on cpu20 > p4tcc20: on cpu20 > est21: on cpu21 > p4tcc21: on cpu21 > est22: on cpu22 > p4tcc22: on cpu22 > est23: on cpu23 > p4tcc23: on cpu23 > Timecounters tick every 1.000 msec > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 12Mbps Full Speed USB v1.0 > usbus3: 480Mbps High Speed USB v2.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 12Mbps Full Speed USB v1.0 > usbus7: 480Mbps High Speed USB v2.0 > ugen0.1: at usbus0 > uhub0: on = usbus0 > ugen1.1: at usbus1 > uhub1: on = usbus1 > ugen2.1: at usbus2 > uhub2: on = usbus2 > ugen3.1: at usbus3 > uhub3: on = usbus3 > ugen4.1: at usbus4 > uhub4: on = usbus4 > ugen5.1: at usbus5 > uhub5: on = usbus5 > ugen6.1: at usbus6 > uhub6: on = usbus6 > ugen7.1: at usbus7 > uhub7: on = usbus7 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub2: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub6: 2 ports with 2 removable, self powered > acd0: DVDR at ata2-master UDMA100 SATA 1.5Gb/s > mfi0: 2142 (374610993s/0x0020/info) - Host driver is loaded and = operational > mfid0: on mfi0 > mfid0: 511999MB (1048575744 sectors) RAID volume '' is optimal > SMP: AP CPU #1 Launched! > SMP: AP CPU #14 Launched! > SMP: AP CPU #11 Launched! > SMP: AP CPU #20 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #22 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #16 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #19 Launched! > SMP: AP CPU #4 Launched! > SMP: AP CPU #18 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #23 Launched! > SMP: AP CPU #9 Launched! > SMP: AP CPU #21 Launched! > SMP: AP CPU #8 Launched! > SMP: AP CPU #15 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #12 Launched! > SMP: AP CPU #10 Launched! > SMP: AP CPU #13 Launched! > SMP: AP CPU #17 Launched! > uhub3: 6 ports with 6 removable, self powered > uhub7: 6 ports with 6 removable, self powered > Trying to mount root from cd9660:/dev/acd0 > ugen4.2: at usbus4 > ukbd0: = on usbus4 > kbd2 at ukbd0 > ugen1.2: at usbus1 > ukbd1: on usbus1 > kbd3 at ukbd1 > ums0: on usbus1 > igb0: link state changed to DOWN > igb0: link state changed to UP >=20 > mfiutil show adapter: >=20 > mfi0 Adapter: > Product Name: LSI MegaRAID SAS 9261-8i > Serial Number: SV13605705 > Firmware: 12.12.0-0073 > RAID Levels: JBOD, RAID0, RAID1, RAID5, RAID6, RAID10, RAID50 > Battery Backup: present > NVRAM: 32K > Onboard Memory: 512M > Minimum Stripe: 8K > Maximum Stripe: 1M >=20 > mfiutil show firmware: >=20 > mfi0 Firmware Package Version: 12.12.0-0073 > mfi0 Firmware Images: > Name Version Date Time Status > APP 2.120.183-1415 Oct 05 2011 12:32:12 active > BIOS 3.23.00_4.11.05.00_0x05080000 9/15/2011 > 9/15/2011 > active > PCLI 04.04-018:#%00008 Aug 05 2011 18:14:00 active > BCON 6.0-45-e_40-Rel Sep 26 2011 09:55:15 active > NVDT 2.09.03-0024 Oct 05 2011 00:43:21 active > BTBL 2.02.00.00-0000 Sep 16 2009 21:37:06 active > BOOT 09.250.01.219 4/4/2011 15:58:38 active >=20 From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 08:37:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69658106566C for ; Mon, 14 Nov 2011 08:37:47 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id E49118FC16 for ; Mon, 14 Nov 2011 08:37:46 +0000 (UTC) Received: by wyf23 with SMTP id 23so4728001wyf.13 for ; Mon, 14 Nov 2011 00:37:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=U+WrWqMwn5Qn6+KsxnNGPn7IchsiLlKkt2/r9pjNTF4=; b=ZRg/tQ2I196CQYU2pbGoNmcac2z9J1IY/SVQN5aV06qjtjdsVijlEAhOj90stvPe0p H0KFYgnfV6artNO/lYKiaagFqoUehnQ7XOEAAsMlQforr3t2uYZmJSTPZZVIJYuSuSmF Ta1iucqSBDVZHPsaB7xTrJ5tbYgwJYEPV6rzI= Received: by 10.216.131.85 with SMTP id l63mr1685362wei.1.1321257969208; Mon, 14 Nov 2011 00:06:09 -0800 (PST) Received: from green.tandem.local (utwig.xim.bz. [91.216.237.46]) by mx.google.com with ESMTPS id eu16sm23949414wbb.7.2011.11.14.00.06.07 (version=SSLv3 cipher=OTHER); Mon, 14 Nov 2011 00:06:08 -0800 (PST) Message-ID: <4EC0CBEE.2070300@gmail.com> Date: Mon, 14 Nov 2011 10:06:06 +0200 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111111 Thunderbird/8.0 MIME-Version: 1.0 To: FreeBSD Stable Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: stable fails to build for me X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 08:37:47 -0000 I'm running into this: ===> gnu/lib/libsupc++ (install) clang -O2 -pipe -std=gnu99 -fstack-protector -Wsystem-headers -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -c /usr/src/usr.bin/lex/lib/libyywrap.c install -C -C -o root -g wheel -m 444 libsupc++.a /usr/obj/usr/src/tmp/usr/lib /usr/src/lib/libgpib/ibfoo.c:37:10: fatal error: 'dev/ieee488/ugpib.h' file not found #include ^ 1 error generated. mkdep: compile failed /etc/src.conf: WITHOUT_ACCT='yes' WITHOUT_ATM='yes' WITHOUT_AUTHPF='yes' WITHOUT_CTM='yes' WITHOUT_FREEBSD_UPDATE='yes' WITHOUT_HTML='yes' WITHOUT_I4B='yes' WITHOUT_IPFILTER='yes' WITHOUT_IPFW='yes' WITHOUT_IPX='yes' WITHOUT_NDIS='yes' WITHOUT_PMC='yes' WITHOUT_PORTSNAP='yes' WITHOUT_PROFILE='yes' WITHOUT_QUOTAS='yes' WITHOUT_RCMDS='yes' WITHOUT_RCS='yes' WITHOUT_ROUTED='yes' WITHOUT_SENDMAIL='yes' WITHOUT_SLIP='yes' WITHOUT_SYSINSTALL='yes' WITHOUT_WIRELESS='yes' -- Sphinx of black quartz judge my vow. From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 10:56:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EE00106564A for ; Mon, 14 Nov 2011 10:56:15 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id D34CF8FC0A for ; Mon, 14 Nov 2011 10:56:14 +0000 (UTC) Received: by faar19 with SMTP id r19so7301444faa.13 for ; Mon, 14 Nov 2011 02:56:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=7jBHrWbUAdA/VT7rfcdBn8xg0xFwq+aQ/WiGHNFWR7w=; b=PBjZkSkmwANL7apmDZRSqc5/Mx/5Y8pBFgIQIMkGIHNzNjhOQ96/pYPBoBMNPBvBQy f9k1osu6irbZqQ1eS/KDhm7/4GU09ttowGe7AfVVZI221HXHs7LX9Ag3iBLqnrhpsEur oPi7ksHqyvIfLAPaM7IBFsiYaZC7m5C+mrw7w= Received: by 10.204.154.6 with SMTP id m6mr18971795bkw.20.1321268173639; Mon, 14 Nov 2011 02:56:13 -0800 (PST) Received: from green.tandem.local (utwig.xim.bz. [91.216.237.46]) by mx.google.com with ESMTPS id dq2sm30867233bkb.11.2011.11.14.02.56.12 (version=SSLv3 cipher=OTHER); Mon, 14 Nov 2011 02:56:13 -0800 (PST) Message-ID: <4EC0F3C9.5010205@gmail.com> Date: Mon, 14 Nov 2011 12:56:09 +0200 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111111 Thunderbird/8.0 MIME-Version: 1.0 CC: FreeBSD Stable References: <4EC0CBEE.2070300@gmail.com> In-Reply-To: <4EC0CBEE.2070300@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: stable fails to build for me [bad setup] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 10:56:15 -0000 14.11.2011 10:06, Volodymyr Kostyrko wrote: > I'm running into this: > > ===> gnu/lib/libsupc++ (install) > clang -O2 -pipe -std=gnu99 -fstack-protector -Wsystem-headers -Wall > -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -c > /usr/src/usr.bin/lex/lib/libyywrap.c > install -C -C -o root -g wheel -m 444 libsupc++.a > /usr/obj/usr/src/tmp/usr/lib > /usr/src/lib/libgpib/ibfoo.c:37:10: fatal error: 'dev/ieee488/ugpib.h' > file not found > #include > ^ > 1 error generated. > mkdep: compile failed > > /etc/src.conf: > WITHOUT_ACCT='yes' > WITHOUT_ATM='yes' > WITHOUT_AUTHPF='yes' > WITHOUT_CTM='yes' > WITHOUT_FREEBSD_UPDATE='yes' > WITHOUT_HTML='yes' > WITHOUT_I4B='yes' > WITHOUT_IPFILTER='yes' > WITHOUT_IPFW='yes' > WITHOUT_IPX='yes' > WITHOUT_NDIS='yes' > WITHOUT_PMC='yes' > WITHOUT_PORTSNAP='yes' > WITHOUT_PROFILE='yes' > WITHOUT_QUOTAS='yes' > WITHOUT_RCMDS='yes' > WITHOUT_RCS='yes' > WITHOUT_ROUTED='yes' > WITHOUT_SENDMAIL='yes' > WITHOUT_SLIP='yes' > WITHOUT_SYSINSTALL='yes' > WITHOUT_WIRELESS='yes' > I'm sorry, for some unknown reason the host lacks /usr/include/dev directory... -- Sphinx of black quartz judge my vow. From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 17:57:20 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0882106566B for ; Mon, 14 Nov 2011 17:57:20 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 880878FC14 for ; Mon, 14 Nov 2011 17:57:20 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 3DDC246B46; Mon, 14 Nov 2011 12:57:20 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D522D8A050; Mon, 14 Nov 2011 12:57:19 -0500 (EST) From: John Baldwin To: Vincent Hoffman Date: Mon, 14 Nov 2011 11:29:37 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EA9E0C3.5080306@unsane.co.uk> <201111090939.14177.jhb@freebsd.org> <4EBBAE90.8010101@unsane.co.uk> In-Reply-To: <4EBBAE90.8010101@unsane.co.uk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111141129.37364.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 14 Nov 2011 12:57:19 -0500 (EST) Cc: Jan Mikkelsen , freebsd-stable@freebsd.org Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 17:57:20 -0000 On Thursday, November 10, 2011 5:59:28 am Vincent Hoffman wrote: > On 09/11/2011 14:39, John Baldwin wrote: > > On Tuesday, November 08, 2011 6:39:07 pm Vincent Hoffman wrote: > >> On 08/11/2011 22:24, Vincent Hoffman wrote: > >>> On 08/11/2011 19:50, John Baldwin wrote: > >>>> Hmm, did you try the patch I had posted from that earlier thread? It had > >>>> two changes in it, one was similar to the patch in the PR, the second added > >>>> MSI-X support. I've since tweaked it to make the MSI-X support off by > >>>> default but possible to enable via loader.conf. Would you be willing to > >>>> try the updated patch at www.freebsd.org/~jhb/patches/mfi.patch? > >>> Hi, > >>> yes I tried the patch you posted originally, unfortunately the dell > >>> never finished booting either. The Supermicro is now in production but > >>> I'll take the dell up to 9-STABLE and try your updated patch. > >>> > >> The patch didnt apply quite cleanly for 9-STABLE, 1 reject as it had > >> already been applied. > > Odd, it's against stock head, so I don't know why it would have failed to > > apply. > > > >> I have rebooted the dell and it seems happy with the new patch (msi > >> disabled.) > > Okay, good. I'll commit the non-MSI bits at least and get them merged into > > 9.0 if possible. > > > >> Booting with > >> hw.mfi.msix=1 in /boot/loader.conf causes the timeouts again and stops > >> the boot from completing. > > Ok. Can you try changing it to use MSI instead of MSI-X? Just edit the > > mfi_pci.c call and replace 'pci_alloc_msix' with 'pci_alloc_msi'. > > > Well the dell has been up for about 19 hours now using MSI, I ran > bonnie++ a few times on it and have now stuck it in a permanent loop > (will look in from time to time.) Are there any tests you'd like > run/info you'd like? No, this looks good. I'll probably commit something to mfi to just enable MSI only (but with a tunable that defaults to off) so people can do broader testing. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 17:57:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95B7F106564A for ; Mon, 14 Nov 2011 17:57:21 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6BA198FC17 for ; Mon, 14 Nov 2011 17:57:21 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 17E5546B0C; Mon, 14 Nov 2011 12:57:21 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B06778A053; Mon, 14 Nov 2011 12:57:20 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org Date: Mon, 14 Nov 2011 11:36:01 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EA9E0C3.5080306@unsane.co.uk> <4EB9BD9B.8080604@unsane.co.uk> <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> In-Reply-To: <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111141136.01348.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 14 Nov 2011 12:57:20 -0500 (EST) Cc: Jan Mikkelsen , Jeremy Chadwick , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 17:57:21 -0000 On Monday, November 14, 2011 3:03:42 am Jan Mikkelsen wrote: > Hi, > > I have just tested mfi on a machine that has just arrived. I'm seeing the command timeout problem on boot with 9.0-RC1. > > The message is "mfi0: COMMAND TIMEOUT AFTER 59 seconds", and then repeats every 30 seconds (with the time changed, obviously). > > I have tested the 9.0-RC1 ISO, a 9-stable kernel patched with the patch from the PR I referenced below (also in FreeBSD cvs in revision 1.62 of src/sys/dev/mfi/mfi.c), and 9-stable kernel with the patch at www.freebsd.org/~jhb/patches/mfi.patch with the 'pci_alloc_msix' call changed to 'pci_alloc_msi'. You forgot to mention what happened from those tests, did any of them work? -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 18:45:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0690F106564A for ; Mon, 14 Nov 2011 18:45:25 +0000 (UTC) (envelope-from kron24@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9260B8FC17 for ; Mon, 14 Nov 2011 18:45:24 +0000 (UTC) Received: by faar19 with SMTP id r19so7945907faa.13 for ; Mon, 14 Nov 2011 10:45:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=TiWRz7SMEVw9sONqs34wrmuaLnFAUdcG3Hr17W1PqLs=; b=jIk0McCUw8C3Fq3DVZLwyw1US38CuvmKsA2LszGtlFomOl8ccGw6FW+Lb+YJvghuI/ jgblRjAbtZalJ8iW+IG4rwayC79mIDQMvAvCBU1u8sQKkSYZfay4ehW0jhLur99sd/0C aWdCYcQgAPyzo46QFKbLErOzwSXS2Y+NQPqiQ= Received: by 10.152.105.83 with SMTP id gk19mr14985858lab.30.1321294978523; Mon, 14 Nov 2011 10:22:58 -0800 (PST) Received: from nbvk.local (uidzr185150.sattnet.cz. [212.96.185.150]) by mx.google.com with ESMTPS id jb5sm19626725lab.15.2011.11.14.10.22.55 (version=SSLv3 cipher=OTHER); Mon, 14 Nov 2011 10:22:57 -0800 (PST) Message-ID: <4EC15C7E.2080802@gmail.com> Date: Mon, 14 Nov 2011 19:22:54 +0100 From: kron User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: 9.0-RC2: regression in power management on VIA Samuel 2 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 18:45:25 -0000 Hi, I have a few machines (mostly routers) with VIA Samuel 2. FreeBSD 9.0-RC2 breaks power management on them. 8.2-RELEASE detects: CPU: VIA Samuel 2 (532.64-MHz 686-class CPU) Origin = "CentaurHauls" Id = 0x673 Family = 6 Model = 7 Stepping = 3 Features=0x803035 and sysctl dev.cpu.0 says: dev.cpu.0.%desc: ACPI CPU dev.cpu.0.%driver: cpu dev.cpu.0.%location: handle=\_PR_.CPU0 dev.cpu.0.%pnpinfo: _HID=none _UID=0 dev.cpu.0.%parent: acpi0 dev.cpu.0.freq: 532 dev.cpu.0.freq_levels: 532/-1 266/-1 dev.cpu.0.cx_supported: C1/0 C2/90 C3/900 dev.cpu.0.cx_lowest: C2 dev.cpu.0.cx_usage: 0.45% 99.54% 0.00% last 783us In 8.2 I can use dev.cpu.0.cx_lowest=C2 in /etc/sysctl.conf and powerd runs fine switching between 532 and 266 MHz. FreeBSD 9.0-RC2 detects: CPU: VIA Samuel 2 (532.65-MHz 686-class CPU) Origin = "CentaurHauls" Id = 0x673 Family = 6 Model = 7 Stepping = 3 Features=0x803035 AMD Features=0x80000000<3DNow!> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the only difference and sysctl dev.cpu.0 is: dev.cpu.0.%desc: ACPI CPU dev.cpu.0.%driver: cpu dev.cpu.0.%location: handle=\_PR_.CPU0 dev.cpu.0.%pnpinfo: _HID=none _UID=0 dev.cpu.0.%parent: acpi0 dev.cpu.0.cx_supported: C1/0 dev.cpu.0.cx_lowest: C1 dev.cpu.0.cx_usage: 100.00% last 208us Of course, dev.cpu.0.cx_lowest=C2 is not available and powerd cannot start - "powerd -v" fails: powerd: lookup freq: No such file or directory I haven't studied sources yet but it seems to me AMD is detected (wrong) and for that reason a wrong power management code path takes place. Is anybody willing to help me with debugging? I have no experience in such "low level" programming but at least one spare machine and some time to waste. TIA, Oli From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 19:42:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50FF7106566C for ; Mon, 14 Nov 2011 19:42:18 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 281738FC13 for ; Mon, 14 Nov 2011 19:42:18 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id BADDE46B46; Mon, 14 Nov 2011 14:42:17 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 566448A050; Mon, 14 Nov 2011 14:42:17 -0500 (EST) From: John Baldwin To: Vincent Hoffman Date: Mon, 14 Nov 2011 14:42:16 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EA9E0C3.5080306@unsane.co.uk> <201111090939.14177.jhb@freebsd.org> <4EBBAE90.8010101@unsane.co.uk> In-Reply-To: <4EBBAE90.8010101@unsane.co.uk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111141442.16722.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 14 Nov 2011 14:42:17 -0500 (EST) Cc: Jan Mikkelsen , freebsd-stable@freebsd.org Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 19:42:18 -0000 On Thursday, November 10, 2011 5:59:28 am Vincent Hoffman wrote: > On 09/11/2011 14:39, John Baldwin wrote: > > On Tuesday, November 08, 2011 6:39:07 pm Vincent Hoffman wrote: > >> On 08/11/2011 22:24, Vincent Hoffman wrote: > >>> On 08/11/2011 19:50, John Baldwin wrote: > >>>> Hmm, did you try the patch I had posted from that earlier thread? It had > >>>> two changes in it, one was similar to the patch in the PR, the second added > >>>> MSI-X support. I've since tweaked it to make the MSI-X support off by > >>>> default but possible to enable via loader.conf. Would you be willing to > >>>> try the updated patch at www.freebsd.org/~jhb/patches/mfi.patch? > >>> Hi, > >>> yes I tried the patch you posted originally, unfortunately the dell > >>> never finished booting either. The Supermicro is now in production but > >>> I'll take the dell up to 9-STABLE and try your updated patch. > >>> > >> The patch didnt apply quite cleanly for 9-STABLE, 1 reject as it had > >> already been applied. > > Odd, it's against stock head, so I don't know why it would have failed to > > apply. > > > >> I have rebooted the dell and it seems happy with the new patch (msi > >> disabled.) > > Okay, good. I'll commit the non-MSI bits at least and get them merged into > > 9.0 if possible. > > > >> Booting with > >> hw.mfi.msix=1 in /boot/loader.conf causes the timeouts again and stops > >> the boot from completing. > > Ok. Can you try changing it to use MSI instead of MSI-X? Just edit the > > mfi_pci.c call and replace 'pci_alloc_msix' with 'pci_alloc_msi'. > > > Well the dell has been up for about 19 hours now using MSI, I ran > bonnie++ a few times on it and have now stuck it in a permanent loop > (will look in from time to time.) Are there any tests you'd like > run/info you'd like? Actually, can you please test www.freebsd.org/~jhb/patches/mfi_msi.patch? You will have to set the hw.mfi.msi=1 tunable to enable MSI support. This is a commit candidate if it works. Thanks. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 20:31:43 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 7E660106564A for ; Mon, 14 Nov 2011 20:31:43 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 4F71114DC92; Mon, 14 Nov 2011 20:31:43 +0000 (UTC) Message-ID: <4EC17AAF.9050807@FreeBSD.org> Date: Mon, 14 Nov 2011 12:31:43 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik Subject: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 20:31:43 -0000 Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 in a busy web hosting environment I came across the following post: http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html That basically describes what we're seeing as well, including the "doesn't happen on Linux" part. Does anyone have any ideas about this? With incredibly similar stuff running on 7.x we didn't see this problem, so it seems to be something new in 8. -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 20:49:53 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7D42106566C for ; Mon, 14 Nov 2011 20:49:53 +0000 (UTC) (envelope-from erwin@mail.droso.net) Received: from mail.droso.net (grizzly.droso.net [IPv6:2a01:4f8:100:9424::3]) by mx1.freebsd.org (Postfix) with ESMTP id 7C9768FC12 for ; Mon, 14 Nov 2011 20:49:53 +0000 (UTC) Received: by mail.droso.net (Postfix, from userid 1001) id BB0CCB72F; Mon, 14 Nov 2011 21:49:48 +0100 (CET) Date: Mon, 14 Nov 2011 21:49:48 +0100 From: Erwin Lansing To: freebsd-stable@freebsd.org Message-ID: <20111114204948.GM63757@droso.net> References: <20111112153605.e1ccb1fb54c00d8e233c411e@kleppnett.no> <4EBE9B47.8030301@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <4EBE9B47.8030301@gmail.com> X-Operating-System: FreeBSD/amd64 8.2-RELEASE-p3 User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: pkg_add -r does not find packages FreeBSD 9.0 rc1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 20:49:53 -0000 On Sat, Nov 12, 2011 at 06:13:59PM +0200, Luchesar V. ILIEV wrote: > On 12/11/2011 16:36, Kenneth Hatteland wrote: > > I have installed RC1 and after getting confused with the new install > > routines, I have my system up. But when I try to install packages > > like nano and cvsup-without-gui to build my machine up from base the > > system reports no packages found. This have only rarely happened on > > my other systems and never on a fresh install.....any clues ??? > > Could it be related to this PR? > > http://www.freebsd.org/cgi/query-pr.cgi?pr=162490 > This is an unfortunate sideeffect of our current release process for major version releases where it's hard to synchronize the change needed to pkg_add and the package sets on the mirrors. There are some workarounds that may make this a bit less painfull, but in general this is one of the reasons why we really should aim for maing the src and ports releases independent from eachother. For now, the best option for BETA and RSs is to override PACKAGESITE as Bane Ivosev suggested. Erwin -- Erwin Lansing http://droso.org Prediction is very difficult especially about the future erwin@FreeBSD.org From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 20:51:36 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 226C31065675 for ; Mon, 14 Nov 2011 20:51:36 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id E6045155652; Mon, 14 Nov 2011 20:51:35 +0000 (UTC) Message-ID: <4EC17F57.5030008@FreeBSD.org> Date: Mon, 14 Nov 2011 12:51:35 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@FreeBSD.org References: <4EC17AAF.9050807@FreeBSD.org> In-Reply-To: <4EC17AAF.9050807@FreeBSD.org> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 20:51:36 -0000 On 11/14/2011 12:31, Doug Barton wrote: > Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > in a busy web hosting environment I came across the following post: > > http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > That basically describes what we're seeing as well, including the > "doesn't happen on Linux" part. > > Does anyone have any ideas about this? > > With incredibly similar stuff running on 7.x we didn't see this problem, > so it seems to be something new in 8. Just took a closer look at our ktrace, and actually our pattern is slightly different than the one in that post. In ours the second option is null, but the third is set: 74195 httpd 0.000017 RET sigprocmask 0 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) 74195 httpd 0.000009 RET sigprocmask 0 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) 74195 httpd 0.000009 RET sigprocmask 0 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) But repeated hundreds of times in a row. -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 20:56:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D1291065676; Mon, 14 Nov 2011 20:56:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 658128FC1A; Mon, 14 Nov 2011 20:56:14 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 1DAC946B06; Mon, 14 Nov 2011 15:56:14 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B640D8A050; Mon, 14 Nov 2011 15:56:13 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org Date: Mon, 14 Nov 2011 15:56:12 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EC17AAF.9050807@FreeBSD.org> In-Reply-To: <4EC17AAF.9050807@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111141556.13295.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 14 Nov 2011 15:56:13 -0500 (EST) Cc: Daniil Cherednik , Doug Barton , Konstantin Belousov Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 20:56:14 -0000 On Monday, November 14, 2011 3:31:43 pm Doug Barton wrote: > Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > in a busy web hosting environment I came across the following post: > > http://lists.freebsd.org/pipermail/freebsd-questions/2011- October/234520.html > > That basically describes what we're seeing as well, including the > "doesn't happen on Linux" part. > > Does anyone have any ideas about this? > > With incredibly similar stuff running on 7.x we didn't see this problem, > so it seems to be something new in 8. I suspect it has to do with some of the changes to rtld such that it now always blocks signals while resolving symbols (or something along those lines IIRC). It makes throwing exceptions slow as well. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 21:59:22 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 209A2106566C; Mon, 14 Nov 2011 21:59:22 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id A0266151080; Mon, 14 Nov 2011 21:59:04 +0000 (UTC) Message-ID: <4EC18F28.20307@FreeBSD.org> Date: Mon, 14 Nov 2011 13:59:04 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: John Baldwin References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org> In-Reply-To: <201111141556.13295.jhb@freebsd.org> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , Daniil Cherednik , freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 21:59:22 -0000 On 11/14/2011 12:56, John Baldwin wrote: > On Monday, November 14, 2011 3:31:43 pm Doug Barton wrote: >> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >> in a busy web hosting environment I came across the following post: >> >> http://lists.freebsd.org/pipermail/freebsd-questions/2011- > October/234520.html >> >> That basically describes what we're seeing as well, including the >> "doesn't happen on Linux" part. >> >> Does anyone have any ideas about this? >> >> With incredibly similar stuff running on 7.x we didn't see this problem, >> so it seems to be something new in 8. > > I suspect it has to do with some of the changes to rtld such that it now > always blocks signals while resolving symbols (or something along those > lines IIRC). It makes throwing exceptions slow as well. The calls to sigprocmask() in rtld seem to be doing what you suggest here, but they involve setting and restoring the mask. In my followup post I pasted what we're seeing, which is different, and much more voluminous. For example, 13,500 calls in 30 seconds from a single apache worker process. Although this does seem to explain why our test cases have more calls when compiled with C++ than they do when compiled with C. :) Thanks for the response in any case. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 22:08:59 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A951A1065672 for ; Mon, 14 Nov 2011 22:08:59 +0000 (UTC) (envelope-from stuartb@4gh.net) Received: from smtp02.lnh.mail.rcn.net (smtp02.lnh.mail.rcn.net [207.172.157.102]) by mx1.freebsd.org (Postfix) with ESMTP id 642558FC08 for ; Mon, 14 Nov 2011 22:08:59 +0000 (UTC) Received: from mr16.lnh.mail.rcn.net ([207.172.157.36]) by smtp02.lnh.mail.rcn.net with ESMTP; 14 Nov 2011 16:40:09 -0500 Received: from smtp04.lnh.mail.rcn.net (smtp04.lnh.mail.rcn.net [207.172.157.104]) by mr16.lnh.mail.rcn.net (MOS 4.3.3-GA) with ESMTP id BKI33127; Mon, 14 Nov 2011 16:40:08 -0500 X-Auth-ID: stuartb.4gh@starpower.net Received: from unknown (HELO freeman.4gh.net) ([208.58.4.75]) by smtp04.lnh.mail.rcn.net with ESMTP; 14 Nov 2011 16:40:08 -0500 Received: by freeman.4gh.net (Postfix, from userid 1001) id B71EF130DE4; Mon, 14 Nov 2011 16:40:07 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by freeman.4gh.net (Postfix) with ESMTP id B103D130D36 for ; Mon, 14 Nov 2011 16:40:07 -0500 (EST) Date: Mon, 14 Nov 2011 16:40:07 -0500 (EST) From: Stuart Barkley To: freebsd-stable@freebsd.org In-Reply-To: <20111114204948.GM63757@droso.net> Message-ID: References: <20111112153605.e1ccb1fb54c00d8e233c411e@kleppnett.no> <4EBE9B47.8030301@gmail.com> <20111114204948.GM63757@droso.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Re: pkg_add -r does not find packages FreeBSD 9.0 rc1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 22:08:59 -0000 On Mon, 14 Nov 2011 at 15:49 -0000, Erwin Lansing wrote: > This is an unfortunate sideeffect of our current release process for > major version releases where it's hard to synchronize the change > needed to pkg_add and the package sets on the mirrors. This might be another sign of too frequent major releases. Like some others I would prefer to see fewer major version number changes. A longer lived stable series is preferable (like 3.X and 4.X). Since FreeBSD in not the OS I use most often (unfortunately, since I do prefer BSD), I tend to lag on my personal installations. I'm still at 8.1 and would probably go to 8.3 sometime soon if it became available. I'm not ready for the breakage 9.X might entail for me. Yes, these are just numbers and might not have actual significance. I will probably also be happy to go with 9.0 when released. Common use (and declared FreeBSD use) is that changes with breakage imply the major version number changes and changes without major breakage change the minor version number. I have been the frustrated programmer who is eager to get my code out running on more systems. I have also been the frustrated user waiting for some needed new functionality only in a development/test environment. Trading off the various needs will never satisfy anyone. Note: I'm far more frustrated seeing Firefox hit 8.0 and am glad that Firefox 3.6 is still in ports (even if not the default version). Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 23:16:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07F5B106564A for ; Mon, 14 Nov 2011 23:16:26 +0000 (UTC) (envelope-from gr@ubuntu64.secure-side.com) Received: from admin.secure-side.com (admin.secure-side.com [213.251.166.100]) by mx1.freebsd.org (Postfix) with SMTP id 85F3B8FC16 for ; Mon, 14 Nov 2011 23:16:25 +0000 (UTC) Received: (qmail 26624 invoked from network); 14 Nov 2011 22:49:48 -0000 Received: from localhost (HELO ubuntu64.secure-side.com) (@127.0.0.1) by qmail with SMTP; 14 Nov 2011 22:49:48 -0000 Received: from localhost (localhost [127.0.0.1]) by ubuntu64.secure-side.com (Postfix) with ESMTP id C2165469E5; Mon, 14 Nov 2011 23:49:35 +0100 (CET) Received: from ubuntu64.secure-side.com ([127.0.0.1]) by localhost (ubuntu64.secure-side.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 05vKXvcjZsiu; Mon, 14 Nov 2011 23:49:31 +0100 (CET) Received: from ubuntu64.secure-side.com (ubuntu64.secure-side.com [192.168.1.146]) by ubuntu64.secure-side.com (Postfix) with ESMTP id 150CB46859; Mon, 14 Nov 2011 23:49:31 +0100 (CET) Date: Mon, 14 Nov 2011 23:49:30 +0100 (CET) From: GomoR To: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Message-ID: <40d293fd-1909-4ccd-a32d-750bdba58c21@ubuntu64> In-Reply-To: <4279f879-81b0-4e08-a2d7-63aad377d437@ubuntu64> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Originating-IP: [82.231.148.242] X-Mailer: Zimbra 7.1.3_GA_3346 (ZimbraWebClient - GC15 ([unknown])/7.1.3_GA_3346) Cc: Subject: Something broken with libdnet? Or FreeBSD 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 23:16:26 -0000 Hello list, I was updating one of my Perl module (Net::Libdnet), and came accross a bug under FreeBSD 9.0-RC1 and libdnet. My setup is one interface configured with multiple aliases. See details at the end of the message. One host works ok (FreeBSD 8.2-RELEASE), and another fails to find correct IP information (FreeBSD 9.0-RC1). The main bug is that under 9.0-RC1, libdnet fails to find the correct inet address for the interface. It gets the first alias instead. I don't know if the bug comes from libdnet or 9.0-RC1, so I let knowledgeable people here to look at it (or not). For those who want to test, you can install libdnet from /usr/ports/net/libdnet. Host FreeBSD 8.2-RELEASE (ok): % ifconfig re0 re0: flags=8843 metric 0 mtu 1500 options=389b ether MAC_ADDR inet PUBLIC_IP4 netmask 0xffffff00 broadcast PUBLIC_IP4.255 inet6 LOCAL_IP6%re0 prefixlen 64 scopeid 0x1 inet 192.168.0.10 netmask 0xffffffff broadcast 192.168.0.10 [...many aliases...] inet6 PUBLIC_IP6::2 prefixlen 128 inet6 PUBLIC_IP6::3 prefixlen 128 inet6 PUBLIC_IP6::4 prefixlen 128 inet6 PUBLIC_IP6::5 prefixlen 128 inet6 PUBLIC_IP6::1 prefixlen 56 % dnet intf get re0 re0: flags=0x31 mtu 1500 inet PUBLIC_IP4/24 link MAC_ADDR alias fe80:1::21c:c0ff:feca:1178/64 alias 192.168.0.10 [...many aliases...] alias PUBLIC_IP6::2/0 alias PUBLIC_IP6::3/0 alias PUBLIC_IP6::4/0 alias PUBLIC_IP6::5/0 alias PUBLIC_IP6::1/0 Host FreeBSD 9.0-RC1 (not ok): % ifconfig re0 re0: flags=8943 metric 0 mtu 1500 options=389b ether MAC_ADDR inet 192.168.2.10 netmask 0xffffffff broadcast 192.168.2.10 [...many aliases...] inet 192.168.1.148 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active % dnet intf get re0 re0: flags=0x31 mtu 1500 inet 192.168.2.10 link MAC_ADDR alias 192.168.2.11 alias 192.168.2.12 alias 192.168.2.13 alias 192.168.2.14 alias 192.168.1.148 # Correct inet address -- ^ ___ ___ http://www.GomoR.org/ <-+ | / __ |__/ Senior Security Engineer | | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 23:17:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 106D71065670 for ; Mon, 14 Nov 2011 23:17:42 +0000 (UTC) (envelope-from gr@ubuntu64.secure-side.com) Received: from admin.secure-side.com (admin.secure-side.com [213.251.166.100]) by mx1.freebsd.org (Postfix) with SMTP id 8E01A8FC08 for ; Mon, 14 Nov 2011 23:17:41 +0000 (UTC) Received: (qmail 27563 invoked from network); 14 Nov 2011 22:51:04 -0000 Received: from localhost (HELO ubuntu64.secure-side.com) (@127.0.0.1) by qmail with SMTP; 14 Nov 2011 22:51:04 -0000 Received: from localhost (localhost [127.0.0.1]) by ubuntu64.secure-side.com (Postfix) with ESMTP id D1C20469E5; Mon, 14 Nov 2011 23:50:52 +0100 (CET) Received: from ubuntu64.secure-side.com ([127.0.0.1]) by localhost (ubuntu64.secure-side.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3J7PgkQRaTIZ; Mon, 14 Nov 2011 23:50:47 +0100 (CET) Received: from ubuntu64.secure-side.com (ubuntu64.secure-side.com [192.168.1.146]) by ubuntu64.secure-side.com (Postfix) with ESMTP id B652946859; Mon, 14 Nov 2011 23:50:47 +0100 (CET) Date: Mon, 14 Nov 2011 23:50:47 +0100 (CET) From: GR To: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Message-ID: In-Reply-To: <40d293fd-1909-4ccd-a32d-750bdba58c21@ubuntu64> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Originating-IP: [82.231.148.242] X-Mailer: Zimbra 7.1.3_GA_3346 (ZimbraWebClient - GC15 ([unknown])/7.1.3_GA_3346) Cc: Subject: Something broken with libdnet? Or FreeBSD 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 23:17:42 -0000 Hello list, I was updating one of my Perl module (Net::Libdnet), and came accross a bug under FreeBSD 9.0-RC1 and libdnet. My setup is one interface configured with multiple aliases. See details at the end of the message. One host works ok (FreeBSD 8.2-RELEASE), and another fails to find correct IP information (FreeBSD 9.0-RC1). The main bug is that under 9.0-RC1, libdnet fails to find the correct inet address for the interface. It gets the first alias instead. I don't know if the bug comes from libdnet or 9.0-RC1, so I let knowledgeable people here to look at it (or not). For those who want to test, you can install libdnet from /usr/ports/net/libdnet. Host FreeBSD 8.2-RELEASE (ok): % ifconfig re0 re0: flags=8843 metric 0 mtu 1500 options=389b ether MAC_ADDR inet PUBLIC_IP4 netmask 0xffffff00 broadcast PUBLIC_IP4.255 inet6 LOCAL_IP6%re0 prefixlen 64 scopeid 0x1 inet 192.168.0.10 netmask 0xffffffff broadcast 192.168.0.10 [...many aliases...] inet6 PUBLIC_IP6::2 prefixlen 128 inet6 PUBLIC_IP6::3 prefixlen 128 inet6 PUBLIC_IP6::4 prefixlen 128 inet6 PUBLIC_IP6::5 prefixlen 128 inet6 PUBLIC_IP6::1 prefixlen 56 % dnet intf get re0 re0: flags=0x31 mtu 1500 inet PUBLIC_IP4/24 link MAC_ADDR alias fe80:1::21c:c0ff:feca:1178/64 alias 192.168.0.10 [...many aliases...] alias PUBLIC_IP6::2/0 alias PUBLIC_IP6::3/0 alias PUBLIC_IP6::4/0 alias PUBLIC_IP6::5/0 alias PUBLIC_IP6::1/0 Host FreeBSD 9.0-RC1 (not ok): % ifconfig re0 re0: flags=8943 metric 0 mtu 1500 options=389b ether MAC_ADDR inet 192.168.2.10 netmask 0xffffffff broadcast 192.168.2.10 [...many aliases...] inet 192.168.1.148 netmask 0xffffff00 broadcast 192.168.1.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active % dnet intf get re0 re0: flags=0x31 mtu 1500 inet 192.168.2.10 link MAC_ADDR alias 192.168.2.11 alias 192.168.2.12 alias 192.168.2.13 alias 192.168.2.14 alias 192.168.1.148 # Correct inet address -- ^ ___ ___ http://www.GomoR.org/ <-+ | / __ |__/ Senior Security Engineer | | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ From owner-freebsd-stable@FreeBSD.ORG Mon Nov 14 23:56:34 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F6BA1065675 for ; Mon, 14 Nov 2011 23:56:34 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 8DA188FC17 for ; Mon, 14 Nov 2011 23:56:33 +0000 (UTC) Received: from vhoffman-macbooklocal.local ([10.10.10.20]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pAENuUlL083414 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Mon, 14 Nov 2011 23:56:31 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EC1AAAE.1030007@unsane.co.uk> Date: Mon, 14 Nov 2011 23:56:30 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: FreeBSD Stable Mailing List X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Nov 2011 23:56:34 -0000 I've been trying to work out why my nfs mounts seem to be a little speed limited (thats another email though) and though I'd go through the stats netstat makes available. >From the manpage netstat -i | -I interface -s [-f protocol_family | -p protocol] [-M core] [-N system] Display per-interface statistics for each network protocol, for a particular protocol_family, or for a single protocol. I was expecting to be able to see at least some tcp/ip stats per interface, but [root@banshee ~]# ifconfig -l igb0 igb1 lo0 lagg0 lagg0.53 lagg0.52 pflog0 lagg0.66 [root@banshee ~]# for int in $(ifconfig -l) ; do netstat -I $int -s -f inet; done [root@banshee ~]# I can get system wide using netstat -s -f inet and if use netstat -I $int -s I get a bunch of ipv6 stats. and of course i can get the stats from netstat -I $int too. Is this expected behaviour? if so maybe the manpage could be made a little clearer. Thanks, Vince ps tested on [root@ostracod ~]# uname -a FreeBSD ostracod.unsane.co.uk 9.0-RC1 FreeBSD 9.0-RC1 #15 r226879: Fri Oct 28 22:25:26 BST 2011 toor@ostracod.unsane.co.uk:/usr/obj/usr/src/sys/OSTRACOD amd64 [root@ostracod ~]# FreeBSD banshee.namesco.net 8.2-STABLE FreeBSD 8.2-STABLE #5: Wed Nov 2 14:53:03 GMT 2011 toor@banshee.lon.domain.com:/usr/obj/usr/src/sys/BANSHEE amd64 FreeBSD zfstest 9.0-RC2 FreeBSD 9.0-RC2 #0 r227497M: Mon Nov 14 16:14:54 GMT 2011 toor@zfstest:/usr/obj/usr/src/sys/ZFSTEST amd64 From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 00:27:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92AA3106566C for ; Tue, 15 Nov 2011 00:27:21 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 2066B8FC16 for ; Tue, 15 Nov 2011 00:27:20 +0000 (UTC) Received: (qmail 12061 invoked by uid 907); 15 Nov 2011 00:27:18 -0000 Received: from jmmacpro.transactionware.com (HELO jmmacpro.transactionware.com) (192.168.1.33) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Tue, 15 Nov 2011 11:27:18 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Jan Mikkelsen In-Reply-To: <201111141136.01348.jhb@freebsd.org> Date: Tue, 15 Nov 2011 11:27:18 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4EA9E0C3.5080306@unsane.co.uk> <4EB9BD9B.8080604@unsane.co.uk> <8F475C3C-35FE-44CD-8FD6-E39F9A2EA6C6@transactionware.com> <201111141136.01348.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 00:27:21 -0000 Hi, Sorry about being unclear. They all failed in the same way. So, these combinations have continuous timeout errors and fail to = completely boot: Plain 9.0-RC1 9-stable with 1.62 of mfi.c 9-stable with www.freebsd.org/~jhb/patches/mfi.patch, pci_alloc_msi instead of pci_alloc_msix and hw.mfi.msix=3D0 This boots, but gets "mfi0: Cannot allocate interrupt" and there are no = /dev/mfi* devices: 9-stable with www.freebsd.org/~jhb/patches/mfi.patch and hw.mfi.msix=3D1= This seems to work, but I have not put any load on it yet: 9-stable with www.freebsd.org/~jhb/patches/mfi.patch, pci_alloc_msi instead of pci_alloc_msix and hw.mfi.msix=3D1 I see you have a new patch, www.freebsd.org/~jhb/patches/mfi_msi.patch. = This patch doesn't seem to include the dummy read from your earlier = patch, or the one in 1.62 of mfi.c. I assume I need to apply the 1.62 = mfi.c diff to by 9-stable sources as well. Is that correct? Will test out the new patch. Thanks, Jan Mikkelsen On 15/11/2011, at 3:36 AM, John Baldwin wrote: > On Monday, November 14, 2011 3:03:42 am Jan Mikkelsen wrote: >> Hi, >>=20 >> I have just tested mfi on a machine that has just arrived. I'm seeing = the command timeout problem on boot with 9.0-RC1. >>=20 >> The message is "mfi0: COMMAND TIMEOUT AFTER 59 seconds", and = then repeats every 30 seconds (with the time changed, obviously). >>=20 >> I have tested the 9.0-RC1 ISO, a 9-stable kernel patched with the = patch from the PR I referenced below (also in FreeBSD cvs in revision = 1.62 of=20 > src/sys/dev/mfi/mfi.c), and 9-stable kernel with the patch at = www.freebsd.org/~jhb/patches/mfi.patch with the 'pci_alloc_msix' call = changed to=20 > 'pci_alloc_msi'. >=20 > You forgot to mention what happened from those tests, did any of them = work? >=20 > --=20 > John Baldwin From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 00:32:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 816EB1065674 for ; Tue, 15 Nov 2011 00:32:24 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.emeryville.ca.mail.comcast.net (qmta02.emeryville.ca.mail.comcast.net [76.96.30.24]) by mx1.freebsd.org (Postfix) with ESMTP id 6A7D88FC17 for ; Tue, 15 Nov 2011 00:32:24 +0000 (UTC) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta02.emeryville.ca.mail.comcast.net with comcast id xCY91h0090FhH24A2CYGrk; Tue, 15 Nov 2011 00:32:16 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta08.emeryville.ca.mail.comcast.net with comcast id xCXd1h01a1t3BNj8UCXeDe; Tue, 15 Nov 2011 00:31:38 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id CC250102C1D; Mon, 14 Nov 2011 16:32:22 -0800 (PST) Date: Mon, 14 Nov 2011 16:32:22 -0800 From: Jeremy Chadwick To: Vincent Hoffman Message-ID: <20111115003222.GA82751@icarus.home.lan> References: <4EC1AAAE.1030007@unsane.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EC1AAAE.1030007@unsane.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Mailing List Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 00:32:24 -0000 On Mon, Nov 14, 2011 at 11:56:30PM +0000, Vincent Hoffman wrote: > I've been trying to work out why my nfs mounts seem to be a little speed > limited (thats another email though) and though I'd go through the stats > netstat makes available. > >From the manpage > > netstat -i | -I interface -s [-f protocol_family | -p protocol] [-M > core] > [-N system] > Display per-interface statistics for each network protocol, > for a > particular protocol_family, or for a single protocol. > > > I was expecting to be able to see at least some tcp/ip stats per > interface, but > [root@banshee ~]# ifconfig -l > igb0 igb1 lo0 lagg0 lagg0.53 lagg0.52 pflog0 lagg0.66 > [root@banshee ~]# for int in $(ifconfig -l) ; do netstat -I $int -s -f > inet; done > [root@banshee ~]# > > I can get system wide using netstat -s -f inet and if use netstat -I > $int -s I get a bunch of ipv6 stats. > and of course i can get the stats from netstat -I $int too. > > Is this expected behaviour? if so maybe the manpage could be made a > little clearer. > > Thanks, > Vince > > ps tested on > [root@ostracod ~]# uname -a > FreeBSD ostracod.unsane.co.uk 9.0-RC1 FreeBSD 9.0-RC1 #15 r226879: Fri > Oct 28 22:25:26 BST 2011 > toor@ostracod.unsane.co.uk:/usr/obj/usr/src/sys/OSTRACOD amd64 > [root@ostracod ~]# > FreeBSD banshee.namesco.net 8.2-STABLE FreeBSD 8.2-STABLE #5: Wed Nov 2 > 14:53:03 GMT 2011 > toor@banshee.lon.domain.com:/usr/obj/usr/src/sys/BANSHEE amd64 > > FreeBSD zfstest 9.0-RC2 FreeBSD 9.0-RC2 #0 r227497M: Mon Nov 14 16:14:54 > GMT 2011 toor@zfstest:/usr/obj/usr/src/sys/ZFSTEST amd64 On my system: netstat -s = returns system-wide statistics for all IP-related bits netstat -I {iface} = returns network I/O statistics and related bits associated with that interface netstat -I {iface} -s = returns nothing (no output) netstat -I {iface} -f inet = same as "netstat -I {iface}" but only with IPv4 specific data netstat -I {iface} -s -f inet = returns nothing (no output) I think what you're actually looking for is, BTW: netstat -I em0 -inbd And if you want real-time I/O rates: netstat -I em0 -bd 1 Where "1" is the number of seconds between polling iterations. I don't particularly care what the man page says, I just know what works. :-) If you're wanting "netstat -s" output but on a per-interface basis, those statistics are not tracked per-interface. Maybe netgraph can do that for you, I do not know. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 01:22:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D838106566B for ; Tue, 15 Nov 2011 01:22:56 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from glenbarber.us (onyx.glenbarber.us [199.48.134.227]) by mx1.freebsd.org (Postfix) with SMTP id 97A278FC0A for ; Tue, 15 Nov 2011 01:22:55 +0000 (UTC) Received: (qmail 16671 invoked by uid 0); 14 Nov 2011 19:56:14 -0500 Received: from unknown (HELO glenbarber.us) (76.124.49.145) by 0 with SMTP; 14 Nov 2011 19:56:14 -0500 Date: Mon, 14 Nov 2011 19:56:07 -0500 From: Glen Barber To: Jeremy Chadwick Message-ID: <20111115005607.GA21345@glenbarber.us> References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="+QahgC5+KEYLbs62" Content-Disposition: inline In-Reply-To: <20111115003222.GA82751@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Mailing List , Vincent Hoffman Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 01:22:56 -0000 --+QahgC5+KEYLbs62 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Jeremy Chadwick wrote:=20 [...] > I don't particularly care what the man page says, I just know what > works. :-) >=20 If there is something in the man page that is not precise, I personally am interested in any inconsistencies with reality so they can be fixed. Even given your detailed explanation, I'm still not 100% clear on what, if anything, is incorrect in the manual. Regards, Glen --=20 Glen Barber | gjb@FreeBSD.org FreeBSD Documentation Project --+QahgC5+KEYLbs62 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) iQEcBAEBCAAGBQJOwbinAAoJEFJPDDeguUajx3oH/1NRE3yQaDKmQsRHqp2D94bn XVVLx+m9MTIWtBsuceBuO3qsUORhG4n3eh7BkrmQIR+cqRFnZfBzf5nwA7e4jGkq dGZO6IWpliSyN6W060PSZcGLZHDdrFcmttuLMlc1spuqc22S1tgXIrK4z12LA93e AzWM8JS8O3mgDhUCJ3xWXqQstU2Hc+7xql9UuV5fbHWir+53McVQm/E1aLUoEDoY ID+bhbzCaZkagu+IxE+lAioGDSeqi5hjC/cWxqQK0BwlABoHTURb9FJnVMBvlbQN /uXyhquWRaomNPcDK6wMc1viz70lVzRTXMKPLCYGY299aqctRjGugHatHQ8I6I8= =P0t4 -----END PGP SIGNATURE----- --+QahgC5+KEYLbs62-- From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 01:46:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 898F01065670 for ; Tue, 15 Nov 2011 01:46:03 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.emeryville.ca.mail.comcast.net (qmta14.emeryville.ca.mail.comcast.net [76.96.27.212]) by mx1.freebsd.org (Postfix) with ESMTP id 70C478FC0C for ; Tue, 15 Nov 2011 01:46:03 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta14.emeryville.ca.mail.comcast.net with comcast id xDaV1h0081afHeLAEDlv4b; Tue, 15 Nov 2011 01:45:55 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta17.emeryville.ca.mail.comcast.net with comcast id xDXn1h00Z1t3BNj8dDXnDD; Tue, 15 Nov 2011 01:31:47 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6DE5A102C1D; Mon, 14 Nov 2011 17:46:02 -0800 (PST) Date: Mon, 14 Nov 2011 17:46:02 -0800 From: Jeremy Chadwick To: Glen Barber Message-ID: <20111115014602.GA84020@icarus.home.lan> References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> <20111115005607.GA21345@glenbarber.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111115005607.GA21345@glenbarber.us> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Mailing List , Vincent Hoffman Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 01:46:03 -0000 On Mon, Nov 14, 2011 at 07:56:07PM -0500, Glen Barber wrote: > Jeremy Chadwick wrote: > > [...] > > > I don't particularly care what the man page says, I just know what > > works. :-) > > If there is something in the man page that is not precise, I personally > am interested in any inconsistencies with reality so they can be fixed. > Even given your detailed explanation, I'm still not 100% clear on what, > if anything, is incorrect in the manual. FWIW, neither am I. The OP seems to be basing his argument syntaxes based on what's in the man page, so if something is vague, confusing, or inaccurate, he will need to state clearly what it is that's an anomaly. My comment was more along the lines of: "if it's wrong in the man page, I'm really not that concerned" (e.g. file a PR and be done with it). I'm still trying to work out exactly what it is the OP wants statistics-wise on a per-interface basis. I'm left thinking he wants "netstat -s" output per-interface, but that's not going to happen. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 01:52:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1B7E106566B for ; Tue, 15 Nov 2011 01:52:30 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from glenbarber.us (onyx.glenbarber.us [199.48.134.227]) by mx1.freebsd.org (Postfix) with SMTP id 77F2F8FC0A for ; Tue, 15 Nov 2011 01:52:30 +0000 (UTC) Received: (qmail 17515 invoked by uid 0); 14 Nov 2011 20:52:29 -0500 Received: from unknown (HELO glenbarber.us) (76.124.49.145) by 0 with SMTP; 14 Nov 2011 20:52:29 -0500 Date: Mon, 14 Nov 2011 20:52:23 -0500 From: Glen Barber To: Jeremy Chadwick Message-ID: <20111115015223.GA22167@glenbarber.us> References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> <20111115005607.GA21345@glenbarber.us> <20111115014602.GA84020@icarus.home.lan> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="OXfL5xGRrasGEqWY" Content-Disposition: inline In-Reply-To: <20111115014602.GA84020@icarus.home.lan> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Mailing List , Vincent Hoffman Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 01:52:30 -0000 --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 14, 2011 at 05:46:02PM -0800, Jeremy Chadwick wrote: > On Mon, Nov 14, 2011 at 07:56:07PM -0500, Glen Barber wrote: > > Jeremy Chadwick wrote:=20 > >=20 > > [...] > >=20 > > > I don't particularly care what the man page says, I just know what > > > works. :-) > >=20 > > If there is something in the man page that is not precise, I personally > > am interested in any inconsistencies with reality so they can be fixed. > > Even given your detailed explanation, I'm still not 100% clear on what, > > if anything, is incorrect in the manual. >=20 > FWIW, neither am I. The OP seems to be basing his argument syntaxes > based on what's in the man page, so if something is vague, confusing, > or inaccurate, he will need to state clearly what it is that's an > anomaly. >=20 > My comment was more along the lines of: "if it's wrong in the man page, > I'm really not that concerned" (e.g. file a PR and be done with it). > I'm still trying to work out exactly what it is the OP wants > statistics-wise on a per-interface basis. I'm left thinking he wants > "netstat -s" output per-interface, but that's not going to happen. >=20 Got it, thanks for clarifying. I (mis)read your statement as "the manual is wrong, but here's the correct explanation." Thanks. Glen --=20 Glen Barber | gjb@FreeBSD.org FreeBSD Documentation Project --OXfL5xGRrasGEqWY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) iQEcBAEBCAAGBQJOwcXXAAoJEFJPDDeguUajWKQIAIisuvDFy1/0TIS9XiXwB1r2 cf2/aB2ER68brjP/jMmf5VsqLvGR58TaQwNlIG0HBoXnNjAlwXRvIZ/cplUk+ogQ OEh9NHZjZCSRysnal099c+rNZslArZwB0Q+8As8SwRBmQJJocuL7BsKtoTGRNaHz kkzmG4BpE7c4IkQDYPQkGWhMbTmbo+2T0ubeAZ3E7B+pfE9Snkhz8PHnZAgCtW6m BZZLU9W0FnO00wjgOObFAAPoFrlN15e7iufa0WQqsU2SH5rj76kHuSY70I+h+d7R FXCAAPyqAn/4JUfCcdqBQ/mUhI30AlGZm2AAsb1JD4G7jtZAhpqsJZpx2VCQgeY= =6K1Z -----END PGP SIGNATURE----- --OXfL5xGRrasGEqWY-- From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 08:33:05 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1E781065677 for ; Tue, 15 Nov 2011 08:33:05 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id D88B78FC08 for ; Tue, 15 Nov 2011 08:33:04 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Tue, 15 Nov 2011 12:32:01 +0400 Message-ID: <4EC22380.20509@masterhost.ru> Date: Tue, 15 Nov 2011 12:32:00 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org> <4EC18F28.20307@FreeBSD.org> In-Reply-To: <4EC18F28.20307@FreeBSD.org> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 08:33:05 -0000 Hi. I was trying to understand a cause for this problem, and made an ugly hack: diff -u ./rtld_lock.c.orig ./rtld_lock.c --- ./rtld_lock.c.orig 2011-11-15 07:56:14.000000000 +0000 +++ ./rtld_lock.c 2011-11-15 07:54:42.000000000 +0000 @@ -118,7 +118,7 @@ sigset_t tmp_oldsigmask; for ( ; ; ) { - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); +// sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) break; sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); @@ -135,7 +135,7 @@ atomic_add_rel_int(&l->lock, -RC_INCR); else { atomic_add_rel_int(&l->lock, -WAFLAG); - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); +// sigprocmask(SIG_SETMASK, &oldsigmask, NULL); } } this reduced number of syscalls, but I am not sure about stability and correctness of this hack. And performance problems of apache remained (this hack only gave 5-10% increase of performance). Also I found problem in longjmp syscalls. In FreeBSD we are doing this syscalls from gen/setjmp.S without condition check, for example in Linux we are doing it only if stack has been saved. Linux code: if (env[0].__mask_was_saved) /* Restore the saved signal mask. */ (void) __sigprocmask (SIG_SETMASK, &env[0].__saved_mask, (sigset_t *) NULL); After all that I was trying to compare perfomance of return from fork() in Linux and FreeBSD (see http://lists.freebsd.org/pipermail/freebsd-hackers/2011-October/036705.html) and fork() in FreeBSD was slower. I am not trying to start a holy war, but I really need to increase performance of our hosting in FreeBSD. On 15.11.2011 01:59, Doug Barton wrote: > On 11/14/2011 12:56, John Baldwin wrote: >> On Monday, November 14, 2011 3:31:43 pm Doug Barton wrote: >>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>> in a busy web hosting environment I came across the following post: >>> >>> http://lists.freebsd.org/pipermail/freebsd-questions/2011- >> October/234520.html >>> That basically describes what we're seeing as well, including the >>> "doesn't happen on Linux" part. >>> >>> Does anyone have any ideas about this? >>> >>> With incredibly similar stuff running on 7.x we didn't see this problem, >>> so it seems to be something new in 8. >> I suspect it has to do with some of the changes to rtld such that it now >> always blocks signals while resolving symbols (or something along those >> lines IIRC). It makes throwing exceptions slow as well. > The calls to sigprocmask() in rtld seem to be doing what you suggest > here, but they involve setting and restoring the mask. In my followup > post I pasted what we're seeing, which is different, and much more > voluminous. For example, 13,500 calls in 30 seconds from a single apache > worker process. > > Although this does seem to explain why our test cases have more calls > when compiled with C++ than they do when compiled with C. :) > > Thanks for the response in any case. > > > Doug > -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 08:33:20 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C015E10657AB; Tue, 15 Nov 2011 08:33:20 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id F26658FC0C; Tue, 15 Nov 2011 08:33:19 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Tue, 15 Nov 2011 12:22:20 +0400 Message-ID: <4EC2213B.7030005@masterhost.ru> Date: Tue, 15 Nov 2011 12:22:19 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org> <4EC18F28.20307@FreeBSD.org> In-Reply-To: <4EC18F28.20307@FreeBSD.org> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] Cc: Doug Barton , Konstantin Belousov , John Baldwin Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 08:33:20 -0000 Hi. I was trying to understand a cause for this problem, and made an ugly hack: diff -u ./rtld_lock.c.orig ./rtld_lock.c --- ./rtld_lock.c.orig 2011-11-15 07:56:14.000000000 +0000 +++ ./rtld_lock.c 2011-11-15 07:54:42.000000000 +0000 @@ -118,7 +118,7 @@ sigset_t tmp_oldsigmask; for ( ; ; ) { - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); +// sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) break; sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); @@ -135,7 +135,7 @@ atomic_add_rel_int(&l->lock, -RC_INCR); else { atomic_add_rel_int(&l->lock, -WAFLAG); - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); +// sigprocmask(SIG_SETMASK, &oldsigmask, NULL); } } this reduced number of syscalls, but I am not sure about stability and correctness of this hack. And performance problems of apache remained (this hack only gave 5-10% increase of performance). Also I found problem in longjmp syscalls. In FreeBSD we are doing this syscalls from gen/setjmp.S without condition check, for example in Linux we are doing it only if stack has been saved. Linux code: if (env[0].__mask_was_saved) /* Restore the saved signal mask. */ (void) __sigprocmask (SIG_SETMASK, &env[0].__saved_mask, (sigset_t *) NULL); After all that I was trying to compare perfomance of return from fork() in Linux and FreeBSD (see http://lists.freebsd.org/pipermail/freebsd-hackers/2011-October/036705.html) and fork() in FreeBSD was slower. I am not trying to start a holy war, but I really need to increase performance of our hosting in FreeBSD. On 15.11.2011 01:59, Doug Barton wrote: > On 11/14/2011 12:56, John Baldwin wrote: >> On Monday, November 14, 2011 3:31:43 pm Doug Barton wrote: >>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>> in a busy web hosting environment I came across the following post: >>> >>> http://lists.freebsd.org/pipermail/freebsd-questions/2011- >> October/234520.html >>> That basically describes what we're seeing as well, including the >>> "doesn't happen on Linux" part. >>> >>> Does anyone have any ideas about this? >>> >>> With incredibly similar stuff running on 7.x we didn't see this problem, >>> so it seems to be something new in 8. >> I suspect it has to do with some of the changes to rtld such that it now >> always blocks signals while resolving symbols (or something along those >> lines IIRC). It makes throwing exceptions slow as well. > The calls to sigprocmask() in rtld seem to be doing what you suggest > here, but they involve setting and restoring the mask. In my followup > post I pasted what we're seeing, which is different, and much more > voluminous. For example, 13,500 calls in 30 seconds from a single apache > worker process. > > Although this does seem to explain why our test cases have more calls > when compiled with C++ than they do when compiled with C. :) > > Thanks for the response in any case. > > > Doug > -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 08:48:57 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F8AF1065673 for ; Tue, 15 Nov 2011 08:48:57 +0000 (UTC) (envelope-from amarat@ksu.ru) Received: from webmail.hitv.ru (mail.hitv.ru [217.66.16.37]) by mx1.freebsd.org (Postfix) with ESMTP id AC9DB8FC08 for ; Tue, 15 Nov 2011 08:48:56 +0000 (UTC) Received: from webmail.hitv.ru (localhost [127.0.0.1]) by webmail.hitv.ru (Postfix) with ESMTP id AEEA64AD1F4; Tue, 15 Nov 2011 12:48:54 +0400 (MSK) Received: from zealot.ksu.ru (zealot.hitv.ru [83.151.8.230]) by webmail.hitv.ru (Postfix) with ESMTP id 7592F4AD1F3; Tue, 15 Nov 2011 12:48:54 +0400 (MSK) Received: from zealot.ksu.ru (localhost.lnet [127.0.0.1]) by zealot.ksu.ru (8.14.4/8.14.4) with ESMTP id pAF8msAT000312; Tue, 15 Nov 2011 11:48:54 +0300 (MSK) (envelope-from amarat@ksu.ru) Message-ID: <4EC22776.1080204@ksu.ru> Date: Tue, 15 Nov 2011 12:48:54 +0400 From: "Marat N.Afanasyev" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111009 Firefox/7.0.1 SeaMonkey/2.4.1 MIME-Version: 1.0 To: Daniil Cherednik References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org> <4EC18F28.20307@FreeBSD.org> <4EC22380.20509@masterhost.ru> In-Reply-To: <4EC22380.20509@masterhost.ru> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms080807090208090905000907" X-Virus-Scanned: ClamAV using ClamSMTP X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 08:48:57 -0000 This is a cryptographically signed message in MIME format. --------------ms080807090208090905000907 Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: quoted-printable Daniil Cherednik wrote: > After all that I was trying to compare perfomance of return from fork()= > in Linux and FreeBSD (see > http://lists.freebsd.org/pipermail/freebsd-hackers/2011-October/036705.= html) > and fork() in FreeBSD was slower. our fork() differs from linux fork() in handling parent and child memory = pages, and ours is generally slower, that's why sphinxsearch (prior to=20 1.1 with its prefork model) was unusable on FreeBSD under moderate and=20 heavy load, too many forks %) --=20 SY, Marat --------------ms080807090208090905000907-- From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 09:20:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54F841065672 for ; Tue, 15 Nov 2011 09:20:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 8CEEF8FC0A for ; Tue, 15 Nov 2011 09:20:14 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAF97k9O046285 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 15 Nov 2011 11:07:47 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAF97kVt002458; Tue, 15 Nov 2011 11:07:46 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAF97kIX002457; Tue, 15 Nov 2011 11:07:46 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 15 Nov 2011 11:07:45 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111115090745.GO50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tB7rUN7dYh5/jfh0" Content-Disposition: inline In-Reply-To: <4EC17F57.5030008@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 09:20:15 -0000 --tB7rUN7dYh5/jfh0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > On 11/14/2011 12:31, Doug Barton wrote: > > Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > in a busy web hosting environment I came across the following post: > >=20 > > http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/23452= 0.html > >=20 > > That basically describes what we're seeing as well, including the > > "doesn't happen on Linux" part. > >=20 > > Does anyone have any ideas about this? > >=20 > > With incredibly similar stuff running on 7.x we didn't see this problem, > > so it seems to be something new in 8. >=20 > Just took a closer look at our ktrace, and actually our pattern is > slightly different than the one in that post. In ours the second option > is null, but the third is set: >=20 > 74195 httpd 0.000017 RET sigprocmask 0 > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > 74195 httpd 0.000009 RET sigprocmask 0 > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > 74195 httpd 0.000009 RET sigprocmask 0 > 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >=20 > But repeated hundreds of times in a row. The calls cannot come from rtld, they are generated by some setjmp() invocation. If signal-safety is not needed, sigsetjmp() should be used instead. Quick grep of the apache httpd source shows a single setjmp() in their copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). --tB7rUN7dYh5/jfh0 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEUEARECAAYFAk7CK+EACgkQC3+MBN1Mb4g34ACeIehqW4571CMhvULXoPb/YAmr eCMAl2asO0EOax3iKkh0SMh81Y/X5Og= =TdFv -----END PGP SIGNATURE----- --tB7rUN7dYh5/jfh0-- From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 10:09:07 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 756981065672 for ; Tue, 15 Nov 2011 10:09:07 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 21CA28FC0A for ; Tue, 15 Nov 2011 10:09:06 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta07.westchester.pa.mail.comcast.net with comcast id xN8G1h0011ei1Bg57N97yk; Tue, 15 Nov 2011 10:09:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id xN951h0151t3BNj3kN96c0; Tue, 15 Nov 2011 10:09:07 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 7366E102C1D; Tue, 15 Nov 2011 02:09:04 -0800 (PST) Date: Tue, 15 Nov 2011 02:09:04 -0800 From: Jeremy Chadwick To: Kostik Belousov Message-ID: <20111115100904.GA92795@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111115090745.GO50300@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Daniil Cherednik , Doug Barton , freebsd-stable@freebsd.org, freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 10:09:07 -0000 On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > > On 11/14/2011 12:31, Doug Barton wrote: > > > Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > > in a busy web hosting environment I came across the following post: > > > > > > http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > > > > > That basically describes what we're seeing as well, including the > > > "doesn't happen on Linux" part. > > > > > > Does anyone have any ideas about this? > > > > > > With incredibly similar stuff running on 7.x we didn't see this problem, > > > so it seems to be something new in 8. > > > > Just took a closer look at our ktrace, and actually our pattern is > > slightly different than the one in that post. In ours the second option > > is null, but the third is set: > > > > 74195 httpd 0.000017 RET sigprocmask 0 > > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > 74195 httpd 0.000009 RET sigprocmask 0 > > 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > 74195 httpd 0.000009 RET sigprocmask 0 > > 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > > > But repeated hundreds of times in a row. > > The calls cannot come from rtld, they are generated by some setjmp() > invocation. If signal-safety is not needed, sigsetjmp() should be used > instead. > > Quick grep of the apache httpd source shows a single setjmp() in their > copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). I hate cross-posting, but: adding freebsd-apache@ to the list. Some of the Apache folks (not just port committers) may have some insight to Kostik's findings. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 10:28:30 2011 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 214AF106564A; Tue, 15 Nov 2011 10:28:30 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 892CA8FC14; Tue, 15 Nov 2011 10:28:29 +0000 (UTC) Received: from vhoffman.lon.namesco.net (lon.namesco.net [195.7.254.102]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pAFASRDY007319 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 15 Nov 2011 10:28:28 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EC23ECB.3000805@unsane.co.uk> Date: Tue, 15 Nov 2011 10:28:27 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> <20111115005607.GA21345@glenbarber.us> <20111115014602.GA84020@icarus.home.lan> In-Reply-To: <20111115014602.GA84020@icarus.home.lan> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Glen Barber , FreeBSD Stable Mailing List Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 10:28:30 -0000 On 15/11/2011 01:46, Jeremy Chadwick wrote: > On Mon, Nov 14, 2011 at 07:56:07PM -0500, Glen Barber wrote: >> Jeremy Chadwick wrote: >> >> [...] >> >>> I don't particularly care what the man page says, I just know what >>> works. :-) >> If there is something in the man page that is not precise, I personally >> am interested in any inconsistencies with reality so they can be fixed. >> Even given your detailed explanation, I'm still not 100% clear on what, >> if anything, is incorrect in the manual. > FWIW, neither am I. The OP seems to be basing his argument syntaxes > based on what's in the man page, so if something is vague, confusing, > or inaccurate, he will need to state clearly what it is that's an > anomaly. > > My comment was more along the lines of: "if it's wrong in the man page, > I'm really not that concerned" (e.g. file a PR and be done with it). > I'm still trying to work out exactly what it is the OP wants > statistics-wise on a per-interface basis. I'm left thinking he wants > "netstat -s" output per-interface, but that's not going to happen. > You are correct, my reading of the manpage suggested that per interface stats should be available. netstat -i | -I interface -s [-f protocol_family | -p protocol] [-M core] [-N system] Display per-interface statistics for each network protocol, for a particular protocol_family, or for a single protocol. As that was incorrect then I do consider the manpage to need updating. I'll file a PR but I wasnt sure if this was correct behavior or not. Thanks, Vince From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 11:44:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21252106564A for ; Tue, 15 Nov 2011 11:44:48 +0000 (UTC) (envelope-from prvs=1300dcceb8=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 9DF8D8FC12 for ; Tue, 15 Nov 2011 11:44:47 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Tue, 15 Nov 2011 11:33:50 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 15 Nov 2011 11:33:50 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50016405679.msg for ; Tue, 15 Nov 2011 11:33:50 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1300dcceb8=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-stable@freebsd.org Message-ID: <9C9F2187F38F47F6BC0138413E445184@multiplay.co.uk> From: "Steven Hartland" To: "Daniil Cherednik" , References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org><4EC18F28.20307@FreeBSD.org> <4EC22380.20509@masterhost.ru> Date: Tue, 15 Nov 2011 11:34:40 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="UTF-8"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 11:44:48 -0000 ----- Original Message ----- From: "Daniil Cherednik" > I am not trying to start a holy war, but I really need to increase > performance of our hosting in FreeBSD. Is there something you need from apache that means you cant use nginx for instead? nginx + php-fpm is much higher performing, we switch a year ago and would never go back. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 11:46:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52035106566C for ; Tue, 15 Nov 2011 11:46:19 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id C057E8FC0A for ; Tue, 15 Nov 2011 11:46:18 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Tue, 15 Nov 2011 15:46:16 +0400 Message-ID: <4EC25107.7010901@masterhost.ru> Date: Tue, 15 Nov 2011 15:46:15 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: Steven Hartland References: <4EC17AAF.9050807@FreeBSD.org> <201111141556.13295.jhb@freebsd.org><4EC18F28.20307@FreeBSD.org> <4EC22380.20509@masterhost.ru> <9C9F2187F38F47F6BC0138413E445184@multiplay.co.uk> In-Reply-To: <9C9F2187F38F47F6BC0138413E445184@multiplay.co.uk> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] Cc: freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 11:46:19 -0000 We know about it. But unfortunately we can`t use php-fpm or other fcgi solution. We must use .htaccess with php directive. On 15.11.2011 15:34, Steven Hartland wrote: > ----- Original Message ----- From: "Daniil Cherednik" > > >> I am not trying to start a holy war, but I really need to increase >> performance of our hosting in FreeBSD. > > Is there something you need from apache that means you > cant use nginx for instead? > > nginx + php-fpm is much higher performing, we switch a year > ago and would never go back. > > Regards > Steve > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. > and the person or entity to whom it is addressed. In the event of > misdirection, the recipient is prohibited from using, copying, > printing or otherwise disseminating it or any information contained in > it. > In the event of misdirection, illegible or incomplete transmission > please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. > -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 15:52:54 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E9216106566C for ; Tue, 15 Nov 2011 15:52:54 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id BF0D38FC12 for ; Tue, 15 Nov 2011 15:52:54 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id CB76546B17; Tue, 15 Nov 2011 10:52:53 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 5AF8C8A050; Tue, 15 Nov 2011 10:52:53 -0500 (EST) From: John Baldwin To: Jan Mikkelsen Date: Tue, 15 Nov 2011 08:45:15 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EA9E0C3.5080306@unsane.co.uk> <201111141136.01348.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111150845.15833.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 15 Nov 2011 10:52:53 -0500 (EST) Cc: freebsd-stable@freebsd.org, Jeremy Chadwick , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 15:52:55 -0000 On Monday, November 14, 2011 7:27:18 pm Jan Mikkelsen wrote: > Hi, > > Sorry about being unclear. They all failed in the same way. > > So, these combinations have continuous timeout errors and fail to completely boot: > > Plain 9.0-RC1 > > 9-stable with 1.62 of mfi.c > > 9-stable with www.freebsd.org/~jhb/patches/mfi.patch, pci_alloc_msi > instead of pci_alloc_msix and hw.mfi.msix=0 > > This boots, but gets "mfi0: Cannot allocate interrupt" and there are no /dev/mfi* devices: > > 9-stable with www.freebsd.org/~jhb/patches/mfi.patch and hw.mfi.msix=1 > > This seems to work, but I have not put any load on it yet: > > 9-stable with www.freebsd.org/~jhb/patches/mfi.patch, pci_alloc_msi > instead of pci_alloc_msix and hw.mfi.msix=1 Ok, so MSI interrupts seem to work for you. > > I see you have a new patch, www.freebsd.org/~jhb/patches/mfi_msi.patch. This patch doesn't seem to include the dummy read from your earlier patch, or the one in 1.62 of mfi.c. I assume I need to apply the 1.62 mfi.c diff to by 9-stable sources as well. Is that correct? You can just apply this patch, no need to backport the fix in 1.62 as that fix should not be needed if you are using MSI. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 16:25:20 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE12B106566B; Tue, 15 Nov 2011 16:25:20 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5D1A58FC12; Tue, 15 Nov 2011 16:25:20 +0000 (UTC) Received: from vhoffman.lon.namesco.net (lon.namesco.net [195.7.254.102]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pAFGPI7s020525 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 15 Nov 2011 16:25:19 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EC2926E.6070201@unsane.co.uk> Date: Tue, 15 Nov 2011 16:25:18 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Glen Barber References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> <20111115005607.GA21345@glenbarber.us> In-Reply-To: <20111115005607.GA21345@glenbarber.us> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 16:25:21 -0000 On 15/11/2011 00:56, Glen Barber wrote: > Hi, > > Jeremy Chadwick wrote: > > [...] > >> I don't particularly care what the man page says, I just know what >> works. :-) >> > > If there is something in the man page that is not precise, I personally > am interested in any inconsistencies with reality so they can be fixed. > Even given your detailed explanation, I'm still not 100% clear on what, > if anything, is incorrect in the manual. Hi Glen, Filed as as misc/162587, sorry i did start using category docs and class docs, but I timed out the original web form and forgot to change from defaults while copying and pasting to the new page. Serves me right for using the web form. Regards, Vince > > Regards, > > Glen > From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 16:35:11 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1A9A1065673 for ; Tue, 15 Nov 2011 16:35:11 +0000 (UTC) (envelope-from gjb@freebsd.org) Received: from glenbarber.us (onyx.glenbarber.us [199.48.134.227]) by mx1.freebsd.org (Postfix) with SMTP id A90108FC12 for ; Tue, 15 Nov 2011 16:35:11 +0000 (UTC) Received: (qmail 43373 invoked by uid 0); 15 Nov 2011 11:35:10 -0500 Received: from unknown (HELO glenbarber.us) (75.146.225.65) by 0 with SMTP; 15 Nov 2011 11:35:10 -0500 Date: Tue, 15 Nov 2011 11:35:05 -0500 From: Glen Barber To: Vincent Hoffman Message-ID: <20111115163505.GA24626@glenbarber.us> References: <4EC1AAAE.1030007@unsane.co.uk> <20111115003222.GA82751@icarus.home.lan> <20111115005607.GA21345@glenbarber.us> <4EC2926E.6070201@unsane.co.uk> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="FL5UXtIhxfXey3p5" Content-Disposition: inline In-Reply-To: <4EC2926E.6070201@unsane.co.uk> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: question on netstat statistics. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 16:35:12 -0000 --FL5UXtIhxfXey3p5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Vincent, On Tue, Nov 15, 2011 at 04:25:18PM +0000, Vincent Hoffman wrote: > On 15/11/2011 00:56, Glen Barber wrote: > > Hi, > > > > Jeremy Chadwick wrote: > > > > [...] > > > >> I don't particularly care what the man page says, I just know what > >> works. :-) > >> > > > > If there is something in the man page that is not precise, I personally > > am interested in any inconsistencies with reality so they can be fixed. > > Even given your detailed explanation, I'm still not 100% clear on what, > > if anything, is incorrect in the manual. > Hi Glen, > Filed as as misc/162587, sorry i did start using category > docs and class docs, but I timed out the original web form and forgot to > change from defaults while copying and pasting to the new page. Serves > me right for using the web form. >=20 Thanks for pointing me to the PR. I've assigned it to myself, and will look into this. Glen --=20 Glen Barber | gjb@FreeBSD.org FreeBSD Documentation Project --FL5UXtIhxfXey3p5 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) iQEcBAEBCAAGBQJOwpS4AAoJEFJPDDeguUajP2sH/iB3TbIdc1ixAsgIpCt8DBu9 f2O4nNSs7utvEGFtJe56t8MtzpaNZTOlHDwDlqojqp+o3JAzTQvHVjlsVgubmUzx jA4XajyX2wO6HFNbED2XLxQkexXua83JxGbJ1VzWurUlq7YHYOYwpxQCHlZDSRaa wn1q2rUIewHGjEu91UJ667g2t1DAegaFLYkwg7uhZ2R+WggrxSgXHXzIRiV6QBii y606NwOGXMvi2p99qrS64sGac6ApC6X1EOFSxnXEL4SaZjRDtvUPVERrR4Kf+aRq R6bomdqGXQP6ReHALuQ8IxL6E5Ro6lTcryEgZa1T+RCH/hMmWRwZ4Ifm3Jp9Vbk= =5aUL -----END PGP SIGNATURE----- --FL5UXtIhxfXey3p5-- From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 17:10:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 352D9106566C for ; Tue, 15 Nov 2011 17:10:18 +0000 (UTC) (envelope-from gr@ubuntu64.secure-side.com) Received: from admin.secure-side.com (admin.secure-side.com [213.251.166.100]) by mx1.freebsd.org (Postfix) with SMTP id 934DC8FC1A for ; Tue, 15 Nov 2011 17:10:17 +0000 (UTC) Received: (qmail 52672 invoked from network); 15 Nov 2011 17:10:20 -0000 Received: from localhost (HELO ubuntu64.secure-side.com) (@127.0.0.1) by qmail with SMTP; 15 Nov 2011 17:10:20 -0000 Received: from localhost (localhost [127.0.0.1]) by ubuntu64.secure-side.com (Postfix) with ESMTP id 3FCC246A39; Tue, 15 Nov 2011 18:10:08 +0100 (CET) Received: from ubuntu64.secure-side.com ([127.0.0.1]) by localhost (ubuntu64.secure-side.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CfNBsByakRu4; Tue, 15 Nov 2011 18:10:01 +0100 (CET) Received: from ubuntu64.secure-side.com (ubuntu64.secure-side.com [192.168.1.146]) by ubuntu64.secure-side.com (Postfix) with ESMTP id 4455746A38; Tue, 15 Nov 2011 18:10:01 +0100 (CET) Date: Tue, 15 Nov 2011 18:10:01 +0100 (CET) From: GR To: freebsd-current@freebsd.org Message-ID: In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Originating-IP: [90.32.174.62] X-Mailer: Zimbra 7.1.3_GA_3346 (ZimbraWebClient - GC14 (Linux)/7.1.3_GA_3346) Cc: freebsd-stable@freebsd.org Subject: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 17:10:18 -0000 Hello list, more insights since my last post. Here is a small code to trigger the bug (end of email). When you run it on 9.0-RC1, it gets an alias address instead of the main inet address: % ./get-ip re0 inet: 192.168.2.10 # Main address being 192.168.1.148 On 8.2-RELEASE, all goes well: % ./get-ip re0 inet: PUBLIC_IP4 Is something broken, or a behaviour has changed since 8.2-RELEASE? Best regards, --8<-- #include #include #include #include #include #include #include #include #include #include int main(int argc, char *argv[]) { int fd; struct ifreq ifr; const struct sockaddr_in *sa; fd = socket(AF_INET, SOCK_DGRAM, 0); if (fd < 0) { perror("socket"); exit(-1); } memset(&ifr, 0, sizeof(struct ifreq)); strlcpy(ifr.ifr_name, argv[1], sizeof(ifr.ifr_name)); if (ioctl(fd, SIOCGIFADDR, &ifr) == 0) { sa = (const struct sockaddr_in *) &ifr.ifr_addr; printf("inet: %s\n", inet_ntoa(sa->sin_addr)); } else { perror("ioctl"); exit(-1); } exit(0); } --8<-- -- ^ ___ ___ http://www.GomoR.org/ <-+ | / __ |__/ Senior Security Engineer | | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 20:12:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFB7710656A5 for ; Tue, 15 Nov 2011 20:12:58 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 5F01E8FC17 for ; Tue, 15 Nov 2011 20:12:58 +0000 (UTC) Received: by wwg14 with SMTP id 14so8639855wwg.31 for ; Tue, 15 Nov 2011 12:12:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=OQGB4wRWKlS/TChO4SQuEej3AWm5jjGue2B7+lkekWw=; b=veD5U0JIgb9G0L0jqx0vwX7PKVAWtj2UXS/5BFNrUdlqP/W3cuCFC0LjJ0beBR/2sX vbgSnNZY2du/iDCbm13QQtcgEA54dGfNXlWY4UZur1kacJMDya0fhFO5kK6EKaF0yNR6 KTmGb7pZcxWjcCmgfatwt9twxyZ+FiZscQKU8= MIME-Version: 1.0 Received: by 10.182.108.100 with SMTP id hj4mr6389760obb.34.1321386302374; Tue, 15 Nov 2011 11:45:02 -0800 (PST) Received: by 10.182.53.9 with HTTP; Tue, 15 Nov 2011 11:45:02 -0800 (PST) Date: Tue, 15 Nov 2011 11:45:02 -0800 Message-ID: From: Chuck Tuffli To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 20:12:58 -0000 Is it possible to do a buildkernel of 9-stable (r227536) on a stock 8.2 system? Most of it seems to work, but the linker fails towards the end with ... MAKE=make sh /usr/home/ctuffli/dev/releng_9/src/sys/conf/newvers.sh GENERIC /usr/local/bin/svnversion cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -nostdinc -I. -I/usr/home/ctuffli/dev/releng_9/src/sys -I/usr/home/ctuffli/dev/releng_9/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-omit-frame-pointer -mno-sse -mcmodel=kernel -mno-red-zone -mno-mmx -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector -Werror vers.c linking kernel.debug ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax error *** Error code 1 Stop in /usr/home/ctuffli/dev/releng_9/obj/usr/home/ctuffli/dev/releng_9/src/sys/GENERIC. *** Error code 1 Stop in /usr/home/ctuffli/dev/releng_9/src. *** Error code 1 Stop in /usr/home/ctuffli/dev/releng_9/src. From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 22:14:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79A891065675; Tue, 15 Nov 2011 22:14:56 +0000 (UTC) (envelope-from kp@sigsegv.be) Received: from mercury.codepro.be (mercury.codepro.be [IPv6:2001:4b98:dc0:51:216:3eff:feb7:3147]) by mx1.freebsd.org (Postfix) with ESMTP id 1299C8FC17; Tue, 15 Nov 2011 22:14:56 +0000 (UTC) Received: from adrastea.jupiter.sigsegv.be (adrastea.jupiter.sigsegv.be [IPv6:2001:6f8:1498:1::3]) by mercury.codepro.be (Postfix) with ESMTP id BD888361; Tue, 15 Nov 2011 23:14:53 +0100 (CET) Received: from thebe.jupiter.sigsegv.be (thebe.jupiter.sigsegv.be [172.16.1.5]) by adrastea.jupiter.sigsegv.be (Postfix) with ESMTP id 69F61266B; Tue, 15 Nov 2011 23:14:53 +0100 (CET) Received: by thebe.jupiter.sigsegv.be (Postfix, from userid 1000) id 55F6534774; Tue, 15 Nov 2011 23:14:53 +0100 (CET) Date: Tue, 15 Nov 2011 23:14:53 +0100 From: Kristof Provost To: GR Message-ID: <20111115221453.GO53701@thebe.jupiter.sigsegv.be> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-PGP-Fingerprint: E114 D9EA 909E D469 8F57 17A5 7D15 91C6 9EFA F286 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 22:14:56 -0000 On 2011-11-15 18:10:01 (+0100), GR wrote: > more insights since my last post. Here is a small code to trigger the bug (end of email). > When you run it on 9.0-RC1, it gets an alias address instead of the main inet address: > > % ./get-ip re0 > inet: 192.168.2.10 > # Main address being 192.168.1.148 > > On 8.2-RELEASE, all goes well: > % ./get-ip re0 > inet: PUBLIC_IP4 > > Is something broken, or a behaviour has changed since 8.2-RELEASE? > I think the relevant bit of the code is found in sys/netinet/in.c. If your ioctl doesn't specify an IP address we end up in this bit: TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { iap = ifatoia(ifa); if (iap->ia_addr.sin_family == AF_INET) { if (td != NULL && prison_check_ip4(td->td_ucred, &iap->ia_addr.sin_addr) != 0) continue; ia = iap; break; } } The 'ia' pointer is later used to return the IP address. In other words: it returns the first address on the interface of type IF_INET (which isn't assigned to a jail). I think the order of the addresses is not fixed, or rather it depends on the order in which you assign addresses. In the handling of SIOCSIFADDR new addresses are just appended: TAILQ_INSERT_TAIL(&ifp->if_addrhead, ifa, ifa_link); I don't believe this has changed since 8.0. Is it possible something changed in the network initialisation, leading to the addresses being assigned in a different order? Eagerly awaiting to be told I'm wrong, Kristof From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 22:38:29 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18A12106566B; Tue, 15 Nov 2011 22:38:29 +0000 (UTC) (envelope-from gleb.kurtsou@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6E1B18FC1A; Tue, 15 Nov 2011 22:38:27 +0000 (UTC) Received: by bkbzs8 with SMTP id zs8so10681670bkb.13 for ; Tue, 15 Nov 2011 14:38:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=L3rIWCPylJ7R9pp8VVSBFwG9hjYgAgwt/8Sbmb/2ba4=; b=GSFMKHDQ21YukXfJlnXiORDZiNZ/WOVFMPvwdmutTqaVFxw/jWZtVPUzPxxLjNftkZ Z2lUjMTOeBPmWDNeWqZuRudCP/jLx8+T2BxCl8vlbmd+Va3qGErw1q5M8MAQIB9SyvE2 5iRXgbOnF/dQXSZBjr8ZbairEm0YCjCrMJpDA= Received: by 10.204.141.8 with SMTP id k8mr26417959bku.14.1321395232056; Tue, 15 Nov 2011 14:13:52 -0800 (PST) Received: from localhost ([78.157.92.5]) by mx.google.com with ESMTPS id q6sm37479428bka.6.2011.11.15.14.13.50 (version=SSLv3 cipher=OTHER); Tue, 15 Nov 2011 14:13:50 -0800 (PST) Date: Wed, 16 Nov 2011 00:13:49 +0200 From: Gleb Kurtsou To: GR Message-ID: <20111115221349.GA8701@reks> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 22:38:29 -0000 On (15/11/2011 18:10), GR wrote: > Hello list, > > more insights since my last post. Here is a small code to trigger the bug (end of email). > When you run it on 9.0-RC1, it gets an alias address instead of the main inet address: > > % ./get-ip re0 > inet: 192.168.2.10 > # Main address being 192.168.1.148 > > On 8.2-RELEASE, all goes well: > % ./get-ip re0 > inet: PUBLIC_IP4 > > Is something broken, or a behaviour has changed since 8.2-RELEASE? Your test case looks ok and works as expexted for me on 10-CURRENT, both without aliases and after adding alias to interface. Perhaps it's the way you add aliases (libdnet ?). I've used: ifconfing em0 alias OTHERIP Thanks, Gleb. > > Best regards, > > > --8<-- > #include > #include > #include > #include > #include > #include > #include > #include > #include > #include > > int > main(int argc, char *argv[]) > { > int fd; > struct ifreq ifr; > const struct sockaddr_in *sa; > > fd = socket(AF_INET, SOCK_DGRAM, 0); > if (fd < 0) { > perror("socket"); > exit(-1); > } > > memset(&ifr, 0, sizeof(struct ifreq)); > strlcpy(ifr.ifr_name, argv[1], sizeof(ifr.ifr_name)); > > if (ioctl(fd, SIOCGIFADDR, &ifr) == 0) { > sa = (const struct sockaddr_in *) &ifr.ifr_addr; > printf("inet: %s\n", inet_ntoa(sa->sin_addr)); > } > else { > perror("ioctl"); > exit(-1); > } > > exit(0); > } > --8<-- > > -- > ^ ___ ___ http://www.GomoR.org/ <-+ > | / __ |__/ Senior Security Engineer | > | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | > +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 22:46:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22A4E1065673 for ; Tue, 15 Nov 2011 22:46:48 +0000 (UTC) (envelope-from gr@ubuntu64.secure-side.com) Received: from admin.secure-side.com (admin.secure-side.com [213.251.166.100]) by mx1.freebsd.org (Postfix) with SMTP id 7E2F28FC0C for ; Tue, 15 Nov 2011 22:46:46 +0000 (UTC) Received: (qmail 1109 invoked from network); 15 Nov 2011 22:46:51 -0000 Received: from localhost (HELO ubuntu64.secure-side.com) (@127.0.0.1) by qmail with SMTP; 15 Nov 2011 22:46:51 -0000 Received: from localhost (localhost [127.0.0.1]) by ubuntu64.secure-side.com (Postfix) with ESMTP id 6513E469D9; Tue, 15 Nov 2011 23:35:42 +0100 (CET) Received: from ubuntu64.secure-side.com ([127.0.0.1]) by localhost (ubuntu64.secure-side.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xLr7NCM5+jeH; Tue, 15 Nov 2011 23:35:37 +0100 (CET) Received: from ubuntu64.secure-side.com (ubuntu64.secure-side.com [192.168.1.146]) by ubuntu64.secure-side.com (Postfix) with ESMTP id 9B69D4694D; Tue, 15 Nov 2011 23:35:37 +0100 (CET) Date: Tue, 15 Nov 2011 23:35:37 +0100 (CET) From: GR To: Kristof Provost Message-ID: In-Reply-To: <20111115221453.GO53701@thebe.jupiter.sigsegv.be> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Originating-IP: [82.231.148.242] X-Mailer: Zimbra 7.1.3_GA_3346 (ZimbraWebClient - GC15 ([unknown])/7.1.3_GA_3346) Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 22:46:48 -0000 >From "Kristof Provost" : [..] > The 'ia' pointer is later used to return the IP address. > > In other words: it returns the first address on the interface > of type IF_INET (which isn't assigned to a jail). > > I think the order of the addresses is not fixed, or rather it depends > on > the order in which you assign addresses. In the handling of > SIOCSIFADDR > new addresses are just appended: > > TAILQ_INSERT_TAIL(&ifp->if_addrhead, ifa, ifa_link); > > I don't believe this has changed since 8.0. Is it possible something > changed in the network initialisation, leading to the addresses being > assigned in a different order? > > Eagerly awaiting to be told I'm wrong, > Kristof Thanks Kristof. It appears you are right, the order of assignement is important. I configured my interface using DHCP, and added aliases (all in /etc/rc.conf). But on the 8.2-RELEASE, I used static configuration. So, I switched to static assignement and it changes the behaviour (and "fixes" the "bug"). My guess is that during the time waiting for the DHCP offer, all aliases are already configured on the network interface, and the IP address given by DHCP is added at the end of the tail. Is that a wanted behaviour? I find it dangerous (i.e. not exactly what a user is expecting). Note: my aliases are attributed to jails. Regards, -- ^ ___ ___ http://www.GomoR.org/ <-+ | / __ |__/ Senior Security Engineer | | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ From owner-freebsd-stable@FreeBSD.ORG Tue Nov 15 23:16:27 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 427B4106564A for ; Tue, 15 Nov 2011 23:16:27 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 0195D8FC12 for ; Tue, 15 Nov 2011 23:16:26 +0000 (UTC) Received: from omta11.westchester.pa.mail.comcast.net ([76.96.62.36]) by qmta03.westchester.pa.mail.comcast.net with comcast id xbDs1h0070mv7h053bGTr7; Tue, 15 Nov 2011 23:16:27 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta11.westchester.pa.mail.comcast.net with comcast id xbGQ1h00S1t3BNj3XbGSAi; Tue, 15 Nov 2011 23:16:27 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8807D102C1D; Tue, 15 Nov 2011 15:16:23 -0800 (PST) Date: Tue, 15 Nov 2011 15:16:23 -0800 From: Jeremy Chadwick To: GR Message-ID: <20111115231623.GA5712@icarus.home.lan> References: <20111115221453.GO53701@thebe.jupiter.sigsegv.be> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Kristof Provost , freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Nov 2011 23:16:27 -0000 On Tue, Nov 15, 2011 at 11:35:37PM +0100, GR wrote: > >From "Kristof Provost" : > [..] > > The 'ia' pointer is later used to return the IP address. > > > > In other words: it returns the first address on the interface > > of type IF_INET (which isn't assigned to a jail). > > > > I think the order of the addresses is not fixed, or rather it depends > > on > > the order in which you assign addresses. In the handling of > > SIOCSIFADDR > > new addresses are just appended: > > > > TAILQ_INSERT_TAIL(&ifp->if_addrhead, ifa, ifa_link); > > > > I don't believe this has changed since 8.0. Is it possible something > > changed in the network initialisation, leading to the addresses being > > assigned in a different order? > > > > Eagerly awaiting to be told I'm wrong, > > Kristof > > Thanks Kristof. It appears you are right, the order of assignement is important. > I configured my interface using DHCP, and added aliases (all in /etc/rc.conf). > But on the 8.2-RELEASE, I used static configuration. > > So, I switched to static assignement and it changes the behaviour (and "fixes" the "bug"). > My guess is that during the time waiting for the DHCP offer, all aliases are already configured on the network interface, and the IP address given by DHCP is added at the end of the tail. > > Is that a wanted behaviour? I find it dangerous (i.e. not exactly what a user is expecting). > > Note: my aliases are attributed to jails. I would recommend adding synchronous_dhclient="yes" to /etc/rc.conf. This will cause dhclient (the DHCP client) to wait until it gets an answer + IP back from the DHCP server before continuing with the rc.d scripts. The default is "no". -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 00:29:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF5001065674 for ; Wed, 16 Nov 2011 00:29:47 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from glenbarber.us (onyx.glenbarber.us [199.48.134.227]) by mx1.freebsd.org (Postfix) with SMTP id A35AB8FC0A for ; Wed, 16 Nov 2011 00:29:47 +0000 (UTC) Received: (qmail 51270 invoked by uid 0); 15 Nov 2011 19:29:46 -0500 Received: from unknown (HELO glenbarber.us) (76.124.49.145) by 0 with SMTP; 15 Nov 2011 19:29:46 -0500 Date: Tue, 15 Nov 2011 19:29:44 -0500 From: Glen Barber To: Chuck Tuffli Message-ID: <20111116002944.GC24626@glenbarber.us> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 00:29:48 -0000 Hi, On Tue, Nov 15, 2011 at 11:45:02AM -0800, Chuck Tuffli wrote: > Is it possible to do a buildkernel of 9-stable (r227536) on a stock > 8.2 system? Most of it seems to work, but the linker fails towards the > end with > > ... > MAKE=make sh /usr/home/ctuffli/dev/releng_9/src/sys/conf/newvers.sh GENERIC > /usr/local/bin/svnversion > cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g > -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef > -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs > -fdiagnostics-show-option -nostdinc -I. > -I/usr/home/ctuffli/dev/releng_9/src/sys > -I/usr/home/ctuffli/dev/releng_9/src/sys/contrib/altq -D_KERNEL > -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common > -finline-limit=8000 --param inline-unit-growth=100 --param > large-function-growth=1000 -fno-omit-frame-pointer -mno-sse > -mcmodel=kernel -mno-red-zone -mno-mmx -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector > -Werror vers.c > linking kernel.debug > ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax error > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/obj/usr/home/ctuffli/dev/releng_9/src/sys/GENERIC. > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/src. > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/src. You'll need to do 'buildworld' first. Regards, Glen -- Glen Barber From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 01:32:14 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3FA4106566B for ; Wed, 16 Nov 2011 01:32:14 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id BAED78FC18 for ; Wed, 16 Nov 2011 01:32:14 +0000 (UTC) Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51]) by qmta06.emeryville.ca.mail.comcast.net with comcast id xcz51h00216AWCUA6dY7fz; Wed, 16 Nov 2011 01:32:07 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta06.emeryville.ca.mail.comcast.net with comcast id xdXp1h0081t3BNj8SdXtP6; Wed, 16 Nov 2011 01:31:53 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id B2777102C1D; Tue, 15 Nov 2011 17:32:09 -0800 (PST) Date: Tue, 15 Nov 2011 17:32:09 -0800 From: Jeremy Chadwick To: Glen Barber Message-ID: <20111116013209.GA8414@icarus.home.lan> References: <20111116002944.GC24626@glenbarber.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111116002944.GC24626@glenbarber.us> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Chuck Tuffli Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 01:32:15 -0000 On Tue, Nov 15, 2011 at 07:29:44PM -0500, Glen Barber wrote: > Hi, > > On Tue, Nov 15, 2011 at 11:45:02AM -0800, Chuck Tuffli wrote: > > Is it possible to do a buildkernel of 9-stable (r227536) on a stock > > 8.2 system? Most of it seems to work, but the linker fails towards the > > end with > > > > ... > > MAKE=make sh /usr/home/ctuffli/dev/releng_9/src/sys/conf/newvers.sh GENERIC > > /usr/local/bin/svnversion > > cc -c -O2 -frename-registers -pipe -fno-strict-aliasing -std=c99 -g > > -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes > > -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef > > -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs > > -fdiagnostics-show-option -nostdinc -I. > > -I/usr/home/ctuffli/dev/releng_9/src/sys > > -I/usr/home/ctuffli/dev/releng_9/src/sys/contrib/altq -D_KERNEL > > -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common > > -finline-limit=8000 --param inline-unit-growth=100 --param > > large-function-growth=1000 -fno-omit-frame-pointer -mno-sse > > -mcmodel=kernel -mno-red-zone -mno-mmx -msoft-float > > -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector > > -Werror vers.c > > linking kernel.debug > > ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax error > > *** Error code 1 > > > > Stop in /usr/home/ctuffli/dev/releng_9/obj/usr/home/ctuffli/dev/releng_9/src/sys/GENERIC. > > *** Error code 1 > > > > Stop in /usr/home/ctuffli/dev/releng_9/src. > > *** Error code 1 > > > > Stop in /usr/home/ctuffli/dev/releng_9/src. > > You'll need to do 'buildworld' first. Not to mention, one should always do buildworld first. The absolute correct procedure is outlined in /usr/src/Makefile: # 1. `cd /usr/src' (or to the directory containing your source tree). # 2. `make buildworld' # 3. `make buildkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC). # 4. `make installkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC). # [steps 3. & 4. can be combined by using the "kernel" target] # 5. `reboot' (in single user mode: boot -s from the loader prompt). # 6. `mergemaster -p' # 7. `make installworld' # 8. `make delete-old' # 9. `mergemaster' (you may wish to use -i, along with -U or -F). # 10. `reboot' # 11. `make delete-old-libs' (in case no 3rd party program uses them anymore) People (not you Glen :-) ) need to realise that doing buildworld actually builds the necessary "build toolchain" used for the buildkernel portion (the results are in /usr/obj). Failure to do that results in the buildkernel bits using the already-installed-on-the-system compiler/build toolchain (e.g. the stuff in /usr), which may not be compatible with the version of the kernel source you're trying to build. Hope this makes sense to readers. Two "extra steps" which I do: - Before step 1, I do rm -fr /usr/obj/* - Before step 7, I do rm -fr /usr/share/man/* Step 1 is a precaution, and step 7 ensures that no old/leftover compressed man pages (from old versions of software) are retained. If needed/asked, Doug Barton can talk a bit more about the latter. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 04:48:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E10A0106564A for ; Wed, 16 Nov 2011 04:48:49 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id A36488FC08 for ; Wed, 16 Nov 2011 04:48:49 +0000 (UTC) Received: by ggnk3 with SMTP id k3so11819390ggn.13 for ; Tue, 15 Nov 2011 20:48:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=yQauINvZoaEBl68JNdHKg03PiCy9TXl+v8GTAdIDj8E=; b=R8QRtabNF4vIWz1Iqf26L2Yf7P6yhGHGfU6QWfKcgyYHaBHIXw+XvyIkmSmTrEY9nC sGj6Ja1+GxbDZRWDVrIZn7cZSFSLO3RGHwDyvSwFdyiRSQ1kjs6uFMeqGusGZQyXvgpx Wv0Ud4zmU616CDrbOkTXnynlH24wO38EoRP5o= MIME-Version: 1.0 Received: by 10.182.72.100 with SMTP id c4mr5085962obv.39.1321418929088; Tue, 15 Nov 2011 20:48:49 -0800 (PST) Received: by 10.182.61.193 with HTTP; Tue, 15 Nov 2011 20:48:49 -0800 (PST) In-Reply-To: References: Date: Wed, 16 Nov 2011 07:48:49 +0300 Message-ID: From: Sergey Kandaurov To: Chuck Tuffli Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 04:48:50 -0000 On 15 November 2011 23:45, Chuck Tuffli wrote: > Is it possible to do a buildkernel of 9-stable (r227536) on a stock > 8.2 system? Most of it seems to work, but the linker fails towards the > end with > > ... > MAKE=3Dmake sh /usr/home/ctuffli/dev/releng_9/src/sys/conf/newvers.sh GEN= ERIC > /usr/local/bin/svnversion > cc -c -O2 -frename-registers -pipe -fno-strict-aliasing =A0-std=3Dc99 -g > -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes > -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual =A0-Wundef > -Wno-pointer-sign -fformat-extensions =A0-Wmissing-include-dirs > -fdiagnostics-show-option -nostdinc =A0-I. > -I/usr/home/ctuffli/dev/releng_9/src/sys > -I/usr/home/ctuffli/dev/releng_9/src/sys/contrib/altq -D_KERNEL > -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common > -finline-limit=3D8000 --param inline-unit-growth=3D100 --param > large-function-growth=3D1000 =A0-fno-omit-frame-pointer -mno-sse > -mcmodel=3Dkernel -mno-red-zone -mno-mmx -msoft-float > -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector > -Werror =A0vers.c > linking kernel.debug > ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax e= rror > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/obj/usr/home/ctuffli/dev/releng_9/= src/sys/GENERIC. > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/src. > *** Error code 1 > > Stop in /usr/home/ctuffli/dev/releng_9/src. IIRC 8.x has sufficiently old binutils (ld is part of them) that doesn't understand opcodes and suchlike constructs used in 9.x based on top of newer biinutils= . And you are trying to build 9.x using binutils from 8.x. --=20 wbr, pluknet From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 06:43:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB87B106566C; Wed, 16 Nov 2011 06:43:48 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) by mx1.freebsd.org (Postfix) with ESMTP id 42FFC8FC1D; Wed, 16 Nov 2011 06:43:48 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 493B525D3A93; Wed, 16 Nov 2011 06:43:47 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 8440EBD4FEC; Wed, 16 Nov 2011 06:43:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id x37B3kc8yD5o; Wed, 16 Nov 2011 06:43:45 +0000 (UTC) Received: from nv.sbone.de (nv.sbone.de [IPv6:fde9:577b:c1a9:31::2013:138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 35001BD4FEB; Wed, 16 Nov 2011 06:43:45 +0000 (UTC) Date: Wed, 16 Nov 2011 06:43:44 +0000 (UTC) From: "Bjoern A. Zeeb" To: freebsd-stable@freebsd.org Message-ID: X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: mav@freebsd.org Subject: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 06:43:48 -0000 Hey, we have seen this or a very similar panic for about 1 year now once in a while and I think I reported it before; this is FreeBSD as guest on vmware. Seems it was a double panic this time. Could someone please see what's going on there? It was on 8.x-STABLE in the past and this is 8.2-RELEASE-p4. Thanks /bz acd0: WARNING - READ_TOC taskqueue timeout - completing request directly Fatal trap 12: page fault while in kernel mode cpuid = 4; apic id = 04 fault virtual address = 0x1f4 fault code = supervisor read, page not present instruction pointer = 0x20:0xc08a1e9f stack pointer = 0x28:0xe6ad5b9c Fatal trap 12: page fault while in kernel mode frame pointer = 0x28:0xe6ad5bb4 cpuid = 2; code segment = base 0x0, limit 0xfffff, type 0x1bapic id = 02 = DPL 0, pres 1, def32 1, gran 1 fault virtual address = 0x1f4 processor eflags = fault code = supervisor read, page not presentinterrupt enabled, instruction pointer = 0x20:0xc08a1e9fresume, stack pointer = 0x28:0xe8e9e808IOPL = 0 frame pointer = 0x28:0xe8e9e820 current process = code segment = base 0x0, limit 0xfffff, type 0x1b12 (swi6: task queue) = DPL 0, pres 1, def32 1, gran 1 trap number = 12 processor eflags = interrupt enabled, panic: page faultresume, cpuid = 4IOPL = 0 current process = KDB: stack backtrace:25162 (bsnmpd) trap number = 12#0 0xc08e0d07 at kdb_backtrace+0x47 #1 0xc08b1dc7 at panic+0x117 #2 0xc0be4b53 at trap_fatal+0x323 #3 0xc0be4dd0 at trap_pfault+0x270 #4 0xc0be5315 at trap+0x465 #5 0xc0bcbecc at calltrap+0x6 #6 0xc08b0d86 at _sema_post+0x46 #7 0xc056fa47 at ata_completed+0x727 #8 0xc08eb97a at taskqueue_run_locked+0xca #9 0xc08ebc8a at taskqueue_run+0xaa #10 0xc08ebd53 at taskqueue_swi_run+0x13 #11 0xc088903b at intr_event_execute_handlers+0x13b #12 0xc088a75b at ithread_loop+0x6b #13 0xc0886d51 at fork_exit+0x91 #14 0xc0bcbf44 at fork_trampoline+0x8 Uptime: 5d20h1m56s (gdb) l *ata_completed+0x727 489 (request->callback)(request); 490 else 491 sema_post(&request->done); 492 493 /* only call ata_start if channel is present */ 494 if (ch) 495 ata_start(ch->dev); 496 } 497 498 void -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family. From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 07:51:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C29C106564A; Wed, 16 Nov 2011 07:51:45 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from gilb.zs64.net (gilb.zs64.net [IPv6:2001:470:1f0b:105e::1ea]) by mx1.freebsd.org (Postfix) with ESMTP id 252328FC13; Wed, 16 Nov 2011 07:51:45 +0000 (UTC) Received: by gilb.zs64.net (Postfix, from stb@lassitu.de) id A07A2CB5C8; Wed, 16 Nov 2011 08:51:43 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Stefan Bethke In-Reply-To: Date: Wed, 16 Nov 2011 08:51:42 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: GR X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-current Current , "freebsd-stable@freebsd.org List" Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 07:51:45 -0000 Am 15.11.2011 um 23:35 schrieb GR: > So, I switched to static assignement and it changes the behaviour (and = "fixes" the "bug"). > My guess is that during the time waiting for the DHCP offer, all = aliases are already configured on the network interface, and the IP = address given by DHCP is added at the end of the tail. >=20 > Is that a wanted behaviour? I find it dangerous (i.e. not exactly what = a user is expecting). A bit of background, as best I understand it and remember from Stevens: Interfaces in BSD do not have a notion of "primary" and "additional" = addresses; interfaces just have any number of addresses associated with = them. There's no inherent ordering in this list (except for how the = current implementation seems to keep them in the order they were = configured). To be able to associate proper routes with interface addresses, the = recommendation for multiple IPv4 addresses on an Ethernet interface is = to have one of them have the proper netmask for the network, and = configure the remainder with a netmask of 255.255.255.255. But that's = solely for the benefit of the routing table; the interface itself = doesn't really care. Reading the rc.conf man page could give you the impression that there = are primary and alias addresses, but the networking code doesn't really = work like that. The new ipv4_addrs_ syntax exposes the = actual behavior in a more direct way. Jeremy gave you a hint on how to fix your immediate problem, but the = real answer is that the program needs to be fixed that makes assumptions = about meaning attached to the first configured IPv4 address. HTH, Stefan --=20 Stefan Bethke Fon +49 151 14070811 From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 08:01:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D895106566B for ; Wed, 16 Nov 2011 08:01:03 +0000 (UTC) (envelope-from gr@ubuntu64.secure-side.com) Received: from admin.secure-side.com (admin.secure-side.com [213.251.166.100]) by mx1.freebsd.org (Postfix) with SMTP id B351E8FC14 for ; Wed, 16 Nov 2011 08:01:02 +0000 (UTC) Received: (qmail 23863 invoked from network); 16 Nov 2011 08:01:06 -0000 Received: from localhost (HELO ubuntu64.secure-side.com) (@127.0.0.1) by qmail with SMTP; 16 Nov 2011 08:01:06 -0000 Received: from localhost (localhost [127.0.0.1]) by ubuntu64.secure-side.com (Postfix) with ESMTP id C17D1469D9; Wed, 16 Nov 2011 08:49:56 +0100 (CET) Received: from ubuntu64.secure-side.com ([127.0.0.1]) by localhost (ubuntu64.secure-side.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FpVsDOTFfLqC; Wed, 16 Nov 2011 08:49:52 +0100 (CET) Received: from ubuntu64.secure-side.com (ubuntu64.secure-side.com [192.168.1.146]) by ubuntu64.secure-side.com (Postfix) with ESMTP id E8FC14694D; Wed, 16 Nov 2011 08:49:51 +0100 (CET) Date: Wed, 16 Nov 2011 08:49:51 +0100 (CET) From: GomoR To: Jeremy Chadwick Message-ID: <5343004d-ea79-4aae-8c5d-d1161719b2f7@ubuntu64> In-Reply-To: <20111115231623.GA5712@icarus.home.lan> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 X-Originating-IP: [82.231.148.242] X-Mailer: Zimbra 7.1.3_GA_3346 (ZimbraWebClient - GC15 ([unknown])/7.1.3_GA_3346) Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: SIOCGIFADDR broken on 9.0-RC1? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 08:01:03 -0000 > From: "Jeremy Chadwick" > > I would recommend adding synchronous_dhclient="yes" to /etc/rc.conf. > This will cause dhclient (the DHCP client) to wait until it gets an > answer + IP back from the DHCP server before continuing with the rc.d > scripts. The default is "no". Awesome, thank you. I should have searched for dhclient options in /etc/defaults/rc.conf ;) -- ^ ___ ___ http://www.GomoR.org/ <-+ | / __ |__/ Senior Security Engineer | | \__/ | \ ---[ zsh$ alias psed='perl -pe ' ]--- | +--> Net::Frame <=> http://search.cpan.org/~gomor/ <---+ From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 09:26:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFBB21065670 for ; Wed, 16 Nov 2011 09:26:09 +0000 (UTC) (envelope-from milu@dat.pl) Received: from jab.dat.pl (dat.pl [80.51.155.34]) by mx1.freebsd.org (Postfix) with ESMTP id 9C5AA8FC18 for ; Wed, 16 Nov 2011 09:26:09 +0000 (UTC) Received: from jab.dat.pl (jsrv.dat.pl [127.0.0.1]) by jab.dat.pl (Postfix) with ESMTP id 6B906C0 for ; Wed, 16 Nov 2011 10:06:20 +0100 (CET) X-Virus-Scanned: amavisd-new at dat.pl Received: from jab.dat.pl ([127.0.0.1]) by jab.dat.pl (jab.dat.pl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id tA5IhTLKbcAC for ; Wed, 16 Nov 2011 10:06:17 +0100 (CET) Received: from snifi.localnet (unknown [212.69.68.42]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by jab.dat.pl (Postfix) with ESMTPSA id B6DA737 for ; Wed, 16 Nov 2011 10:06:17 +0100 (CET) From: Maciej Milewski To: freebsd-stable@freebsd.org Date: Wed, 16 Nov 2011 10:06:35 +0100 Message-ID: <2336418.YxNmZBCRYx@snifi> User-Agent: KMail/4.7.3 (Linux/3.1.1-1-ARCH; KDE/4.7.3; x86_64; ; ) In-Reply-To: <20111116013209.GA8414@icarus.home.lan> References: <20111116002944.GC24626@glenbarber.us> <20111116013209.GA8414@icarus.home.lan> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 09:26:10 -0000 Dnia wtorek 15 listopad 2011 17:32:09 Jeremy Chadwick pisze: > Not to mention, one should always do buildworld first. The absolute > correct procedure is outlined in /usr/src/Makefile: And in /usr/src/UPDATING which should be read before updating as it may have some important informations. The same apply to /usr/ports/UPDATING. less +/cross-install /usr/src/UPDATING -- Maciej Milewski From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 09:34:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 599E3106564A; Wed, 16 Nov 2011 09:34:02 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (cl-327.ede-01.nl.sixxs.net [IPv6:2001:7b8:2ff:146::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1CB368FC17; Wed, 16 Nov 2011 09:34:02 +0000 (UTC) Received: from [IPv6:2001:7b8:3a7:0:8542:fbb2:87b1:8f86] (unknown [IPv6:2001:7b8:3a7:0:8542:fbb2:87b1:8f86]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id BA9805C59; Wed, 16 Nov 2011 10:34:00 +0100 (CET) Message-ID: <4EC3838A.9090003@FreeBSD.org> Date: Wed, 16 Nov 2011 10:34:02 +0100 From: Dimitry Andric Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Glen Barber References: <20111116002944.GC24626@glenbarber.us> In-Reply-To: <20111116002944.GC24626@glenbarber.us> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Chuck Tuffli Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 09:34:02 -0000 On 2011-11-16 01:29, Glen Barber wrote: > On Tue, Nov 15, 2011 at 11:45:02AM -0800, Chuck Tuffli wrote: ... >> ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax error >> *** Error code 1 > You'll need to do 'buildworld' first. Actually, doing "make kernel-toolchain" is enough. This builds just the required tools, e.g. binutils, gcc and so on. From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 10:40:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 615C8106566C for ; Wed, 16 Nov 2011 10:40:49 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id E05948FC0A for ; Wed, 16 Nov 2011 10:40:48 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RQcv7-0007ni-TD for freebsd-stable@freebsd.org; Wed, 16 Nov 2011 11:40:45 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Nov 2011 11:40:45 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Nov 2011 11:40:45 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Wed, 16 Nov 2011 11:40:30 +0100 Lines: 110 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig6D0B63AE6ED5528827DC7506" X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111004 Thunderbird/7.0.1 In-Reply-To: X-Enigmail-Version: 1.1.2 Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 10:40:49 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig6D0B63AE6ED5528827DC7506 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 16/11/2011 07:43, Bjoern A. Zeeb wrote: > Hey, >=20 > we have seen this or a very similar panic for about 1 year now once in > a while and I think I reported it before; this is FreeBSD as guest on Yes, IIRC I've also reported it before; it crashes randomly, when the machine is not doing anything with the cdrom. As a workaround, I now remove the cdrom device from vmware instances. > vmware. Seems it was a double panic this time. Could someone please= > see what's going on there? It was on 8.x-STABLE in the past and this= > is 8.2-RELEASE-p4. >=20 > Thanks > /bz >=20 > acd0: WARNING - READ_TOC taskqueue timeout - completing request directl= y >=20 >=20 > Fatal trap 12: page fault while in kernel mode > cpuid =3D 4; apic id =3D 04 > fault virtual address =3D 0x1f4 > fault code =3D supervisor read, page not present > instruction pointer =3D 0x20:0xc08a1e9f >=20 > stack pointer =3D 0x28:0xe6ad5b9c > Fatal trap 12: page fault while in kernel mode > frame pointer =3D 0x28:0xe6ad5bb4 > cpuid =3D 2; > code segment =3D base 0x0, limit 0xfffff, type 0x1bapic id =3D= 02 > =3D DPL 0, pres 1, def32 1, gran 1 > fault virtual address =3D 0x1f4 > processor eflags =3D > fault code =3D supervisor read, page not presentinterrupt > enabled, > instruction pointer =3D 0x20:0xc08a1e9fresume, > stack pointer =3D 0x28:0xe8e9e808IOPL =3D 0 > frame pointer =3D 0x28:0xe8e9e820 > current process =3D > code segment =3D base 0x0, limit 0xfffff, type 0x1b12 (swi6:= > task queue) > =3D DPL 0, pres 1, def32 1, gran 1 > trap number =3D 12 > processor eflags =3D interrupt enabled, > panic: page faultresume, > cpuid =3D 4IOPL =3D 0 > current process =3D > KDB: stack backtrace:25162 (bsnmpd) >=20 > trap number =3D 12#0 0xc08e0d07 at kdb_backtrace+0x47 >=20 > #1 0xc08b1dc7 at panic+0x117 > #2 0xc0be4b53 at trap_fatal+0x323 > #3 0xc0be4dd0 at trap_pfault+0x270 > #4 0xc0be5315 at trap+0x465 > #5 0xc0bcbecc at calltrap+0x6 > #6 0xc08b0d86 at _sema_post+0x46 > #7 0xc056fa47 at ata_completed+0x727 > #8 0xc08eb97a at taskqueue_run_locked+0xca > #9 0xc08ebc8a at taskqueue_run+0xaa > #10 0xc08ebd53 at taskqueue_swi_run+0x13 > #11 0xc088903b at intr_event_execute_handlers+0x13b > #12 0xc088a75b at ithread_loop+0x6b > #13 0xc0886d51 at fork_exit+0x91 > #14 0xc0bcbf44 at fork_trampoline+0x8 > Uptime: 5d20h1m56s >=20 >=20 > (gdb) l *ata_completed+0x727 > 489 (request->callback)(request); > 490 else > 491 sema_post(&request->done); > 492 > 493 /* only call ata_start if channel is present */ > 494 if (ch) > 495 ata_start(ch->dev); > 496 } > 497 > 498 void >=20 >=20 --------------enig6D0B63AE6ED5528827DC7506 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7Dkx8ACgkQldnAQVacBcg48QCg3jLvw5kLVODybNhYw70UiewR lSUAnRT021vdutgA5/MGnLLbZPs91vI6 =AaC6 -----END PGP SIGNATURE----- --------------enig6D0B63AE6ED5528827DC7506-- From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 10:43:55 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B41A0106566B; Wed, 16 Nov 2011 10:43:55 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 49C118FC08; Wed, 16 Nov 2011 10:43:55 +0000 (UTC) Received: from vhoffman.lon.namesco.net (lon.namesco.net [195.7.254.102]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pAGAhq3X061692 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 16 Nov 2011 10:43:53 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EC393E8.9070500@unsane.co.uk> Date: Wed, 16 Nov 2011 10:43:52 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: John Baldwin References: <4EA9E0C3.5080306@unsane.co.uk> <201111090939.14177.jhb@freebsd.org> <4EBBAE90.8010101@unsane.co.uk> <201111141442.16722.jhb@freebsd.org> In-Reply-To: <201111141442.16722.jhb@freebsd.org> X-Enigmail-Version: 1.3.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Jan Mikkelsen , freebsd-stable@freebsd.org Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 10:43:55 -0000 On 14/11/2011 19:42, John Baldwin wrote: > On Thursday, November 10, 2011 5:59:28 am Vincent Hoffman wrote: >> Well the dell has been up for about 19 hours now using MSI, I ran >> bonnie++ a few times on it and have now stuck it in a permanent loop >> (will look in from time to time.) Are there any tests you'd like >> run/info you'd like? > Actually, can you please test www.freebsd.org/~jhb/patches/mfi_msi.patch? > You will have to set the hw.mfi.msi=1 tunable to enable MSI support. This > is a commit candidate if it works. Thanks. > Applied and running with bonnie++ overnight. All good for me at least. Vince From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 11:03:34 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 304631065673; Wed, 16 Nov 2011 11:03:34 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 870898FC0A; Wed, 16 Nov 2011 11:03:33 +0000 (UTC) Received: by eyd10 with SMTP id 10so375678eyd.13 for ; Wed, 16 Nov 2011 03:03:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=/c1g2+Ksifb5mj0Z8o7m88GBRKjlW3GuyMZ7nezVAuE=; b=vFkt/zhz6QatZT2NKtG0X/MesVFfK8wsDONdVUt9HwyGnSk2qoWzTr3tXHIiDNXfDI FHaBJQ29swvfis8XHL042n7XkJ4DCLm8rpFz9re9jTES+bCDjdh48Crsi+mvbMDEmI8X 2va1cguIkF4mpQaZmPQV4Aq/3wkuWbyDdu9rc= Received: by 10.213.3.77 with SMTP id 13mr529565ebm.148.1321439965572; Wed, 16 Nov 2011 02:39:25 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id d6sm78378352eec.10.2011.11.16.02.39.23 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 16 Nov 2011 02:39:24 -0800 (PST) Sender: Alexander Motin Message-ID: <4EC392DA.2030302@FreeBSD.org> Date: Wed, 16 Nov 2011 12:39:22 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 11:03:34 -0000 Hi. On 11/16/11 08:43, Bjoern A. Zeeb wrote: > we have seen this or a very similar panic for about 1 year now once in > a while and I think I reported it before; this is FreeBSD as guest on > vmware. Seems it was a double panic this time. Could someone please > see what's going on there? It was on 8.x-STABLE in the past and this > is 8.2-RELEASE-p4. The part of code reporting "completing request directly" is IMHO broken by design. It returns request completion before request will actually be completed by lower levels without any knowledge of what's going on there. There is kind of protection against double request completion, but it looks like not always working. May be because that part of code is not locked and nothing prevents that semaphore timeout and normal request timeout/completion to happen simultaneously. It is surprising to see even two traps same time, not sure what synchronized them so precisely. Simple removing that semaphore timeout is not an option, because it will cause deadlock when this wait happen within taskqueue thread that is used to handle requests completion and abort that wait. Avoid waiting inside taskqueue is also impossible without major rewrite. That's why ATA_CAM drops that code completely. -- Alexander Motin From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 14:14:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD7CB1065675; Wed, 16 Nov 2011 14:14:50 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) by mx1.freebsd.org (Postfix) with ESMTP id 29BC68FC16; Wed, 16 Nov 2011 14:14:50 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 30A0125D38A5; Wed, 16 Nov 2011 14:14:49 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 40E1ABD5045; Wed, 16 Nov 2011 14:14:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id D66Yvyk9N1iH; Wed, 16 Nov 2011 14:14:47 +0000 (UTC) Received: from nv.sbone.de (nv.sbone.de [IPv6:fde9:577b:c1a9:31::2013:138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 30246BD5044; Wed, 16 Nov 2011 14:14:47 +0000 (UTC) Date: Wed, 16 Nov 2011 14:14:46 +0000 (UTC) From: "Bjoern A. Zeeb" To: Alexander Motin In-Reply-To: <4EC392DA.2030302@FreeBSD.org> Message-ID: References: <4EC392DA.2030302@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-stable@freebsd.org Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 14:14:50 -0000 On Wed, 16 Nov 2011, Alexander Motin wrote: > Hi. > > On 11/16/11 08:43, Bjoern A. Zeeb wrote: >> we have seen this or a very similar panic for about 1 year now once in >> a while and I think I reported it before; this is FreeBSD as guest on >> vmware. Seems it was a double panic this time. Could someone please >> see what's going on there? It was on 8.x-STABLE in the past and this >> is 8.2-RELEASE-p4. > > The part of code reporting "completing request directly" is IMHO broken > by design. It returns request completion before request will actually be > completed by lower levels without any knowledge of what's going on > there. There is kind of protection against double request completion, > but it looks like not always working. May be because that part of code > is not locked and nothing prevents that semaphore timeout and normal > request timeout/completion to happen simultaneously. It is surprising to > see even two traps same time, not sure what synchronized them so precisely. > > Simple removing that semaphore timeout is not an option, because it will > cause deadlock when this wait happen within taskqueue thread that is > used to handle requests completion and abort that wait. Avoid waiting > inside taskqueue is also impossible without major rewrite. That's why > ATA_CAM drops that code completely. So the bottom line of what you are saying is: 1) it's hard to fix right in 8 2) it's not an issue in 9 anymore at all? -- Bjoern A. Zeeb You have to have visions! Stop bit received. Insert coin for new address family. From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 14:33:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B132A1065674; Wed, 16 Nov 2011 14:33:04 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1A33A8FC1E; Wed, 16 Nov 2011 14:33:03 +0000 (UTC) Received: by eyd10 with SMTP id 10so753931eyd.13 for ; Wed, 16 Nov 2011 06:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=VLA16HeM6UCLIQnpAXHKYC8KFjqsJU30xLvMVBqbMTs=; b=ueWPuZavmLZN5y71pIHVezhzeu4LmumDNAu8GGsT3WVnFpssaj9yd/EcYR4uBQwqBa a0E0SO/ITBnXhIsYJIjUiSMVxmtBhL3HgablPXAjbNLuslRaNfc7m7Wgypu1GvWQqL/i QgjadtGGjnffm0OZrTF289ty8HZpMz71EB7ms= Received: by 10.213.4.140 with SMTP id 12mr1233430ebr.106.1321453982978; Wed, 16 Nov 2011 06:33:02 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id f36sm80184393eef.4.2011.11.16.06.33.01 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 16 Nov 2011 06:33:01 -0800 (PST) Sender: Alexander Motin Message-ID: <4EC3C99C.8020203@FreeBSD.org> Date: Wed, 16 Nov 2011 16:33:00 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4EC392DA.2030302@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 14:33:04 -0000 On 11/16/11 16:14, Bjoern A. Zeeb wrote: > On Wed, 16 Nov 2011, Alexander Motin wrote: > >> Hi. >> >> On 11/16/11 08:43, Bjoern A. Zeeb wrote: >>> we have seen this or a very similar panic for about 1 year now once in >>> a while and I think I reported it before; this is FreeBSD as guest on >>> vmware. Seems it was a double panic this time. Could someone please >>> see what's going on there? It was on 8.x-STABLE in the past and this >>> is 8.2-RELEASE-p4. >> >> The part of code reporting "completing request directly" is IMHO broken >> by design. It returns request completion before request will actually be >> completed by lower levels without any knowledge of what's going on >> there. There is kind of protection against double request completion, >> but it looks like not always working. May be because that part of code >> is not locked and nothing prevents that semaphore timeout and normal >> request timeout/completion to happen simultaneously. It is surprising to >> see even two traps same time, not sure what synchronized them so >> precisely. >> >> Simple removing that semaphore timeout is not an option, because it will >> cause deadlock when this wait happen within taskqueue thread that is >> used to handle requests completion and abort that wait. Avoid waiting >> inside taskqueue is also impossible without major rewrite. That's why >> ATA_CAM drops that code completely. > > So the bottom line of what you are saying is: > 1) it's hard to fix right in 8 > 2) it's not an issue in 9 anymore at all? Right. -- Alexander Motin From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 15:04:10 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6CBD2106564A; Wed, 16 Nov 2011 15:04:10 +0000 (UTC) (envelope-from joel@vnode.se) Received: from mail.vnode.se (mail.vnode.se [62.119.52.80]) by mx1.freebsd.org (Postfix) with ESMTP id 1F7348FC14; Wed, 16 Nov 2011 15:04:09 +0000 (UTC) Received: from mail.vnode.se (localhost [127.0.0.1]) by mail.vnode.se (Postfix) with ESMTP id 03B8BE3F07A; Wed, 16 Nov 2011 15:45:34 +0100 (CET) X-Virus-Scanned: amavisd-new at vnode.se Received: from mail.vnode.se ([127.0.0.1]) by mail.vnode.se (mail.vnode.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kblE1TAV6jf2; Wed, 16 Nov 2011 15:45:28 +0100 (CET) Received: from goofy01.vnodelab.local (unknown [212.247.52.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.vnode.se (Postfix) with ESMTPSA id 8C92DE3F079; Wed, 16 Nov 2011 15:45:28 +0100 (CET) Date: Wed, 16 Nov 2011 15:45:26 +0100 From: Joel Dahl To: Alexander Motin Message-ID: <20111116144526.GV83971@goofy01.vnodelab.local> References: <4EC392DA.2030302@FreeBSD.org> <4EC3C99C.8020203@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EC3C99C.8020203@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "Bjoern A. Zeeb" , freebsd-stable@freebsd.org Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 15:04:10 -0000 On 16-11-2011 16:33, Alexander Motin wrote: > On 11/16/11 16:14, Bjoern A. Zeeb wrote: > > On Wed, 16 Nov 2011, Alexander Motin wrote: > > > >> Hi. > >> > >> On 11/16/11 08:43, Bjoern A. Zeeb wrote: > >>> we have seen this or a very similar panic for about 1 year now once in > >>> a while and I think I reported it before; this is FreeBSD as guest on > >>> vmware. Seems it was a double panic this time. Could someone please > >>> see what's going on there? It was on 8.x-STABLE in the past and this > >>> is 8.2-RELEASE-p4. > >> > >> The part of code reporting "completing request directly" is IMHO broken > >> by design. It returns request completion before request will actually be > >> completed by lower levels without any knowledge of what's going on > >> there. There is kind of protection against double request completion, > >> but it looks like not always working. May be because that part of code > >> is not locked and nothing prevents that semaphore timeout and normal > >> request timeout/completion to happen simultaneously. It is surprising to > >> see even two traps same time, not sure what synchronized them so > >> precisely. > >> > >> Simple removing that semaphore timeout is not an option, because it will > >> cause deadlock when this wait happen within taskqueue thread that is > >> used to handle requests completion and abort that wait. Avoid waiting > >> inside taskqueue is also impossible without major rewrite. That's why > >> ATA_CAM drops that code completely. > > > > So the bottom line of what you are saying is: > > 1) it's hard to fix right in 8 > > 2) it's not an issue in 9 anymore at all? > > Right. Hmm. We're running many FreeBSD 8.2 machines as guests in VMware but have never encountered the panic described above. Should I be worried? :-) -- Joel From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 16:12:29 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17D5F1065670 for ; Wed, 16 Nov 2011 16:12:29 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id A9B568FC0C for ; Wed, 16 Nov 2011 16:12:28 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 15282153436 for ; Wed, 16 Nov 2011 17:12:26 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qNUIw7dUj4A5 for ; Wed, 16 Nov 2011 17:12:20 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id 2874D153433 for ; Wed, 16 Nov 2011 17:12:20 +0100 (CET) Message-ID: <4EC3E0E4.5010704@digiware.nl> Date: Wed, 16 Nov 2011 17:12:20 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "stable@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 16:12:29 -0000 Hi, I'm getting these: Nov 16 16:40:49 zfs kernel: ata6: port is not ready (timeout 15000ms) tfd = 00000080 Nov 16 16:40:49 zfs kernel: ata6: hardware reset timeout Nov 16 16:41:50 zfs kernel: ata6: port is not ready (timeout 15000ms) tfd = 00000080 Nov 16 16:41:50 zfs kernel: ata6: hardware reset timeout When inserting the tray with a SSD disk connected to that controller. Which is probably due to a BIOS upgrade.... At least it started after upgrading the BIOS. So I'm asking SuperMicro for an older version. When this happens, the system sometimes panics, haven't written the details yet down right now. somewhere in get_devices... After the panic I really need to powerdown the machine, otherwise it boots but stalls at finding any disks. It does not just find no disks, it "freezes" at the point it should report the found disks in the bios-boot. So apparently the ata controller are left in a very confused state. Why is the controller found at boot, and works as it should. And why later it just starts generating these hardware resets?? --WjW From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 16:26:16 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6EE00106566B for ; Wed, 16 Nov 2011 16:26:16 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 22BCF8FC13 for ; Wed, 16 Nov 2011 16:26:15 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RQiJR-0001Cv-4B for freebsd-stable@freebsd.org; Wed, 16 Nov 2011 17:26:13 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Nov 2011 17:26:13 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 16 Nov 2011 17:26:13 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Wed, 16 Nov 2011 17:25:59 +0100 Lines: 32 Message-ID: References: <4EC392DA.2030302@FreeBSD.org> <4EC3C99C.8020203@FreeBSD.org> <20111116144526.GV83971@goofy01.vnodelab.local> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigA2C8CF3294ADA75040E2ECD9" X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111004 Thunderbird/7.0.1 In-Reply-To: <20111116144526.GV83971@goofy01.vnodelab.local> X-Enigmail-Version: 1.1.2 Subject: Re: ATA/Cdrom(?) panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 16:26:16 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigA2C8CF3294ADA75040E2ECD9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 16/11/2011 15:45, Joel Dahl wrote: >=20 > Hmm. We're running many FreeBSD 8.2 machines as guests in VMware but ha= ve > never encountered the panic described above. Should I be worried? :-) >=20 I've encountered them often enough that I started removing cdrom devices from the VMs. --------------enigA2C8CF3294ADA75040E2ECD9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk7D5BcACgkQldnAQVacBcj5uACgiYrp7RhzoOVS8GMzLO+K4xvH GrMAnRgyV0+o6BR9M2lecSfVhLpLSEII =m7KJ -----END PGP SIGNATURE----- --------------enigA2C8CF3294ADA75040E2ECD9-- From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 17:22:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5C75106566B for ; Wed, 16 Nov 2011 17:22:25 +0000 (UTC) (envelope-from peter.maloney@brockmann-consult.de) Received: from mo-p05-ob6.rzone.de (mo-p05-ob6.rzone.de [IPv6:2a01:238:20a:202:53f5::1]) by mx1.freebsd.org (Postfix) with ESMTP id 574B28FC0C for ; Wed, 16 Nov 2011 17:22:25 +0000 (UTC) X-RZG-AUTH: :LWIKdA2leu0bPbLmhzXgqn0MTG6qiKEwQRWfNxSw4HzYIwjsnvdDt2oX8drk23mm07JRnF8H X-RZG-CLASS-ID: mo05 Received: from [192.168.179.42] (hmbg-5f76616b.pool.mediaWays.net [95.118.97.107]) by post.strato.de (mrclete mo57) (RZmta 26.10 AUTH) with (DHE-RSA-AES128-SHA encrypted) ESMTPA id w031c7nAGEvJ7d for ; Wed, 16 Nov 2011 18:22:05 +0100 (MET) Message-ID: <4EC3F13B.4020700@brockmann-consult.de> Date: Wed, 16 Nov 2011 18:22:03 +0100 From: Peter Maloney User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <4EC3E0E4.5010704@digiware.nl> In-Reply-To: <4EC3E0E4.5010704@digiware.nl> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 17:22:26 -0000 Willem, I can only guess, but... Is AHCI enabled in the bios? If you are not using 'fake-raid' for any disks, you should [depending on FreeBSD version, HBA, etc.] probably enable AHCI. Some servers actually come with SATA set in IDE mode. And if you are using zfs, the controller optimally should not be RAID at all. And if you have AHCI enabled already, try disabling it (losing hot swapping ability, and some performance). What version of FreeBSD are you using? I had a terrible experience with ZFS on FreeBSD 8.2 release, and 8.2-stable-April2011. I would recommend upgrading to the latest 8-stable with cvsup. This thread seems related: http://forums.freebsd.org/showthread.php?t=24189 The guy was using 8.2 release, and he downgraded to an old version of the driver to fix, saying that a patch also existed in 8-stable that fixes the problem. Are you using an expander? What HBA / hard disk controller are you using? Peter Am 16.11.2011 17:12, schrieb Willem Jan Withagen: > Hi, > > I'm getting these: > > Nov 16 16:40:49 zfs kernel: ata6: port is not ready (timeout 15000ms) > tfd = 00000080 > Nov 16 16:40:49 zfs kernel: ata6: hardware reset timeout > Nov 16 16:41:50 zfs kernel: ata6: port is not ready (timeout 15000ms) > tfd = 00000080 > Nov 16 16:41:50 zfs kernel: ata6: hardware reset timeout > > When inserting the tray with a SSD disk connected to that controller. > > Which is probably due to a BIOS upgrade.... > At least it started after upgrading the BIOS. So I'm asking SuperMicro > for an older version. > > When this happens, the system sometimes panics, haven't written the > details yet down right now. somewhere in get_devices... > > After the panic I really need to powerdown the machine, otherwise it > boots but stalls at finding any disks. It does not just find no disks, > it "freezes" at the point it should report the found disks in the > bios-boot. > So apparently the ata controller are left in a very confused state. > > Why is the controller found at boot, and works as it should. > And why later it just starts generating these hardware resets?? > > --WjW > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 17:31:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40833106564A for ; Wed, 16 Nov 2011 17:31:45 +0000 (UTC) (envelope-from slonoman2011@yandex.ru) Received: from forward12.mail.yandex.net (forward12.mail.yandex.net [IPv6:2a02:6b8:0:801::2]) by mx1.freebsd.org (Postfix) with ESMTP id A5B1B8FC14 for ; Wed, 16 Nov 2011 17:31:43 +0000 (UTC) Received: from web137.yandex.ru (web137.yandex.ru [95.108.131.158]) by forward12.mail.yandex.net (Yandex) with ESMTP id 229A7C26214 for ; Wed, 16 Nov 2011 21:31:42 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1321464702; bh=lJ99lS2Jr6s9V4eLRJwkDuunxJgfd70NX9IOMow5ha4=; h=From:To:In-Reply-To:References:Subject:MIME-Version:Message-Id: Date:Content-Transfer-Encoding:Content-Type; b=EW3kG7UdbVPkXWZ7pbaeGmyq0bgAuGWYbyn0uSHGZJG2H6GGxUZx5MfjNf3o8RqPd ZMMbxdl5LwFqsQmRrFPBYyGnQ5shvIYDfaN9WzaLeG+byVHggJCKw+8BZ0oysuZnvF Cic7ZKDv1RbAaZkwLJHC8RqwfCXe7DeM6CHJ/UHw= Received: from localhost (localhost.localdomain [127.0.0.1]) by web137.yandex.ru (Yandex) with ESMTP id 0641D1270037 for ; Wed, 16 Nov 2011 21:31:41 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1321464702; bh=lJ99lS2Jr6s9V4eLRJwkDuunxJgfd70NX9IOMow5ha4=; h=From:To:In-Reply-To:References:Subject:MIME-Version:Message-Id: Date:Content-Transfer-Encoding:Content-Type; b=EW3kG7UdbVPkXWZ7pbaeGmyq0bgAuGWYbyn0uSHGZJG2H6GGxUZx5MfjNf3o8RqPd ZMMbxdl5LwFqsQmRrFPBYyGnQ5shvIYDfaN9WzaLeG+byVHggJCKw+8BZ0oysuZnvF Cic7ZKDv1RbAaZkwLJHC8RqwfCXe7DeM6CHJ/UHw= X-Yandex-Spam: 1 Received: from nat140-249-205-109.tvoe.tv (nat140-249-205-109.tvoe.tv [109.205.249.140]) by web137.yandex.ru with HTTP; Wed, 16 Nov 2011 21:31:41 +0400 From: Slono Slono To: freebsd-stable@freebsd.org In-Reply-To: <131321321455326@web143.yandex.ru> References: <131321321455326@web143.yandex.ru> MIME-Version: 1.0 Message-Id: <53751321464701@web137.yandex.ru> Date: Wed, 16 Nov 2011 21:31:41 +0400 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset=koi8-r Subject: Re: PF with VIMAGE still crash kernel X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 17:31:45 -0000 SGkuDQoNClBlbmRpbmcsIHdoaWxlIGZyZWVic2QtdmlydHVhbGl6YXRpb25AIGV4cGVydHMgY2Fu IGZpeCBhIHByb2JsZW0gLSB0aGVyZSBpcyBhbiBvZmZlciBhcyB3b3JrLWFyb3VuZCBmb3IgdXBj b21pbmcgOS4wLVJFTEVBU0UgdG8gbWFrZSBjaGVjayB2aW1hZ2UgYXQgcGYgdGhhdCBwZW9wbGUg YXQgbGVhc3QgZGlkbid0IHJlY2VpdmUgYSBrZXJuZWwgcGFuaWMNCg0KMTYuMTEuMjAxMSwgMTg6 NTUsICJTbG9ubyBTbG9ubyIgPHNsb25vbWFuMjAxMUB5YW5kZXgucnU+Og0KDQo+IJpIaSwNCj4N Cj4gmkp1c3QgaW5mb3JtIHRoYXQgUFI6IGh0dHA6Ly93d3cuZnJlZWJzZC5vcmcvY2dpL3F1ZXJ5 LXByLmNnaT9wcj1rZXJuLzE0ODE1NSBpcyBzdGlsbCBhY3R1YWwgZm9yIGZvcnRoY29taW5nIHJl bGVhc2UgOS4wIJqaKCA5LjAtUFJFUkVMRUFTRSk6DQo+DQo+IJpIb3cgdG8gcmVwZWF0ZToNCj4N Cj4gmjEpIFJlYm9vdCBzeXN0ZW0gdG8ga2VybmVsIHdpdGgNCj4gmm9wdGlvbnMgVklNQUdFDQo+ DQo+IJoyKSBUd28gZGlmZmVyZW50IHZhcmlhbnRzOg0KPiCaYSkgZWNobyAibWFwIGxvMCBmcm9t IDEwLjAuMC4wLzI0IHRvICEgMTAuMC4wLjAvMjQgLT4gMTI3LjAuMC4xLzMyIiA+IC9ldGMvaXBu YXQucnVsZXMgOyCac2VydmljZSBpcG5hdCBvbmVyZXN0YXJ0DQo+IJpiKSBrbGRsb2FkIHBmICYm IGtsZHVubG9hZCBwZg0KPg0KPiCaSWYgaXQgaXMgcHJvYmxlbSBub3QgdG8gc29sdmUgcXVpY2ts eSAtIEkgc3VnZ2VzdCB0byBtYWtlIHRoZSBtZXNzYWdlIG9uIG11dHVhbCBleGNsdXNpb24gaW4g YSBrZXJuZWwgb2Ygb3B0aW9ucyBWSU1BR0UgYW5kIHBmDQo+IJphdCBjb25maWcsIGFuZCBjaGVj ayBhdCAia2xkbG9hZCBwZiIgYWJvdXQgcHJlc2VuY2Uga2Vybi5mZWF0dXJlcy52aW1hZ2UNCj4N Cj4gmlRoaXMgcHJvYmxlbSBpcyByYXRoZXIgc2VyaW91cyBhbHNvIG1hbnkgb24gOS4wLVJFTEVB U0UgaXQgd2lsbCBub3RpY2UNCj4NCj4gmmNvcmUudHh0LjAgZnJvbSBkZWF0aCBieSBtZXRob2Qg KGEpOg0KPg0KPiCaLS0vY3V0Ly0tDQo+IJpic2QubXkuZG9tYWluIGR1bXBlZCBjb3JlIC0gc2Vl IC92YXIvY3Jhc2gvdm1jb3JlLjANCj4NCj4gmldlZCBOb3YgMTYgMTg6NTI6MzEgRUVUIDIwMTEN Cj4NCj4gmkZyZWVCU0QgYnNkLm15LmRvbWFpbiA5LjAtUFJFUkVMRUFTRSBGcmVlQlNEIDkuMC1Q UkVSRUxFQVNFICM3OiBXZWQgTm92IDE2IDE4OjM2OjQ0IEVFVCAyMDExIJqamppyb290QGJzZC5t eS5kb21haW46L3Vzci9vYmovdXNyL3NyYy9zeXMvR0VORVJJQyCaYW1kNjQNCj4NCj4gmnBhbmlj OiBwYWdlIGZhdWx0DQo+DQo+IJpHTlUgZ2RiIDYuMS4xIFtGcmVlQlNEXQ0KPiCaQ29weXJpZ2h0 IDIwMDQgRnJlZSBTb2Z0d2FyZSBGb3VuZGF0aW9uLCBJbmMuDQo+IJpHREIgaXMgZnJlZSBzb2Z0 d2FyZSwgY292ZXJlZCBieSB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UsIGFuZCB5b3Ug YXJlDQo+IJp3ZWxjb21lIHRvIGNoYW5nZSBpdCBhbmQvb3IgZGlzdHJpYnV0ZSBjb3BpZXMgb2Yg aXQgdW5kZXIgY2VydGFpbiBjb25kaXRpb25zLg0KPiCaVHlwZSAic2hvdyBjb3B5aW5nIiB0byBz ZWUgdGhlIGNvbmRpdGlvbnMuDQo+IJpUaGVyZSBpcyBhYnNvbHV0ZWx5IG5vIHdhcnJhbnR5IGZv ciBHREIuIJpUeXBlICJzaG93IHdhcnJhbnR5IiBmb3IgZGV0YWlscy4NCj4gmlRoaXMgR0RCIHdh cyBjb25maWd1cmVkIGFzICJhbWQ2NC1tYXJjZWwtZnJlZWJzZCIuLi4NCj4NCj4gmlVucmVhZCBw b3J0aW9uIG9mIHRoZSBrZXJuZWwgbWVzc2FnZSBidWZmZXI6DQo+IJpDb3B5cmlnaHQgKGMpIDE5 OTItMjAxMSBUaGUgRnJlZUJTRCBQcm9qZWN0Lg0KPiCaQ29weXJpZ2h0IChjKSAxOTc5LCAxOTgw LCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAxOTk0DQo+IJqampqa mpqamlRoZSBSZWdlbnRzIG9mIHRoZSBVbml2ZXJzaXR5IG9mIENhbGlmb3JuaWEuIEFsbCByaWdo dHMgcmVzZXJ2ZWQuDQo+IJpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhl IEZyZWVCU0QgRm91bmRhdGlvbi4NCj4gmkZyZWVCU0QgOS4wLVBSRVJFTEVBU0UgIzc6IFdlZCBO b3YgMTYgMTg6MzY6NDQgRUVUIDIwMTENCj4gmpqamppyb290QGJzZC5teS5kb21haW46L3Vzci9v YmovdXNyL3NyYy9zeXMvR0VORVJJQyBhbWQ2NA0KPiCaQ1BVOiBBTUQgUGhlbm9tKHRtKSBJSSBY NiAxMDkwVCBQcm9jZXNzb3IgKDMwNTEuNjItTUh6IEs4LWNsYXNzIENQVSkNCj4gmpqaT3JpZ2lu ID0gIkF1dGhlbnRpY0FNRCIgmklkID0gMHgxMDBmYTAgmkZhbWlseSA9IDEwIJpNb2RlbCA9IGEg mlN0ZXBwaW5nID0gMA0KPiCamppGZWF0dXJlcz0weDc4M2ZiZmY8RlBVLFZNRSxERSxQU0UsVFND LE1TUixQQUUsTUNFLENYOCxBUElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsTU1Y LEZYU1IsU1NFLFNTRTI+DQo+IJqamkZlYXR1cmVzMj0weDk8U1NFMyxNT04+DQo+IJqamkFNRCBG ZWF0dXJlcz0weGVhMTAwODAwPFNZU0NBTEwsTlgsRkZYU1IsUkRUU0NQLExNLDNETm93ISssM0RO b3chPg0KPiCamppBTUQgRmVhdHVyZXMyPTB4MTE8TEFIRixDUjg+DQo+IJpyZWFsIG1lbW9yeSCa PSAxMDczNjc2Mjg4ICgxMDIzIE1CKQ0KPiCaYXZhaWwgbWVtb3J5ID0gMTAxMTE4NzcxMiAoOTY0 IE1CKQ0KPiCaRXZlbnQgdGltZXIgIkxBUElDIiBxdWFsaXR5IDQwMA0KPiCaQUNQSSBBUElDIFRh YmxlOiA8VkJPWCCamlZCT1hBUElDPg0KPiCaV0FSTklORzogVklNQUdFICh2aXJ0dWFsaXplZCBu ZXR3b3JrIHN0YWNrKSBpcyBhIGhpZ2hseSBleHBlcmltZW50YWwgZmVhdHVyZS4NCj4gmmlvYXBp YzA6IENoYW5naW5nIEFQSUMgSUQgdG8gMQ0KPiCaaW9hcGljMCA8VmVyc2lvbiAxLjE+IGlycXMg MC0yMyBvbiBtb3RoZXJib2FyZA0KPiCaa2JkMSBhdCBrYmRtdXgwDQo+IJphY3BpMDogPFZCT1gg VkJPWFhTRFQ+IG9uIG1vdGhlcmJvYXJkDQo+IJphY3BpMDogUG93ZXIgQnV0dG9uIChmaXhlZCkN Cj4gmmFjcGkwOiBTbGVlcCBCdXR0b24gKGZpeGVkKQ0KPiCaVGltZWNvdW50ZXIgIkFDUEktZmFz dCIgZnJlcXVlbmN5IDM1Nzk1NDUgSHogcXVhbGl0eSA5MDANCj4gmmFjcGlfdGltZXIwOiA8MzIt Yml0IHRpbWVyIGF0IDMuNTc5NTQ1TUh6PiBwb3J0IDB4NDAwOC0weDQwMGIgb24gYWNwaTANCj4g mmNwdTA6IDxBQ1BJIENQVT4gb24gYWNwaTANCj4gmnBjaWIwOiA8QUNQSSBIb3N0LVBDSSBicmlk Z2U+IHBvcnQgMHhjZjgtMHhjZmYgb24gYWNwaTANCj4gmnBjaTA6IDxBQ1BJIFBDSSBidXM+IG9u IHBjaWIwDQo+IJppc2FiMDogPFBDSS1JU0EgYnJpZGdlPiBhdCBkZXZpY2UgMS4wIG9uIHBjaTAN Cj4gmmlzYTA6IDxJU0EgYnVzPiBvbiBpc2FiMA0KPiCaYXRhcGNpMDogPEludGVsIFBJSVg0IFVE TUEzMyBjb250cm9sbGVyPiBwb3J0IDB4MWYwLTB4MWY3LDB4M2Y2LDB4MTcwLTB4MTc3LDB4Mzc2 LDB4ZDAwMC0weGQwMGYgYXQgZGV2aWNlIDEuMSBvbiBwY2kwDQo+IJphdGEwOiA8QVRBIGNoYW5u ZWwgMD4gb24gYXRhcGNpMA0KPiCaYXRhMTogPEFUQSBjaGFubmVsIDE+IG9uIGF0YXBjaTANCj4g mnZnYXBjaTA6IDxWR0EtY29tcGF0aWJsZSBkaXNwbGF5PiBtZW0gMHhlMDAwMDAwMC0weGUwZmZm ZmZmIGlycSAxOCBhdCBkZXZpY2UgMi4wIG9uIHBjaTANCj4gmmVtMDogPEludGVsKFIpIFBSTy8x MDAwIExlZ2FjeSBOZXR3b3JrIENvbm5lY3Rpb24gMS4wLjM+IHBvcnQgMHhkMDEwLTB4ZDAxNyBt ZW0gMHhmMDAwMDAwMC0weGYwMDFmZmZmIGlycSAxOSBhdCBkZXZpY2UgMy4wIG9uIHBjaTANCj4g mmVtMDogRXRoZXJuZXQgYWRkcmVzczogMDg6MDA6Mjc6NGI6NmY6ZjANCj4gmnBjaTA6IDxiYXNl IHBlcmlwaGVyYWw+IGF0IGRldmljZSA0LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkNCj4gmnBjbTA6 IDxJbnRlbCBJQ0ggKDgyODAxQUEpPiBwb3J0IDB4ZDEwMC0weGQxZmYsMHhkMjAwLTB4ZDIzZiBp cnEgMjEgYXQgZGV2aWNlIDUuMCBvbiBwY2kwDQo+IJpwY20wOiA8U2lnbWFUZWwgU1RBQzk3MDAv ODMvODQgQUM5NyBDb2RlYz4NCj4gmm9oY2kwOiA8T0hDSSAoZ2VuZXJpYykgVVNCIGNvbnRyb2xs ZXI+IG1lbSAweGYwODA0MDAwLTB4ZjA4MDRmZmYgaXJxIDIyIGF0IGRldmljZSA2LjAgb24gcGNp MA0KPiCadXNidXMwOiA8T0hDSSAoZ2VuZXJpYykgVVNCIGNvbnRyb2xsZXI+IG9uIG9oY2kwDQo+ IJpwY2kwOiA8YnJpZGdlPiBhdCBkZXZpY2UgNy4wIChubyBkcml2ZXIgYXR0YWNoZWQpDQo+IJph Y3BpX2FjYWQwOiA8QUMgQWRhcHRlcj4gb24gYWNwaTANCj4gmmF0a2JkYzA6IDxLZXlib2FyZCBj b250cm9sbGVyIChpODA0Mik+IHBvcnQgMHg2MCwweDY0IGlycSAxIG9uIGFjcGkwDQo+IJphdGti ZDA6IDxBVCBLZXlib2FyZD4gaXJxIDEgb24gYXRrYmRjMA0KPiCaa2JkMCBhdCBhdGtiZDANCj4g mmF0a2JkMDogW0dJQU5ULUxPQ0tFRF0NCj4gmnBzbTA6IDxQUy8yIE1vdXNlPiBpcnEgMTIgb24g YXRrYmRjMA0KPiCacHNtMDogW0dJQU5ULUxPQ0tFRF0NCj4gmnBzbTA6IG1vZGVsIEludGVsbGlN b3VzZSBFeHBsb3JlciwgZGV2aWNlIElEIDQNCj4gmmF0dGltZXIwOiA8QVQgdGltZXI+IHBvcnQg MHg0MC0weDQzLDB4NTAtMHg1MyBvbiBhY3BpMA0KPiCaVGltZWNvdW50ZXIgImk4MjU0IiBmcmVx dWVuY3kgMTE5MzE4MiBIeiBxdWFsaXR5IDANCj4gmkV2ZW50IHRpbWVyICJpODI1NCIgZnJlcXVl bmN5IDExOTMxODIgSHogcXVhbGl0eSAxMDANCj4gmnNjMDogPFN5c3RlbSBjb25zb2xlPiBhdCBm bGFncyAweDEwMCBvbiBpc2EwDQo+IJpzYzA6IFZHQSA8MTYgdmlydHVhbCBjb25zb2xlcywgZmxh Z3M9MHgzMDA+DQo+IJp2Z2EwOiA8R2VuZXJpYyBJU0EgVkdBPiBhdCBwb3J0IDB4M2MwLTB4M2Rm IGlvbWVtIDB4YTAwMDAtMHhiZmZmZiBvbiBpc2EwDQo+IJphdHJ0YzA6IDxBVCByZWFsdGltZSBj bG9jaz4gYXQgcG9ydCAweDcwIGlycSA4IG9uIGlzYTANCj4gmkV2ZW50IHRpbWVyICJSVEMiIGZy ZXF1ZW5jeSAzMjc2OCBIeiBxdWFsaXR5IDANCj4gmnBwYzA6IGNhbm5vdCByZXNlcnZlIEkvTyBw b3J0IHJhbmdlDQo+IJpUaW1lY291bnRlcnMgdGljayBldmVyeSAxMC4wMDAgbXNlYw0KPiCacGNt MDogbWVhc3VyZWQgYWM5NyBsaW5rIHJhdGUgYXQgODA0MDgzIEh6DQo+IJp1c2J1czA6IDEyTWJw cyBGdWxsIFNwZWVkIFVTQiB2MS4wDQo+IJp1Z2VuMC4xOiA8QXBwbGU+IGF0IHVzYnVzMA0KPiCa dWh1YjA6IDxBcHBsZSBPSENJIHJvb3QgSFVCLCBjbGFzcyA5LzAsIHJldiAxLjAwLzEuMDAsIGFk ZHIgMT4gb24gdXNidXMwDQo+IJphZGEwIGF0IGF0YTAgYnVzIDAgc2NidXMwIHRhcmdldCAwIGx1 biAwDQo+IJphZGEwOiA8VkJPWCBIQVJERElTSyAxLjA+IEFUQS02IGRldmljZQ0KPiCaYWRhMDog MzMuMzAwTUIvcyB0cmFuc2ZlcnMgKFVETUEyLCBQSU8gNjU1MzZieXRlcykNCj4gmmFkYTA6IDQ0 NDg2TUIgKDkxMTA3MzI4IDUxMiBieXRlIHNlY3RvcnM6IDE2SCA2M1MvVCAxNjM4M0MpDQo+IJph ZGEwOiBQcmV2aW91c2x5IHdhcyBrbm93biBhcyBhZDANCj4gmmNkMCBhdCBhdGExIGJ1cyAwIHNj YnVzMSB0YXJnZXQgMCBsdW4gMA0KPiCaY2QwOiA8VkJPWCBDRC1ST00gMS4wPiBSZW1vdmFibGUg Q0QtUk9NIFNDU0ktMCBkZXZpY2UNCj4gmmNkMDogMzMuMzAwTUIvcyB0cmFuc2ZlcnMgKFVETUEy LCBBVEFQSSAxMmJ5dGVzLCBQSU8gNjU1MzRieXRlcykNCj4gmmNkMDogQXR0ZW1wdCB0byBxdWVy eSBkZXZpY2Ugc2l6ZSBmYWlsZWQ6IE5PVCBSRUFEWSwgTWVkaXVtIG5vdCBwcmVzZW50DQo+IJpU aW1lY291bnRlciAiVFNDIiBmcmVxdWVuY3kgMzA1MTYyMTAwOSBIeiBxdWFsaXR5IDgwMA0KPiCa Um9vdCBtb3VudCB3YWl0aW5nIGZvcjogdXNidXMwDQo+IJp1aHViMDogOCBwb3J0cyB3aXRoIDgg cmVtb3ZhYmxlLCBzZWxmIHBvd2VyZWQNCj4gmlRyeWluZyB0byBtb3VudCByb290IGZyb20gdWZz Oi9kZXYvYWRhMHAyIFtyd10uLi4NCj4gmjwxMTg+U2V0dGluZyBob3N0dXVpZDogYjhlNjNiZjQt Nzc5ZC00MzZkLWEyMGEtYzM3N2Y2M2ZkNTNiLg0KPiCaPDExOD5TZXR0aW5nIGhvc3RpZDogMHhm OWNlZGZjNS4NCj4gmjwxMTg+RW50cm9weSBoYXJ2ZXN0aW5nOiBpbnRlcnJ1cHRzIGV0aGVybmV0 IHBvaW50X3RvX3BvaW50IGtpY2tzdGFydC4NCj4gmjwxMTg+U3RhcnRpbmcgZmlsZSBzeXN0ZW0g Y2hlY2tzOg0KPiCaPDExOD4vZGV2L2FkYTBwMjogRklMRSBTWVNURU0gQ0xFQU47IFNLSVBQSU5H IENIRUNLUw0KPiCaPDExOD4vZGV2L2FkYTBwMjogY2xlYW4sIDc5OTU1NTEgZnJlZSAoMzQzNTkg ZnJhZ3MsIDk5NTE0OSBibG9ja3MsIDAuMyUgZnJhZ21lbnRhdGlvbikNCj4gmjwxMTg+TW91bnRp bmcgbG9jYWwgZmlsZSBzeXN0ZW1zOi4NCj4gmjwxMTg+U2V0dGluZyBob3N0bmFtZTogYnNkLm15 LmRvbWFpbi4NCj4gmjwxMTg+U3RhcnRpbmcgTmV0d29yazogbG8wIGVtMC4NCj4gmjwxMTg+bG8w OiBmbGFncz04MDQ5PFVQLExPT1BCQUNLLFJVTk5JTkcsTVVMVElDQVNUPiBtZXRyaWMgMCBtdHUg MTYzODQNCj4gmjwxMTg+IG9wdGlvbnM9MzxSWENTVU0sVFhDU1VNPg0KPiCaPDExOD4gaW5ldDYg OjoxIHByZWZpeGxlbiAxMjgNCj4gmjwxMTg+IGluZXQ2IGZlODA6OjElbG8wIHByZWZpeGxlbiA2 NCBzY29wZWlkIDB4Mw0KPiCaPDExOD4gaW5ldCAxMjcuMC4wLjEgbmV0bWFzayAweGZmMDAwMDAw DQo+IJo8MTE4PiBuZDYgb3B0aW9ucz0yMTxQRVJGT1JNTlVELEFVVE9fTElOS0xPQ0FMPg0KPiCa PDExOD5lbTA6IGZsYWdzPTg4MDI8QlJPQURDQVNULFNJTVBMRVgsTVVMVElDQVNUPiBtZXRyaWMg MCBtdHUgMTUwMA0KPiCaPDExOD4gb3B0aW9ucz05YjxSWENTVU0sVFhDU1VNLFZMQU5fTVRVLFZM QU5fSFdUQUdHSU5HLFZMQU5fSFdDU1VNPg0KPiCaPDExOD4gZXRoZXIgMDg6MDA6Mjc6NGI6NmY6 ZjANCj4gmjwxMTg+IG5kNiBvcHRpb25zPTI5PFBFUkZPUk1OVUQsSUZESVNBQkxFRCxBVVRPX0xJ TktMT0NBTD4NCj4gmjwxMTg+IG1lZGlhOiBFdGhlcm5ldCBhdXRvc2VsZWN0ICgxMDAwYmFzZVQg PGZ1bGwtZHVwbGV4PikNCj4gmjwxMTg+IHN0YXR1czogYWN0aXZlDQo+IJo8MTE4PlN0YXJ0aW5n IGRldmQuDQo+IJo8MTE4PlN0YXJ0aW5nIE5ldHdvcms6IGVtMC4NCj4gmjwxMTg+ZW0wOiBmbGFn cz04ODAyPEJST0FEQ0FTVCxTSU1QTEVYLE1VTFRJQ0FTVD4gbWV0cmljIDAgbXR1IDE1MDANCj4g mjwxMTg+IG9wdGlvbnM9OWI8UlhDU1VNLFRYQ1NVTSxWTEFOX01UVSxWTEFOX0hXVEFHR0lORyxW TEFOX0hXQ1NVTT4NCj4gmjwxMTg+IGV0aGVyIDA4OjAwOjI3OjRiOjZmOmYwDQo+IJo8MTE4PiBu ZDYgb3B0aW9ucz0yOTxQRVJGT1JNTlVELElGRElTQUJMRUQsQVVUT19MSU5LTE9DQUw+DQo+IJo8 MTE4PiBtZWRpYTogRXRoZXJuZXQgYXV0b3NlbGVjdCAoMTAwMGJhc2VUIDxmdWxsLWR1cGxleD4p DQo+IJo8MTE4PiBzdGF0dXM6IGFjdGl2ZQ0KPiCaPDExOD5TdGFydGluZyBOZXR3b3JrOiB1c2J1 czAuDQo+IJo8MTE4PmFkZCBuZXQgOjpmZmZmOjAuMC4wLjA6IGdhdGV3YXkgOjoxDQo+IJo8MTE4 PmFkZCBuZXQgOjowLjAuMC4wOiBnYXRld2F5IDo6MQ0KPiCaPiA8MTE4PmFkZCBuZXQgZmU4MDo6 OiBnYXRld2F5IDo6MQ0KPiCaPDExOD5hZGQgbmV0IGZmMDI6OjogZ2F0ZXdheSA6OjENCj4gmjwx MTg+RUxGIGxkY29uZmlnIHBhdGg6IC9saWIgL3Vzci9saWIgL3Vzci9saWIvY29tcGF0IC91c3Iv bG9jYWwvbGliDQo+IJo8MTE4PjMyLWJpdCBjb21wYXRpYmlsaXR5IGxkY29uZmlnIHBhdGg6IC91 c3IvbGliMzINCj4gmjwxMTg+Q3JlYXRpbmcgYW5kL29yIHRyaW1taW5nIGxvZyBmaWxlcy4NCj4g mjwxMTg+Tm8gY29yZSBkdW1wcyBmb3VuZC4NCj4gmjwxMTg+Q2xlYXJpbmcgL3RtcCAoWCByZWxh dGVkKS4NCj4gmjwxMTg+VXBkYXRpbmcgbW90ZDouDQo+IJo8MTE4PkNvbmZpZ3VyaW5nIHN5c2Nv bnM6IGtleW1hcCBibGFua3RpbWUuDQo+IJo8MTE4PlN0YXJ0aW5nIGNyb24uDQo+IJo8MTE4PlN0 YXJ0aW5nIGJhY2tncm91bmQgZmlsZSBzeXN0ZW0gY2hlY2tzIGluIDYwIHNlY29uZHMuDQo+IJo8 MTE4Pg0KPiCaPDExOD5XZWQgTm92IDE2IDE4OjQ4OjUwIEVFVCAyMDExDQo+IJpJUCBGaWx0ZXI6 IHY0LjEuMjggaW5pdGlhbGl6ZWQuIJpEZWZhdWx0ID0gcGFzcyBhbGwsIExvZ2dpbmcgPSBlbmFi bGVkDQo+DQo+IJpGYXRhbCB0cmFwIDEyOiBwYWdlIGZhdWx0IHdoaWxlIGluIGtlcm5lbCBtb2Rl DQo+IJpjcHVpZCA9IDA7IGFwaWMgaWQgPSAwMA0KPiCaZmF1bHQgdmlydHVhbCBhZGRyZXNzID0g MHgyOA0KPiCaZmF1bHQgY29kZSA9IHN1cGVydmlzb3IgcmVhZCBkYXRhLCBwYWdlIG5vdCBwcmVz ZW50DQo+IJppbnN0cnVjdGlvbiBwb2ludGVyID0gMHgyMDoweGZmZmZmZmZmODA4ZTI2MWENCj4g mnN0YWNrIHBvaW50ZXIgmpqampqamj0gMHgyODoweGZmZmZmZjgwMDAzOTM3ZjANCj4gmmZyYW1l IHBvaW50ZXIgmpqampqamj0gMHgyODoweGZmZmZmZjgwMDAzOTM4MTANCj4gmmNvZGUgc2VnbWVu dCA9IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0eXBlIDB4MWINCj4gmpqampqampqampqampqa mpqampqampqamj0gRFBMIDAsIHByZXMgMSwgbG9uZyAxLCBkZWYzMiAwLCBncmFuIDENCj4gmnBy b2Nlc3NvciBlZmxhZ3MgPSBpbnRlcnJ1cHQgZW5hYmxlZCwgcmVzdW1lLCBJT1BMID0gMA0KPiCa Y3VycmVudCBwcm9jZXNzID0gMTA4MyAoaXBuYXQpDQo+IJp0cmFwIG51bWJlciA9IDEyDQo+IJpw YW5pYzogcGFnZSBmYXVsdA0KPiCaY3B1aWQgPSAwDQo+IJpLREI6IHN0YWNrIGJhY2t0cmFjZToN Cj4gmiMwIDB4ZmZmZmZmZmY4MDg2OTRhZSBhdCBrZGJfYmFja3RyYWNlKzB4NWUNCj4gmiMxIDB4 ZmZmZmZmZmY4MDgzM2U0NyBhdCBwYW5pYysweDE4Nw0KPiCaIzIgMHhmZmZmZmZmZjgwYjI5NzQw IGF0IHRyYXBfZmF0YWwrMHgyOTANCj4gmiMzIDB4ZmZmZmZmZmY4MGIyOWE4OSBhdCB0cmFwX3Bm YXVsdCsweDFmOQ0KPiCaIzQgMHhmZmZmZmZmZjgwYjI5ZjRmIGF0IHRyYXArMHgzZGYNCj4gmiM1 IDB4ZmZmZmZmZmY4MGIxNDQ3ZiBhdCBjYWxsdHJhcCsweDgNCj4gmiM2IDB4ZmZmZmZmZmY4MTYy ZDg2YSBhdCBmcl9yZXNvbHZlbmljKzB4MWENCj4gmiM3IDB4ZmZmZmZmZmY4MTYxNjliNSBhdCBu YXRfcmVzb2x2ZXJ1bGUrMHgyNQ0KPiCaIzggMHhmZmZmZmZmZjgxNjE3OTQzIGF0IGZyX25hdF9p b2N0bCsweGYwMw0KPiCaIzkgMHhmZmZmZmZmZjgwNzU1MTRiIGF0IGRldmZzX2lvY3RsX2YrMHg3 Yg0KPiCaIzEwIDB4ZmZmZmZmZmY4MDg3YWJhNSBhdCBrZXJuX2lvY3RsKzB4MTE1DQo+IJojMTEg MHhmZmZmZmZmZjgwODdhZGRkIGF0IHN5c19pb2N0bCsweGZkDQo+IJojMTIgMHhmZmZmZmZmZjgw YjI5MDMwIGF0IGFtZDY0X3N5c2NhbGwrMHg0NTANCj4gmiMxMyAweGZmZmZmZmZmODBiMTQ3Njcg YXQgWGZhc3Rfc3lzY2FsbCsweGY3DQo+IJpVcHRpbWU6IDNtMTdzDQo+IJpEdW1waW5nIDgwIG91 dCBvZiAxMDA1IE1COi4uMjAlLi40MCUuLjYwJS4uODAlLi4xMDAlDQo+DQo+IJpSZWFkaW5nIHN5 bWJvbHMgZnJvbSAvYm9vdC9rZXJuZWwvaXBsLmtvLi4uUmVhZGluZyBzeW1ib2xzIGZyb20gL2Jv b3Qva2VybmVsL2lwbC5rby5zeW1ib2xzLi4uZG9uZS4NCj4gmmRvbmUuDQo+IJpMb2FkZWQgc3lt Ym9scyBmb3IgL2Jvb3Qva2VybmVsL2lwbC5rbw0KPiCaIzAgmmRvYWR1bXAgKHRleHRkdW1wPVZh cmlhYmxlICJ0ZXh0ZHVtcCIgaXMgbm90IGF2YWlsYWJsZS4NCj4gmikgYXQgcGNwdS5oOjIyNA0K PiCaMjI0IHBjcHUuaDogTm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeS4NCj4gmpqampqampqaaW4g cGNwdS5oDQo+IJooa2dkYikgIzAgmmRvYWR1bXAgKHRleHRkdW1wPVZhcmlhYmxlICJ0ZXh0ZHVt cCIgaXMgbm90IGF2YWlsYWJsZS4NCj4gmikgYXQgcGNwdS5oOjIyNA0KPiCaIzEgmjB4ZmZmZmZm ZmY4MDgzMzk4NSBpbiBrZXJuX3JlYm9vdCAoaG93dG89MjYwKQ0KPiCampqammF0IC91c3Ivc3Jj L3N5cy9rZXJuL2tlcm5fc2h1dGRvd24uYzo0NDINCj4gmiMyIJoweGZmZmZmZmZmODA4MzNlMzEg aW4gcGFuaWMgKGZtdD1WYXJpYWJsZSAiZm10IiBpcyBub3QgYXZhaWxhYmxlLg0KPiCaKQ0KPiCa mpqammF0IC91c3Ivc3JjL3N5cy9rZXJuL2tlcm5fc2h1dGRvd24uYzo2MDcNCj4gmiMzIJoweGZm ZmZmZmZmODBiMjk3NDAgaW4gdHJhcF9mYXRhbCAoZnJhbWU9MHhjLCBldmE9VmFyaWFibGUgImV2 YSIgaXMgbm90IGF2YWlsYWJsZS4NCj4gmikNCj4gmpqampphdCAvdXNyL3NyYy9zeXMvYW1kNjQv YW1kNjQvdHJhcC5jOjgxOA0KPiCaIzQgmjB4ZmZmZmZmZmY4MGIyOWE4OSBpbiB0cmFwX3BmYXVs dCAoZnJhbWU9MHhmZmZmZmY4MDAwMzkzNzQwLCB1c2VybW9kZT0wKQ0KPiCampqammF0IC91c3Iv c3JjL3N5cy9hbWQ2NC9hbWQ2NC90cmFwLmM6NzM0DQo+IJojNSCaMHhmZmZmZmZmZjgwYjI5ZjRm IGluIHRyYXAgKGZyYW1lPTB4ZmZmZmZmODAwMDM5Mzc0MCkNCj4gmpqampphdCAvdXNyL3NyYy9z eXMvYW1kNjQvYW1kNjQvdHJhcC5jOjQ3Mw0KPiCaIzYgmjB4ZmZmZmZmZmY4MGIxNDQ3ZiBpbiBj YWxsdHJhcCAoKQ0KPiCampqammF0IC91c3Ivc3JjL3N5cy9hbWQ2NC9hbWQ2NC9leGNlcHRpb24u UzoyMjgNCj4gmiM3IJoweGZmZmZmZmZmODA4ZTI2MWEgaW4gaWZ1bml0IChuYW1lPTB4ZmZmZmZl MDAwMjdjZDU0NCAibG8wIikNCj4gmpqampphdCAvdXNyL3NyYy9zeXMvbmV0L2lmLmM6MjAyNw0K PiCaIzggmjB4ZmZmZmZmZmY4MTYyZDg2YSBpbiBmcl9yZXNvbHZlbmljIChuYW1lPVZhcmlhYmxl ICJuYW1lIiBpcyBub3QgYXZhaWxhYmxlLg0KPiCaKQ0KPiCampqammF0IC91c3Ivc3JjL3N5cy9t b2R1bGVzL2lwZmlsdGVyLy4uLy4uL2NvbnRyaWIvaXBmaWx0ZXIvbmV0aW5ldC9maWwuYzo2NTY1 DQo+IJojOSCaMHhmZmZmZmZmZjgxNjE2OWI1IGluIG5hdF9yZXNvbHZlcnVsZSAobj0weGZmZmZm ZTAwMDI3Y2Q0MDApDQo+IJqampqaYXQgL3Vzci9zcmMvc3lzL21vZHVsZXMvaXBmaWx0ZXIvLi4v Li4vY29udHJpYi9pcGZpbHRlci9uZXRpbmV0L2lwX25hdC5jOjExMDgNCj4gmiMxMCAweGZmZmZm ZmZmODE2MTc5NDMgaW4gZnJfbmF0X2lvY3RsIChkYXRhPTB4ZmZmZmZlMDAwMjQ1NTcwMCAiIiwN Cj4gmpqamppjbWQ9MjE1MTE4MjkwOCwgbW9kZT0yLCB1aWQ9MCwgY3R4PTB4ZmZmZmZlMDAwMjg2 OTAwMCkNCj4gmpqampphdCAvdXNyL3NyYy9zeXMvbW9kdWxlcy9pcGZpbHRlci8uLi8uLi9jb250 cmliL2lwZmlsdGVyL25ldGluZXQvaXBfbmF0LmM6OTc2DQo+IJojMTEgMHhmZmZmZmZmZjgwNzU1 MTRiIGluIGRldmZzX2lvY3RsX2YgKGZwPTB4ZmZmZmZlMDAwMjVkMjRiMCwNCj4gmpqamppjb209 MjE1MTE4MjkwOCwgZGF0YT1WYXJpYWJsZSAiZGF0YSIgaXMgbm90IGF2YWlsYWJsZS4NCj4gmikg YXQgL3Vzci9zcmMvc3lzL2ZzL2RldmZzL2RldmZzX3Zub3BzLmM6NzQ0DQo+IJojMTIgMHhmZmZm ZmZmZjgwODdhYmE1IGluIGtlcm5faW9jdGwgKHRkPVZhcmlhYmxlICJ0ZCIgaXMgbm90IGF2YWls YWJsZS4NCj4gmikgYXQgZmlsZS5oOjI3OA0KPiCaIzEzIDB4ZmZmZmZmZmY4MDg3YWRkZCBpbiBz eXNfaW9jdGwgKHRkPTB4ZmZmZmZlMDAwMjg2OTAwMCwNCj4gmpqampp1YXA9MHhmZmZmZmY4MDAw Mzk0YmMwKSBhdCAvdXNyL3NyYy9zeXMva2Vybi9zeXNfZ2VuZXJpYy5jOjY4MQ0KPiCaIzE0IDB4 ZmZmZmZmZmY4MGIyOTAzMCBpbiBhbWQ2NF9zeXNjYWxsICh0ZD0weGZmZmZmZTAwMDI4NjkwMDAs IHRyYWNlZD0wKQ0KPiCampqammF0IHN1YnJfc3lzY2FsbC5jOjEzMQ0KPiCaIzE1IDB4ZmZmZmZm ZmY4MGIxNDc2NyBpbiBYZmFzdF9zeXNjYWxsICgpDQo+IJqampqaYXQgL3Vzci9zcmMvc3lzL2Ft ZDY0L2FtZDY0L2V4Y2VwdGlvbi5TOjM4Nw0KPiCaIzE2IDB4MDAwMDAwMDgwMGI1NzExYyBpbiA/ PyAoKQ0KPiCaUHJldmlvdXMgZnJhbWUgaW5uZXIgdG8gdGhpcyBmcmFtZSAoY29ycnVwdCBzdGFj az8pDQo+IJooa2dkYikNCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCacHMgLWF4bA0KPg0KPiCa mppVSUQgmppQSUQgmlBQSUQgQ1BVIFBSSSBOSSCamppWU1ogmpqaUlNTIE1XQ0hBTiBTVEFUIJpU VCCampqaVElNRSBDT01NQU5EDQo+IJqampqaMCCampqaMCCampqaMCCamjAgLTUyIJowIJqampqa MCCampqamjAgLSCampqamkRMcyCamj8/IJowOjAwLjAwIFtrZXJuZWxdDQo+IJqampqaMCCampqa MSCampqaMCCamjAgmjIwIJowIJqaNjI4MCCampqamjAgd2FpdCCamkRMcyCamj8/IJowOjAwLjAw IFtpbml0XQ0KPiCampqamjAgmpqamjIgmpqamjAgmpowIC0xNiCaMCCampqamjAgmpqampowIHdh aXRpbiBETCCampo/PyCaMDowMC4wMCBbc2N0cF9pdGVyYQ0KPiCampqamjAgmpqamjMgmpqamjAg mpowIC0xNiCaMCCampqamjAgmpqampowIGNjYl9zYyBETCCampo/PyCaMDowMC4wMCBbeHB0X3Ro cmRdDQo+IJqampqaMCCampqaNCCampqaMCCamjAgLTE2IJowIJqampqaMCCampqamjAgcHNsZWVw IERMIJqamj8/IJowOjAwLjAwIFtwYWdlZGFlbW9uDQo+IJqampqaMCCampqaNSCampqaMCCamjAg LTE2IJowIJqampqaMCCampqamjAgcHNsZWVwIERMIJqamj8/IJowOjAwLjAwIFt2bWRhZW1vbl0N Cj4gmpqampowIJqampo2IJqampowIJqaMCAxNTUgmjAgmpqampowIJqampqaMCBwZ3plcm8gREwg mpqaPz8gmjA6MDAuMDAgW3BhZ2V6ZXJvXQ0KPiCampqamjAgmpqamjcgmpqamjAgmpowIC0xNiCa MCCampqamjAgmpqampowIHBzbGVlcCBETCCampo/PyCaMDowMC4wMCBbYnVmZGFlbW9uXQ0KPiCa mpqamjAgmpqamjggmpqamjAgmpowIC0xNiCaMCCampqamjAgmpqampowIHZscnV3dCBETCCampo/ PyCaMDowMC4wMCBbdm5scnVdDQo+IJqampqaMCCampqaOSCampqaMCCamjAgmjE2IJowIJqampqa MCCampqamjAgc3luY2VyIERMIJqamj8/IJowOjAwLjAwIFtzeW5jZXJdDQo+IJqampqaMCCampox MCCampqaMCCamjAgLTE2IJowIJqampqaMCCampqamjAgYXVkaXRfIERMIJqamj8/IJowOjAwLjAw IFthdWRpdF0NCj4gmpqampowIJqamjExIJqampowIJqaMCAxNTUgmjAgmpqampowIJqampqaMCAt IJqampqaUkwgmpqaPz8gmjM6MTQuNTIgW2lkbGVdDQo+IJqampqaMCCampoxMiCampqaMCCamjAg LTg0IJowIJqampqaMCCampqamjAgLSCampqamldMIJqamj8/IJowOjAwLjIyIFtpbnRyXQ0KPiCa mpqamjAgmpqaMTMgmpqamjAgmpowIJotOCCaMCCampqamjAgmpqampowIC0gmpqamppETCCampo/ PyCaMDowMC4wNiBbZ2VvbV0NCj4gmpqampowIJqamjE0IJqampowIJqaMCAtMTYgmjAgmpqampow IJqampqaMCAtIJqampqaREwgmpqaPz8gmjA6MDAuMDMgW3lhcnJvd10NCj4gmpqampowIJqamjE1 IJqampowIJqaMCAtNjggmjAgmpqampowIJqampqaMCAtIJqampqaREwgmpqaPz8gmjA6MDAuMDAg W3VzYl0NCj4gmpqampowIJqamjE2IJqampowIJqaMCAtMTYgmjAgmpqampowIJqampqaMCBzZGZs dXMgREwgmpqaPz8gmjA6MDAuMDAgW3NvZnRkZXBmbHUNCj4gmpqampowIJqaMTA3IJqampoxIJqa MCCaNTIgmjAgmjEwMDYwIJqampqaMCBwYXVzZSCaRHMgmpqaPz8gmjA6MDAuMDAgW2Fkamtlcm50 el0NCj4gmpqampowIJqaNTYwIJqampoxIJqaMCCaMjAgmjAgmjEwMzcyIJqampqaMCBzZWxlY3Qg RHMgmpqaPz8gmjA6MDAuMDAgW2RldmRdDQo+IJqampqaMCCamjkzOCCampqaMSCamjAgmjIwIJow IJoyMDM4NCCampqamjAgc2VsZWN0IERzIJqamj8/IJowOjAwLjAwIFtzZW5kbWFpbF0NCj4gmpqa mjI1IJqaOTQxIJqampoxIJqaMCCaNTIgmjAgmjIwMzg0IJqampqaMCBwYXVzZSCaRHMgmpqaPz8g mjA6MDAuMDAgW3NlbmRtYWlsXQ0KPiCampqamjAgmpo5NDcgmpqamjEgmpowIJoyMCCaMCCaMTQy NjAgmpqampowIG5hbnNscCBEcyCampo/PyCaMDowMC4wMCBbY3Jvbl0NCj4gmpqampowIJoxMDA5 IJqampoxIJqaMCCaNTIgmjAgmjQxMzAwIJqampqaMCB3YWl0IJqaRHMgmpqaPz8gmjA6MDAuMDEg W2xvZ2luXQ0KPiCampqamjAgmjEwMTAgmpqamjEgmpowIJo1MiCaMCCaMTIxODQgmpqampowIHR0 eWluIJpEcysgmpo/PyCaMDowMC4wMCBbZ2V0dHldDQo+IJqampqaMCCaMTAxMSCampqaMSCamjAg mjUyIJowIJoxMjE4NCCampqamjAgdHR5aW4gmkRzKyCamj8/IJowOjAwLjAwIFtnZXR0eV0NCj4g mpqampowIJoxMDEyIJqampoxIJqaMCCaNTIgmjAgmjEyMTg0IJqampqaMCB0dHlpbiCaRHMrIJqa Pz8gmjA6MDAuMDAgW2dldHR5XQ0KPiCampqamjAgmjEwMTMgmpqamjEgmpowIJo1MiCaMCCaMTIx ODQgmpqampowIHR0eWluIJpEcysgmpo/PyCaMDowMC4wMCBbZ2V0dHldDQo+IJqampqaMCCaMTAx NCCampqaMSCamjAgmjUyIJowIJoxMjE4NCCampqamjAgdHR5aW4gmkRzKyCamj8/IJowOjAwLjAw IFtnZXR0eV0NCj4gmpqampowIJoxMDE1IJqampoxIJqaMCCaNTIgmjAgmjEyMTg0IJqampqaMCB0 dHlpbiCaRHMrIJqaPz8gmjA6MDAuMDAgW2dldHR5XQ0KPiCampqamjAgmjEwMTYgmpqamjEgmpow IJo1MiCaMCCaMTIxODQgmpqampowIHR0eWluIJpEcysgmpo/PyCaMDowMC4wMCBbZ2V0dHldDQo+ IJqampqaMCCaMTAxNyCaMTAwOSCamjAgmjIwIJowIJoxNzY2NCCampqamjAgcGF1c2UgmkQgmpqa mj8/IJowOjAwLjAyIFtjc2hdDQo+IJqampqaMCCaMTA2NiCaMTAxNyCamjAgmjUyIJowIJoxNDYz NiCampqamjAgd2FpdCCamkQrIJqamj8/IJowOjAwLjAxIFtzaF0NCj4gmpqampowIJoxMDc4IJox MDY2IJqaMCCaNTIgmjAgmjE0NjM2IJqampqaMCB3YWl0IJqaRCsgmpqaPz8gmjA6MDAuMDAgW3No XQ0KPiCampqamjAgmjEwODMgmjEwNzggmpowIJo3MiCaMCCaMTIxMzYgmpqampowIC0gmpqamppS KyCampo/PyCaMDowMC4wMCBbaXBuYXRdDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmnZtc3Rh dCAtcw0KPg0KPiCampqamjQ1MzgyIGNwdSBjb250ZXh0IHN3aXRjaGVzDQo+IJqampqamjI1ODUg ZGV2aWNlIGludGVycnVwdHMNCj4gmpqampoxMTI4NCBzb2Z0d2FyZSBpbnRlcnJ1cHRzDQo+IJqa mpqaOTA4MDQgdHJhcHMNCj4gmpqamjEzNTE0OSBzeXN0ZW0gY2FsbHMNCj4gmpqampqampoxNiBr ZXJuZWwgdGhyZWFkcyBjcmVhdGVkDQo+IJqampqamjEwNTAgmmZvcmsoKSBjYWxscw0KPiCampqa mpqamjE3IHZmb3JrKCkgY2FsbHMNCj4gmpqampqampqaMCByZm9yaygpIGNhbGxzDQo+IJqampqa mpqamjAgc3dhcCBwYWdlciBwYWdlaW5zDQo+IJqampqampqamjAgc3dhcCBwYWdlciBwYWdlcyBw YWdlZCBpbg0KPiCampqampqampowIHN3YXAgcGFnZXIgcGFnZW91dHMNCj4gmpqampqampqaMCBz d2FwIHBhZ2VyIHBhZ2VzIHBhZ2VkIG91dA0KPiCampqampqaMjMyIHZub2RlIHBhZ2VyIHBhZ2Vp bnMNCj4gmpqampqaMTU5NiB2bm9kZSBwYWdlciBwYWdlcyBwYWdlZCBpbg0KPiCampqampqampow IHZub2RlIHBhZ2VyIHBhZ2VvdXRzDQo+IJqampqampqamjAgdm5vZGUgcGFnZXIgcGFnZXMgcGFn ZWQgb3V0DQo+IJqampqampqamjAgcGFnZSBkYWVtb24gd2FrZXVwcw0KPiCampqampqampowIHBh Z2VzIGV4YW1pbmVkIGJ5IHRoZSBwYWdlIGRhZW1vbg0KPiCampqampqaMjA2IHBhZ2VzIHJlYWN0 aXZhdGVkDQo+IJqampqaMzU2ODggY29weS1vbi13cml0ZSBmYXVsdHMNCj4gmpqampqamjM1NCBj b3B5LW9uLXdyaXRlIG9wdGltaXplZCBmYXVsdHMNCj4gmpqampoyNjcyOSB6ZXJvIGZpbGwgcGFn ZXMgemVyb2VkDQo+IJqampqampqamjAgemVybyBmaWxsIHBhZ2VzIHByZXplcm9lZA0KPiCampqa mpqampowIGludHJhbnNpdCBibG9ja2luZyBwYWdlIGZhdWx0cw0KPiCampqamjg0NzM5IHRvdGFs IFZNIGZhdWx0cyB0YWtlbg0KPiCampqampqampowIHBhZ2VzIGFmZmVjdGVkIGJ5IGtlcm5lbCB0 aHJlYWQgY3JlYXRpb24NCj4gmpqamjUzMzg4MiBwYWdlcyBhZmZlY3RlZCBieSCaZm9yaygpDQo+ IJqampqamjgxMjIgcGFnZXMgYWZmZWN0ZWQgYnkgdmZvcmsoKQ0KPiCampqampqampowIHBhZ2Vz IGFmZmVjdGVkIGJ5IHJmb3JrKCkNCj4gmpqampqampqaMCBwYWdlcyBjYWNoZWQNCj4gmpqampo5 MTUxOSBwYWdlcyBmcmVlZA0KPiCampqampqampowIHBhZ2VzIGZyZWVkIGJ5IGRhZW1vbg0KPiCa mpqampqampowIHBhZ2VzIGZyZWVkIGJ5IGV4aXRpbmcgcHJvY2Vzc2VzDQo+IJqampqamjI0ODMg cGFnZXMgYWN0aXZlDQo+IJqampqamjE2NDUgcGFnZXMgaW5hY3RpdmUNCj4gmpqampqampoxNSBw YWdlcyBpbiBWTSBjYWNoZQ0KPiCampqampo3NDU0IHBhZ2VzIHdpcmVkIGRvd24NCj4gmpqamjIz Nzk0OCBwYWdlcyBmcmVlDQo+IJqampqamjQwOTYgYnl0ZXMgcGVyIHBhZ2UNCj4gmpqampoxNTI2 NCB0b3RhbCBuYW1lIGxvb2t1cHMNCj4gmpqampqampqamppjYWNoZSBoaXRzICg4NSUgcG9zICsg NyUgbmVnKSBzeXN0ZW0gMCUgcGVyLWRpcmVjdG9yeQ0KPiCampqampqampqammRlbGV0aW9ucyAw JSwgZmFsc2VoaXRzIDAlLCB0b29sb25nIDAlDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmnZt c3RhdCAtbQ0KPg0KPiCampqampqampqaVHlwZSBJblVzZSBNZW1Vc2UgSGlnaFVzZSBSZXF1ZXN0 cyCaU2l6ZShzKQ0KPiCampqampqaYXRhX3BjaSCampqaMSCampqaMUsgmpqampqaLSCampqampqa MSCaNjQNCj4gmpqampqamppmZWVkZXIgmpqaMTIgmpqamjFLIJqampqami0gmpqampqaMTQgmjMy LDEyOA0KPiCampqampqammlzYWRldiCampqaOCCampqaMUsgmpqampqaLSCampqampqaOCCaMTI4 DQo+IJqampqampqamppjZGV2IJqampo4IJqampoySyCampqampotIJqampqampo4IJoyNTYNCj4g mpqampqammFjcGlkZXYgmpqaMjQgmpqamjJLIJqampqami0gmpqampqaMjQgmjY0DQo+IJqampqa mpqamm1peGVyIJqampoxIJqampo0SyCampqampotIJqampqampoxIJo0MDk2DQo+IJqampqammZp bGVkZXNjIJqamjM0IJqamjE3SyCampqampotIJqampoxMDg0IJo1MTINCj4gmpqampqampqammtl bnYgmpqaOTkgmpqaMTJLIJqampqami0gmpqampoxMDcgmjE2LDMyLDY0LDEyOA0KPiCampqampqa mmtxdWV1ZSCampqaMCCampqaMEsgmpqampqaLSCampqampoxMCCaMjU2DQo+IJqampqacHJvYy1h cmdzIJqamjE3IJqampoxSyCampqampotIJqampqaNDA0IJoxNiwzMiw2NCwxMjgsMjU2DQo+IJqa mpqampqammhob29rIJqampoyIJqampoxSyCampqampotIJqampqampoyIJoxMjgNCj4gmpqampqa mml0aHJlYWQgmpqaNjAgmpqaMTFLIJqampqami0gmpqampqaNjAgmjMyLDEyOCwyNTYNCj4gmpqa mpqamppLVFJBQ0UgmpoxMDAgmpqaMTNLIJqampqami0gmpqampoxMDAgmjEyOA0KPiCampqampqa mmxpbmtlciCamjE2NyCamjE4N0sgmpqampqaLSCampqamjE3NSCaMTYsMzIsNjQsMTI4LDI1Niw1 MTIsMTAyNCwyMDQ4LDQwOTYNCj4gmpqampqampqabG9ja2YgmpqaMTQgmpqamjJLIJqampqami0g mpqampqaMzggmjY0LDEyOA0KPiCampqabG9naW5jbGFzcyCampqaMiCampqaMUsgmpqampqaLSCa mpqampqaMyCaNjQNCj4gmj4gmpqampqammlwNm5kcCCampqaNCCampqaMUsgmpqampqaLSCampqa mpqaNCCaNjQsMTI4DQo+IJqampqampqampp0ZW1wIJqamjMxIJqaMjQ4SyCampqampotIJqampo2 MDI0IJoxNiwzMiw2NCwxMjgsMjU2LDUxMiwxMDI0LDIwNDgsNDA5Ng0KPiCampqampqammRldmJ1 ZiCamjQwMCCaMTMxNUsgmpqampqaLSCampqamjQxMyCaMTYsMzIsNjQsMTI4LDI1Niw1MTIsMTAy NCwyMDQ4LDQwOTYNCj4gmpqampqampptb2R1bGUgmpo0NDYgmpqaNTZLIJqampqami0gmpqampo0 NDYgmjEyOA0KPiCampqampptdHhfcG9vbCCampqaMiCampoxNksgmpqampqaLSCampqampqaMg0K PiCampqampqac3VicHJvYyCampo3NCCamjE1NksgmpqampqaLSCampqaMTEyNCCaNTEyLDQwOTYN Cj4gmpqampqampqamnByb2MgmpqamjIgmpqaMTZLIJqampqami0gmpqampqamjINCj4gmpqampqa mnNlc3Npb24gmpqaMTQgmpqamjJLIJqampqami0gmpqampqaMTYgmjEyOA0KPiCampqampqampqa cGdycCCampoxNiCampqaMksgmpqampqaLSCampqampoyOSCaMTI4DQo+IJo+IJqampqampqammNy ZWQgmpqaMjQgmpqamjRLIJqampqami0gmpqamjI5OTQgmjY0LDI1Ng0KPiCampqampqadWlkaW5m byCampqaMyCampqaM0sgmpqampqaLSCampqampqaNCCaMTI4LDIwNDgNCj4gmpqampqamppwbGlt aXQgmpqaMTMgmpqamjRLIJqampqami0gmpqampoxNjIgmjI1Ng0KPiCampqamnN5c2N0bHRtcCCa mpqaMCCampqaMEsgmpqampqaLSCampqamjI3MSCaMTYsMzIsNjQsMTI4LDQwOTYNCj4gmpqamppz eXNjdGxvaWQgmjE4MTggmpqaOTBLIJqampqami0gmpqamjE4NTIgmjE2LDMyLDY0LDEyOA0KPiCa mpqampqamnN5c2N0bCCampqaMCCampqaMEsgmpqampqaLSCampqamjQ0MiCaMTYsMzIsNjQNCj4g mpqampqamnRpZGhhc2ggmpqamjEgmpqaMTZLIJqampqami0gmpqampqamjENCj4gmpqampqampqa mnVtdHggmpoxNDQgmpqaMThLIJqampqami0gmpqampoxNDQgmjEyOA0KPiCampqamppwMTAwMy4x YiCampqaMSCampqaMUsgmpqampqaLSCampqampqaMSCaMTYNCj4gmpqampqampqamlNXQVAgmpqa mjIgmpozNDVLIJqampqami0gmpqampqamjIgmjY0DQo+IJqampqampqaYnVzLXNjIJqamjMwIJqa mjUySyCampqampotIJqampoxMjM3IJoxNiwzMiwxMjgsMjU2LDUxMiwxMDI0LDIwNDgsNDA5Ng0K PiCampqampqampqammJ1cyCamjkxMiCampo3NUsgmpqampqaLSCampqaMjc3MCCaMTYsMzIsNjQs MTI4LDI1NiwxMDI0DQo+IJqampqamppkZXZzdGF0IJqampo2IJqamjEzSyCampqampotIJqampqa mpo2IJozMiw0MDk2DQo+IJqaZXZlbnRoYW5kbGVyIJqamjg4IJqampo4SyCampqampotIJqampqa mjg4IJo2NCwxMjgNCj4gmpqampqacGNpX2xpbmsgmpqamjggmpqamjFLIJqampqami0gmpqampqa mjggmjE2LDEyOA0KPiCampqampqampqaa29iaiCamjMyMiCaMTI4OEsgmpqampqaLSCampqamjQw NSCaNDA5Ng0KPiCampqampqaUGVyLWNwdSCampqaMSCampqaMUsgmpqampqaLSCampqampqaMSCa MzINCj4gmpqampqampqamnJtYW4gmpqaODkgmpqaMTFLIJqampqami0gmpqampoyNTQgmjMyLDEy OA0KPiCampqampqampqac2J1ZiCampqaMCCampqaMEsgmpqampqaLSCampqamjU4MCCaMTYsMzIs NjQsMTI4LDI1Niw1MTIsMTAyNCwyMDQ4LDQwOTYNCj4gmpqampqampqac3RhY2sgmpqamjAgmpqa mjBLIJqampqami0gmpqampqamjIgmjI1Ng0KPiCampqamnRhc2txdWV1ZSCampoxNyCampqaMksg mpqampqaLSCampqampoxNyCaMTYsMzIsMTI4DQo+IJqampqampqaVW5pdG5vIJqamjEyIJqampox SyCampqampotIJqampoyNjAwIJozMiw2NA0KPiCampqampqampqammlvdiCampqaMCCampqaMEsg mpqampqaLSCampqampo2MSCaNjQsMTI4LDI1Niw1MTINCj4gmpqampqamppzZWxlY3QgmpqamjMg mpqamjFLIJqampqami0gmpqampqamjMgmjEyOA0KPiCampqampppb2N0bG9wcyCampqaMSCampqa MUsgmpqampqaLSCampqamjgwNSCaMTYsMzIsNjQsMTI4LDI1Niw1MTIsMTAyNA0KPiCampqampqa mpqamm1zZyCampqaNCCampozMEsgmpqampqaLSCampqampqaNCCaMjA0OCw0MDk2DQo+IJqampqa mpqampqac2VtIJqampo0IJqaMTA2SyCampqampotIJqampqampo0IJoyMDQ4LDQwOTYNCj4gmpqa mpqampqamppzaG0gmpqamjEgmpqaMjBLIJqampqami0gmpqampqamjENCj4gmpqampqampqampp0 dHkgmpqaMTkgmpqaMTlLIJqampqami0gmpqampqaMjEgmjEwMjQsMjA0OA0KPiCampqampptYnVm X3RhZyCampqaMCCampqaMEsgmpqampqaLSCampqampqaNiCaMzINCj4gmpqampqampqac2htZmQg mpqamjEgmpqamjhLIJqampqami0gmpqampqamjENCj4gmpqampqampqamppwY2IgmpqaMTMgmpox NTdLIJqampqami0gmpqampqaMTcgmjE2LDMyLDEyOCwxMDI0LDIwNDgsNDA5Ng0KPiCampqampqa mnNvbmFtZSCampqaMSCampqaMUsgmpqampqaLSCampqamjEwMyCaMTYsMzIsMTI4DQo+IJqampqa mnZmc2NhY2hlIJqampoxIJoxMDI0SyCampqampotIJqampqampoxDQo+IJqampqamnZmc19oYXNo IJqampoxIJqaNTEySyCampqampotIJqampqampoxDQo+IJqampqammFjcGlpbnRyIJqampoxIJqa mpoxSyCampqampotIJqampqampoxIJo2NA0KPiCampqampqamnZub2RlcyCampqaMiCampqaMUsg mpqampqaLSCampqampqaMiCaMjU2DQo+IJqampqampqaYWNwaWNhIJoyMDc0IJqaMjMzSyCampqa mpotIJqamjIyMzIxIJoxNiwzMiw2NCwxMjgsMjU2LDEwMjQsNDA5Ng0KPiCampp2bm9kZW1hcmtl ciCampqaMCCampqaMEsgmpqampqaLSCampqampoyMyCaNTEyDQo+IJqampqampqamm1vdW50IJqa mjE2IJqampoxSyCampqampotIJqampqamjg2IJoxNiwzMiw2NCwxMjgsMjU2DQo+IJqampqampqa mpqaQlBGIJqampozIJqampoxSyCampqampotIJqampqampozIJoxMjgNCj4gmpqaZXRoZXJfbXVs dGkgmpqaMTAgmpqamjFLIJqampqami0gmpqampqaMTAgmjE2LDMyLDY0DQo+IJqampqampqaaWZh ZGRyIJqamjM4IJqamjExSyCampqampotIJqampqamjM4IJozMiw2NCwxMjgsMjU2LDUxMiw0MDk2 DQo+IJqampqampqammlmbmV0IJqampo0IJqampo3SyCampqampotIJqampqampo0IJoxMjgsMjA0 OA0KPiCampqampqamppjbG9uZSCampqaNiCampoyNEsgmpqampqaLSCampqampqaNiCaNDA5Ng0K PiCampqampqammFycGNvbSCampqaMSCampqaMUsgmpqampqaLSCampqampqaMSCaMTYNCj4gmpqa mpqammxsdGFibGUgmpqamjggmpqamjRLIJqampqami0gmpqampqamjggmjI1Niw1MTINCj4gmpqa mpqamppVU0JkZXYgmpqamjQgmpqamjJLIJqampqami0gmpqampqamjQgmjY0LDEyOCwxMDI0DQo+ IJqampqampqampqaVVNCIJqampo3IJqampozSyCampqampotIJqampqampo3IJoxNiwzMiwxMjgs MjA0OA0KPiCampqampphY3BpdGFzayCampqaMSCampqaMksgmpqampqaLSCampqampqaMSCaMjA0 OA0KPiCaQ0FNIGRldiBxdWV1ZSCampqaMyCampqaMUsgmpqampqaLSCampqampqaMyCaMTI4DQo+ IJqampqaQ0FNIHF1ZXVlIJqamjEzIJqampoxSyCampqampotIJqampqamjUxIJoxNiwzMg0KPiCa mpqamppyb3V0ZXRibCCampoyNCCampqaNEsgmpqampqaLSCampqampo2OSCaMzIsNjQsMTI4LDI1 Niw1MTINCj4gmnZuZXRfZGF0YV9mcmVlIJqampoxIJqampoxSyCampqampotIJqampqampoxIJoz Mg0KPiCampqamnZuZXRfZGF0YSCampqaMSCampoyOEsgmpqampqaLSCampqampqaMQ0KPiCampqa mpqampqadm5ldCCampqaMSCampqaMUsgmpqampqaLSCampqampqaMSCaNjQNCj4gmpqampqampqa mmlnbXAgmpqamjMgmpqamjFLIJqampqami0gmpqampqamjMgmjI1Ng0KPiCampqampqaYWNwaXNl bSCampoxNiCampqaMksgmpqampqaLSCampqampoxNiCaMTI4DQo+IJqampqamppDQU0gU0lNIJqa mpozIJqampoxSyCampqampotIJqampqampozIJoyNTYNCj4gmpqampqaaW5fbXVsdGkgmpqamjEg mpqamjFLIJqampqami0gmpqampqamjEgmjI1Ng0KPiCampqamnNjdHBfaXRlciCampqaMCCampqa MEsgmpqampqaLSCampqampqaMSCaMjU2DQo+IJqampqamnNjdHBfaWZuIJqampoxIJqampoxSyCa mpqampotIJqampqampoxIJoxMjgNCj4gmpqampqac2N0cF9pZmEgmpqamjMgmpqamjFLIJqampqa mi0gmpqampqamjMgmjEyOA0KPiCampqamppzY3RwX3ZyZiCampqaMSCampqaMUsgmpqampqaLSCa mpqampqaMSCaNjQNCj4gmpqamppzY3RwX2FfaXQgmpqamjAgmpqamjBLIJqampqami0gmpqampqa mjEgmjE2DQo+IJqampqaaG9zdGNhY2hlIJqampoxIJqamjI4SyCampqampotIJqampqampoxDQo+ IJqampqamnN5bmNhY2hlIJqampoxIJqamjk2SyCampqampotIJqampqampoxDQo+IJqampqamppz Y3NpX2NkIJqampowIJqampowSyCampqampotIJqampqampo0IJoxNg0KPiCampqammluNl9tdWx0 aSCampoxMiCampqaMksgmpqampqaLSCampqampoxMiCaMzIsMjU2DQo+IJqampqampplbnRyb3B5 IJoxMDI0IJqamjY0SyCampqampotIJqampoxMDI0IJo2NA0KPiCampqaQ0FNIHBlcmlwaCCampqa NiCampqaMksgmpqampqaLSCampqampoyMCCaMTYsMzIsNjQsMTI4LDI1Ng0KPiCampqampqampqa mm1sZCCampqaMyCampqaMUsgmpqampqaLSCampqampqaMyCaMTI4DQo+IJqampqampqampqacnBj IJqampoyIJqampoxSyCampqampotIJqampqampoyIJoyNTYNCj4gmmF1ZGl0X2V2Y2xhc3Mgmpox NzkgmpqamjZLIJqampqami0gmpqampoyMTggmjMyDQo+IJqampqamppqYmxvY2tzIJqampoyIJqa mpoxSyCampqampotIJqampqampoyIJoxMjgsMjU2DQo+IJqampqamnNhdmVkaW5vIJqampowIJqa mpowSyCampqampotIJqampqampo4IJoyNTYNCj4gmpqampqampqac2JkZXAgmpqamjAgmpqamjBL IJqampqami0gmpqampqamjcgmjY0DQo+IJqampqamppqc2VnZGVwIJqampoxIJqampoxSyCampqa mpotIJqampqamjg3IJo2NA0KPiCampqampqampqaanNlZyCampqaMSCampqaMUsgmpqampqaLSCa mpqampqaNiCaMTI4DQo+IJqampqamppqbmV3YmxrIJqampowIJqampowSyCampqampotIJqampqa mjE3IJoxMjgNCj4gmpqampqammpyZW1yZWYgmpqamjAgmpqamjBLIJqampqami0gmpqampqaMzQg mjEyOA0KPiCampqampqaamFkZHJlZiCampqaMCCampqaMEsgmpqampqaLSCampqampozNiCaMTI4 DQo+IJqampqammZyZWV3b3JrIJqampoxIJqampoxSyCampqampotIJqampqamjE1IJoxMjgNCj4g mpqamppuZXdkaXJibGsgmpqamjAgmpqamjBLIJqampqami0gmpqampqamjYgmjY0DQo+IJqampqa mpqaZGlycmVtIJqampowIJqampowSyCampqampotIJqampqamjIyIJoxMjgNCj4gmpqampqampqa bWtkaXIgmpqamjAgmpqamjBLIJqampqami0gmpqampqaMTIgmjEyOA0KPiCampqampqammRpcmFk ZCCampqaMCCampqaMEsgmpqampqaLSCampqampoyNCCaMTI4DQo+IJqampqammZyZWVmaWxlIJqa mpowIJqampowSyCampqampotIJqampqamjE5IJo2NA0KPiCampqamppmcmVlYmxrcyCampqaMCCa mpqaMEsgmpqampqaLSCampqampoxNCCaMTI4DQo+IJqampqampqabmV3YmxrIJqampoyIJqamjY1 SyCampqampotIJqampqamjE4IJoyNTYNCj4gmpqamppibXNhZmVtYXAgmpqamjEgmpqamjhLIJqa mpqami0gmpqampqamjkgmjI1Ng0KPiCampqampppbm9kZWRlcCCampqaMiCamjUxM0sgmpqampqa LSCampqampo0NCCaNTEyDQo+IJqampqamppwYWdlZGVwIJqampoxIJqamjY0SyCampqampotIJqa mpqamjE2IJoyNTYNCj4gmpqadWZzX2Rpcmhhc2ggmpqaMzAgmpqamjZLIJqampqami0gmpqampqa MzAgmjE2LDMyLDY0LDEyOCw1MTINCj4gmpqampp1ZnNfbW91bnQgmpqamjMgmpqaMTNLIJqampqa mi0gmpqampqamjMgmjUxMiw0MDk2DQo+IJqampqadm1fcGdkYXRhIJqampoyIJqaMTI5SyCampqa mpotIJqampqampoyIJoxMjgNCj4gmpqampqamkNBTSBYUFQgmpqaMzYgmpqaMTZLIJqampqami0g mpqampqaOTQgmjMyLDY0LDEyOCwxMDI0LDIwNDgNCj4gmpqampqampprYmRtdXggmpqamjYgmpqa MThLIJqampqami0gmpqampqamjYgmjE2LDUxMiwxMDI0LDIwNDgNCj4gmpqampqamppERVZGUzEg mpqaODIgmpqaNDFLIJqampqami0gmpqampqaODcgmjUxMg0KPiCampqampqamkRFVkZTMyCampo5 OCCampoyNUsgmpqampqaLSCampqamjExMCCaMjU2DQo+IJqampqampqamkRFVkZTIJqamjE0IJqa mpoxSyCampqampotIJqampqamjE1IJoxNiwxMjgNCj4gmpqampqaYXRrYmRkZXYgmpqamjIgmpqa mjFLIJqampqami0gmpqampqamjIgmjY0DQo+IJqampqampqaYXBtZGV2IJqampoxIJqampoxSyCa mpqampotIJqampqampoxIJoxMjgNCj4gmpqamppwZnNfbm9kZXMgmpqaMjEgmpqamjZLIJqampqa mi0gmpqampqaMjEgmjI1Ng0KPiCampqampqaaW9fYXBpYyCampqaMSCampqaMksgmpqampqaLSCa mpqampqaMSCaMjA0OA0KPiCampqampqampqamkxFRCCampqaMiCampqaMUsgmpqampqaLSCampqa mpqaMiCaMTYsMTI4DQo+IJqampqampqamppHRU9NIJqamjUxIJqamjExSyCampqampotIJqampqa MzY5IJoxNiwzMiw2NCwxMjgsMjU2LDUxMiwxMDI0LDIwNDgNCj4gmpqampqabmV4dXNkZXYgmpqa mjMgmpqamjFLIJqampqami0gmpqampqamjMgmjE2DQo+IJqampqampqampphYzk3IJqampoyIJqa mpoxSyCampqampotIJqampqampoyIJoxNiw1MTINCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCa dm1zdGF0IC16DQo+DQo+IJpJVEVNIJqampqampqampqampqampqamlNJWkUgmkxJTUlUIJqamppV U0VEIJqamppGUkVFIJqampqaUkVRIEZBSUwgU0xFRVANCj4NCj4gmlVNQSBLZWdzOiCampqampqa mpqampqamjIwOCwgmpqampowLCCampqamjg0LCCampqampoxLCCampqamjg0LCCamjAsIJqaMA0K PiCaVU1BIFpvbmVzOiCampqampqampqampqaNTEyLCCampqamjAsIJqampqaODQsIJqampqamjAs IJqampqaODQsIJqaMCwgmpowDQo+IJpVTUEgU2xhYnM6IJqampqampqampqampo1NjgsIJqampqa MCwgmpqamjgzMiwgmpqampqaMSwgmpqaMTI4MSwgmpowLCCamjANCj4gmlVNQSBSQ250U2xhYnM6 IJqampqampqamjU2OCwgmpqampowLCCampqamjY3LCCampqampozLCCampqamjY3LCCamjAsIJqa MA0KPiCaVU1BIEhhc2g6IJqampqampqampqampqaMjU2LCCampqamjAsIJqampqamjMsIJqampqa MTIsIJqampqamjMsIJqaMCwgmpowDQo+IJoxNiBCdWNrZXQ6IJqampqampqampqampoxNTIsIJqa mpqaMCwgmpqampozOSwgmpqampoxMSwgmpqampozOSwgmpowLCCamjANCj4gmjMyIEJ1Y2tldDog mpqampqampqampqamjI4MCwgmpqampowLCCampqamjI0LCCampqampo0LCCampqamjI0LCCamjAs IJqaMA0KPiCaNjQgQnVja2V0OiCampqampqampqampqaNTM2LCCampqamjAsIJqampqaMTIsIJqa mpqamjIsIJqampqaMTIsIJo1NywgmpowDQo+IJoxMjggQnVja2V0OiCampqampqampqamjEwNDgs IJqampqaMCwgmpqampoxOSwgmpqampqaMiwgmpqampoxOSwgNTY5LCCamjANCj4gmlZNIE9CSkVD VDogmpqampqampqampqamjIxNiwgmpqampowLCCampqaNzI3LCCampqamjQ3LCCamjEzODc3LCCa mjAsIJqaMA0KPiCaTUFQOiCampqampqampqampqampqampqaMjMyLCCampqamjAsIJqampqamjcs IJqampqaMjUsIJqampqamjcsIJqaMCwgmpowDQo+IJpLTUFQIEVOVFJZOiCampqampqampqampox MjAsIJo1NTg2MiwgmpqampoyNCwgmpqamjEzMSwgmpqaMTM1NywgmpowLCCamjANCj4gmk1BUCBF TlRSWTogmpqampqampqampqamjEyMCwgmpqampowLCCampqaMzc1LCCampqamjkwLCCamjMwOTc0 LCCamjAsIJqaMA0KPiCaZmFrZXBnOiCampqampqampqampqampqaMTIwLCCampqamjAsIJqampqa mjAsIJqampqamjAsIJqampqamjAsIJqaMCwgmpowDQo+IJptdF96b25lOiCampqampqampqampqa mjQxMTIsIJqampqaMCwgmpqamjMwNCwgmpqampqaNywgmpqamjMwNCwgmpowLCCamjANCj4gmjE2 OiCampqampqampqampqampqampqampoxNiwgmpqampowLCCampqaOTg0LCCampqaMTkyLCCamjEx MjU4LCCamjAsIJqaMA0KPiCaMzI6IJqampqampqampqampqampqampqamjMyLCCampqamjAsIJqa mjEwNTYsIJqampoxNTYsIJqamjY5NzgsIJqaMCwgmpowDQo+IJo2NDogmpqampqampqampqampqa mpqampqaNjQsIJqampqaMCwgmpqaMjQ0MiwgmpqamjE5MCwgmpoxMzk0MiwgmpowLCCamjANCj4g mjEyODogmpqampqampqampqampqampqamjEyOCwgmpqampowLCCampozMzIwLCCampqamjczLCCa mpo0OTY2LCCamjAsIJqaMA0KPiCaMjU2OiCampqampqampqampqampqampqaMjU2LCCampqamjAs IJqampoyOTksIJqampqaMzEsIJqamjIyMzUsIJqaMCwgmpowDQo+IJo1MTI6IJqampqampqampqa mpqampqampo1MTIsIJqampqaMCwgmpqamjIyMywgmpqampoyOSwgmpqaMTgxOSwgmpowLCCamjAN Cj4gmjEwMjQ6IJqampqampqampqampqampqaMTAyNCwgmpqampowLCCampqamjQwLCCampqamjU2 LCCampoxNDc3LCCamjAsIJqaMA0KPiCaMjA0ODogmpqampqampqampqampqampoyMDQ4LCCampqa mjAsIJqampqaMjYsIJqampqamjgsIJqampoxMTYsIJqaMCwgmpowDQo+IJo0MDk2OiCampqampqa mpqampqampqamjQwOTYsIJqampqaMCwgmpqamjM5OCwgmpqampqaOSwgmpqaNjY5OSwgmpowLCCa mjANCj4gmkZpbGVzOiCampqampqampqampqampqampo4MCwgmpqampowLCCampqamjI2LCCampqa mjY0LCCampo0NDczLCCamjAsIJqaMA0KPiCaVFVSTlNUSUxFOiCampqampqampqampqaMTM2LCCa mpqamjAsIJqampqaNzMsIJqampqaMjcsIJqampqaNzMsIJqaMCwgmpowDQo+IJp1bXR4IHBpOiCa mpqampqampqampqampqaOTYsIJqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpow LCCamjANCj4gmk1BQyBsYWJlbHM6IJqampqampqampqampo0MCwgmpqampowLCCampqampowLCCa mpqampowLCCampqampowLCCamjAsIJqaMA0KPiCaUFJPQzogmpqampqampqampqampqampoxMTYw LCCampqamjAsIJqampqaMzMsIJqampqamjYsIJqamjEwODMsIJqaMCwgmpowDQo+IJpUSFJFQUQ6 IJqampqampqampqampqamjExMTIsIJqampqaMCwgmpqampo2NSwgmpqampqaNywgmpqampo2NSwg mpowLCCamjANCj4gmlNMRUVQUVVFVUU6IJqampqampqampqampo4MCwgmpqampowLCCampqamjcz LCCampqamjQzLCCampqamjczLCCamjAsIJqaMA0KPiCaVk1TUEFDRTogmpqampqampqampqampqa MzkyLCCampqamjAsIJqampqaMTgsIJqampqaMTIsIJqamjEwNjgsIJqaMCwgmpowDQo+IJpjcHVz ZXQ6IJqampqampqampqampqampqaNzIsIJqampqaMCwgmpqampqaMiwgmpqampo5OCwgmpqampqa MiwgmpowLCCamjANCj4gmmF1ZGl0X3JlY29yZDogmpqampqampqamjk2MCwgmpqampowLCCampqa mpowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCabWJ1Zl9wYWNrZXQ6IJqampqampqa mpqaMjU2LCCampqamjAsIJqampqamjAsIJqampoxNDIsIJqampqamjQsIJqaMCwgmpowDQo+IJpt YnVmOiCampqampqampqampqampqampoyNTYsIJqampqaMCwgmpqampqaMSwgmpqamjE0MSwgmpqa mpqaNSwgmpowLCCamjANCj4gmm1idWZfY2x1c3RlcjogmpqampqampqaMjA0OCwgmjI1NjAwLCCa mpqaMTI4LCCampqampo2LCCampqaMTI4LCCamjAsIJqaMA0KPiCabWJ1Zl9qdW1ib19wYWdlOiCa mpqampo0MDk2LCCaMTI4MDAsIJqampqamjAsIJqampqamjAsIJqampqamjAsIJqaMCwgmpowDQo+ IJptYnVmX2p1bWJvXzlrOiCampqampqamjkyMTYsIJoxOTIwMCwgmpqampqaMCwgmpqampqaMCwg mpqampqaMCwgmpowLCCamjANCj4gmm1idWZfanVtYm9fMTZrOiCampqampoxNjM4NCwgmjEyODAw LCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCabWJ1Zl9leHRfcmVmY250 OiCampqampqampo0LCCampqamjAsIJqampqamjAsIJqampqamjAsIJqampqamjAsIJqaMCwgmpow DQo+IJpnX2JpbzogmpqampqampqampqampqampoyMzIsIJqampqaMCwgmpqampqaMCwgmpqamjEy OCwgmpqaMzIwNSwgmpowLCCamjANCj4gmnR0eWlucTogmpqampqampqampqampqamjE2MCwgmpqa mpowLCCampqaMjg1LCCampqamjI3LCCampqaNDIwLCCamjAsIJqaMA0KPiCadHR5b3V0cTogmpqa mpqampqampqampqaMjU2LCCampqamjAsIJqampoxNDksIJqampqaMTYsIJqampoyMjEsIJqaMCwg mpowDQo+IJphdGFfcmVxdWVzdDogmpqampqampqampozMjgsIJqampqaMCwgmpqampqaMSwgmpqa mpoyMywgmpqaMjgzNCwgmpowLCCamjANCj4gmmF0YV9jb21wb3NpdGU6IJqampqampqamjMzNiwg mpqampowLCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCaVk5PREU6IJqa mpqampqampqampqampqaNDgwLCCampqamjAsIJqampo0NjksIJqampqaMTksIJqampo0OTEsIJqa MCwgmpowDQo+IJpWTk9ERVBPTEw6IJqampqampqampqampoxMTIsIJqampqaMCwgmpqampqaMCwg mpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmk5BTUVJOiCampqampqampqampqampqaMTAy NCwgmpqampowLCCampqampowLCCampqamjEyLCCampo3MTgzLCCamjAsIJqaMA0KPiCaUyBWRlMg Q2FjaGU6IJqampqampqampqaMTA4LCCampqamjAsIJqampo0NzIsIJqampqaMjMsIJqampo5NzYs IJqaMCwgmpowDQo+IJpMIFZGUyBDYWNoZTogmpqampqampqampozMjgsIJqampqaMCwgmpqampqa MCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmkRJUkhBU0g6IJqampqampqampqampqa MTAyNCwgmpqampowLCCampqamjQyLCCampqampo2LCCampqamjQyLCCamjAsIJqaMA0KPiCaTkNM Tk9ERTogmpqampqampqampqampqaNTYwLCCampqamjAsIJqampqamjAsIJqampqamjAsIJqampqa mjAsIJqaMCwgmpowDQo+IJpNb3VudHBvaW50czogmpqampqampqampo3NjgsIJqampqaMCwgmpqa mpqaMiwgmpqampqaOCwgmpqampqaMiwgmpowLCCamjANCj4gmnBpcGU6IJqampqampqampqampqa mpqamjcyOCwgmpqampowLCCampqampowLCCampqamjEwLCCampqaNjQzLCCamjAsIJqaMA0KPiCa a3NpZ2luZm86IJqampqampqampqampqaMTEyLCCampqamjAsIJqampqaMzIsIJqamjEwMjQsIJqa mpqaMzIsIJqaMCwgmpowDQo+IJppdGltZXI6IJqampqampqampqampqampozNDQsIJqampqaMCwg mpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmktOT1RFOiCampqampqampqa mpqampqamjEyOCwgmpqampowLCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0K PiCac29ja2V0OiCampqampqampqampqampqaNjgwLCCaMjU2MDIsIJqampqamjIsIJqampqaMTAs IJqampoxNDEsIJqaMCwgmpowDQo+IJp1bnBjYjogmpqampqampqampqampqampoyNDAsIJoyNTYw MCwgmpqampqaMSwgmpqampozMSwgmpqampoxNiwgmpowLCCamjANCj4gmmlwcTogmpqampqampqa mpqampqampqampo1NiwgmpqaODE5LCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqa MA0KPiCadT4gZHBfaW5wY2I6IJqampqampqampqampozOTIsIJoyNTYwMCwgmpqampqaMCwgmpqa mpoyMCwgmpqamjExOCwgmpowLCCamjANCj4gmnVkcGNiOiCampqampqampqampqampqampoxNiwg mjI1NzA0LCCampqampowLCCampqaMTY4LCCampqaMTE4LCCamjAsIJqaMA0KPiCadGNwX2lucGNi OiCampqampqampqampqaMzkyLCCaMjU2MDAsIJqampqamjEsIJqampqaMTksIJqampqamjMsIJqa MCwgmpowDQo+IJp0Y3BjYjogmpqampqampqampqampqampo5NzYsIJoyNTYwMCwgmpqampqaMSwg mpqampqaNywgmpqampqaMywgmpowLCCamjANCj4gmnRjcHR3OiCampqampqampqampqampqampo3 Miwgmpo1MTUwLCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCac3luY2Fj aGU6IJqampqampqampqampqaMTUyLCCaMTUzNzUsIJqampqamjAsIJqampqamjAsIJqampqamjAs IJqaMCwgmpowDQo+IJpob3N0Y2FjaGU6IJqampqampqampqampoxMzYsIJoxNTM3Miwgmpqampqa MCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmnRjcHJlYXNzOiCampqampqampqampqa mpo0MCwgmpoxNjgwLCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCac2Fj a2hvbGU6IJqampqampqampqampqamjMyLCCampqamjAsIJqampqamjAsIJqampqamjAsIJqampqa mjAsIJqaMCwgmpowDQo+IJpzY3RwX2VwOiCampqampqampqampqamjEzNjgsIJoyNTYwMCwgmpqa mpqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmnNjdHBfYXNvYzogmpqampqampqa mpqaMjI4MCwgmjQwMDAwLCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0KPiCa c2N0cF9sYWRkcjogmpqampqampqampqamjQ4LCCaODAwNjQsIJqampqamjAsIJqampoxNDQsIJqa mpqamjIsIJqaMCwgmpowDQo+IJpzY3RwX3JhZGRyOiCampqampqampqampo3MDQsIJo4MDAwMCwg mpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmnNjdHBfY2h1bms6IJqampqa mpqampqamjEzNiwgNDAwMDA4LCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqaMA0K PiCac2N0cF9yZWFkcTogmpqampqampqampqaMTA0LCA0MDAwMzIsIJqampqamjAsIJqampqamjAs IJqampqamjAsIJqaMCwgmpowDQo+IJpzY3RwX3N0cmVhbV9tc2dfb3V0OiCampoxMTIsIDQwMDAy NiwgmpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmnNjdHBfYXNjb25mOiCa mpqampqampqampo0MCwgNDAwMDA4LCCampqampowLCCampqampowLCCampqampowLCCamjAsIJqa MA0KPiCac2N0cF9hc2NvbmZfYWNrOiCampqampqamjQ4LCA0MDAwMzIsIJqampqamjAsIJqampqa mjAsIJqampqamjAsIJqaMCwgmpowDQo+IJpyaXBjYjogmpqampqampqampqampqampozOTIsIJoy NTYwMCwgmpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmnJ0ZW50cnk6IJqa mpqampqampqampqamjIwMCwgmpqampowLCCampqamjEwLCCampqamjI4LCCampqamjEwLCCamjAs IJqaMA0KPiCac2VsZmQ6IJqampqampqampqampqampqamjU2LCCampqamjAsIJqampqamjYsIJqa mpoxMjAsIJqampoxMzQsIJqaMCwgmpowDQo+IJpTV0FQTUVUQTogmpqampqampqampqampoyODgs IDExNjUxOSwgmpqampqaMCwgmpqampqaMCwgmpqampqaMCwgmpowLCCamjANCj4gmkZGUyBpbm9k ZTogmpqampqampqampqamjE2OCwgmpqampowLCCampqaNDQ5LCCampqamjM1LCCampqaNDY4LCCa mjAsIJqaMA0KPiCaRkZTMSBkaW5vZGU6IJqampqampqampqaMTI4LCCampqamjAsIJqampqamjAs IJqampqamjAsIJqampqamjAsIJqaMCwgmpowDQo+IJpGRlMyIGRpbm9kZTogmpqampqampqampoy NTYsIJqampqaMCwgmpqamjQ0OSwgmpqampoxNiwgmpqamjQ2OCwgmpowLCCamjANCj4NCj4gmi0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLQ0KPiCadm1zdGF0IC1pDQo+DQo+IJppbnRlcnJ1cHQgmpqampqampqampqa mpqampqampqampqamnRvdGFsIJqampqamnJhdGUNCj4gmmlycTE6IGF0a2JkMCCampqampqampqa mpqampqampqampqampo3MjIgmpqampqampo3Mg0KPiCaaXJxMTQ6IGF0YTAgmpqampqampqampqa mpqampqampqampqaMTg0NSCampqampqaMTg0DQo+IJppcnExNTogYXRhMSCampqampqampqampqa mpqampqampqampqamjE3IJqampqampqamjENCj4gmmNwdTA6dGltZXIgmpqampqampqampqampqa mpqampqampqamjk4Nzkgmpqampqamjk4Nw0KPiCaVG90YWwgmpqampqampqampqampqampqampqa mpqampqampoxMjQ2MyCampqampoxMjQ2DQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmnBzdGF0 IC1UDQo+DQo+IJqaMjYvMTIzMjggZmlsZXMNCj4gmjBNLzIyMjNNIHN3YXAgc3BhY2UNCj4NCj4g mi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLQ0KPiCacHN0YXQgLXMNCj4NCj4gmkRldmljZSCampqampqampo1MTIt YmxvY2tzIJqamppVc2VkIJqamkF2YWlsIENhcGFjaXR5DQo+IJovZGV2L2FkYTBwMyCampqampqa NDU1NDQ5NiCampqampqaMCCaNDU1NDQ5NiCampqaMCUNCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K PiCaaW9zdGF0DQo+DQo+IJppb3N0YXQ6IGt2bV9yZWFkKF90a19uaW4pOiBpbnZhbGlkIGFkZHJl c3MgKDB4MCkNCj4gmmlvc3RhdDogZGlzYWJsaW5nIFRUWSBzdGF0aXN0aWNzDQo+IJqampqampqa mpqampphZGEwIJqampqampqampqamppjZDAgmpqampqampqamppwYXNzMCCampqampqampqamppj cHUNCj4gmpqaS0IvdCB0cHMgmk1CL3MgmppLQi90IHRwcyCaTUIvcyCamktCL3QgdHBzIJpNQi9z IJp1cyBuaSBzeSBpbiBpZA0KPiCamjE4LjcwIJo5NiCaMS43NSCamjAuMDAgmpoxIJowLjAwIJqa MC4wMCCamjAgmjAuMDAgmpowIJowIJoxIJowIDk5DQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4g mmlwY3MgLWENCj4NCj4gmk1lc3NhZ2UgUXVldWVzOg0KPiCaVCCampqampqampqaSUQgmpqampqa mpqaS0VZIE1PREUgmpqampqamk9XTkVSIJqamkdST1VQIJqamkNSRUFUT1IgmkNHUk9VUCCampqa mpqampqampqampqaQ0JZVEVTIJqampqampqampqampqamppRTlVNIJqampqampqampqampqaUUJZ VEVTIJqampqamppMU1BJRCCampqampqaTFJQSUQgU1RJTUUgmpqaUlRJTUUgmpqaQ1RJTUUNCj4N Cj4gmlNoYXJlZCBNZW1vcnk6DQo+IJpUIJqampqampqamppJRCCampqampqamppLRVkgTU9ERSCa mpqampqaT1dORVIgmpqaR1JPVVAgmpqaQ1JFQVRPUiCaQ0dST1VQIJqampqampqaTkFUVENIIJqa mpqamppTRUdTWiCampqampqamkNQSUQgmpqampqamppMUElEIEFUSU1FIJqamkRUSU1FIJqamkNU SU1FDQo+DQo+IJpTZW1hcGhvcmVzOg0KPiCaVCCampqampqampqaSUQgmpqampqampqaS0VZIE1P REUgmpqampqamk9XTkVSIJqamkdST1VQIJqamkNSRUFUT1IgmkNHUk9VUCCampqampqamppOU0VN UyBPVElNRSCamppDVElNRQ0KPg0KPiCaLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IJppcGNzIC1UDQo+DQo+ IJptc2dpbmZvOg0KPiCampqampqampptc2dtYXg6IJqampqampoxNjM4NCAobWF4IGNoYXJhY3Rl cnMgaW4gYSBtZXNzYWdlKQ0KPiCampqampqampptc2dtbmk6IJqampqampqampo0MCAoIyBvZiBt ZXNzYWdlIHF1ZXVlcykNCj4gmpqampqampqabXNnbW5iOiCampqampqamjIwNDggKG1heCBjaGFy YWN0ZXJzIGluIGEgbWVzc2FnZSBxdWV1ZSkNCj4gmpqampqampqabXNndHFsOiCampqampqampqa NDAgKG1heCAjIG9mIG1lc3NhZ2VzIGluIHN5c3RlbSkNCj4gmpqampqampqabXNnc3N6OiCampqa mpqampqamjggKHNpemUgb2YgYSBtZXNzYWdlIHNlZ21lbnQpDQo+IJqampqampqamm1zZ3NlZzog mpqampqampoyMDQ4ICgjIG9mIG1lc3NhZ2Ugc2VnbWVudHMgaW4gc3lzdGVtKQ0KPg0KPiCac2ht aW5mbzoNCj4gmpqampqampqac2htbWF4OiCampo1MzY4NzA5MTIgKG1heCBzaGFyZWQgbWVtb3J5 IHNlZ21lbnQgc2l6ZSkNCj4gmpqampqampqac2htbWluOiCampqampqampqamjEgKG1pbiBzaGFy ZWQgbWVtb3J5IHNlZ21lbnQgc2l6ZSkNCj4gmpqampqampqac2htbW5pOiCampqampqampoxOTIg KG1heCBudW1iZXIgb2Ygc2hhcmVkIG1lbW9yeSBpZGVudGlmaWVycykNCj4gmpqampqampqac2ht c2VnOiCampqampqampoxMjggKG1heCBzaGFyZWQgbWVtb3J5IHNlZ21lbnRzIHBlciBwcm9jZXNz KQ0KPiCampqampqamppzaG1hbGw6IJqampqamjEzMTA3MiAobWF4IGFtb3VudCBvZiBzaGFyZWQg bWVtb3J5IGluIHBhZ2VzKQ0KPg0KPiCac2VtaW5mbzoNCj4gmpqampqampqac2VtbW5pOiCampqa mpqampqaNTAgKCMgb2Ygc2VtYXBob3JlIGlkZW50aWZpZXJzKQ0KPiCampqampqamppzZW1tbnM6 IJqampqampqamjM0MCAoIyBvZiBzZW1hcGhvcmVzIGluIHN5c3RlbSkNCj4gmpqampqampqac2Vt bW51OiCampqampqampoxNTAgKCMgb2YgdW5kbyBzdHJ1Y3R1cmVzIGluIHN5c3RlbSkNCj4gmpqa mpqampqac2VtbXNsOiCampqampqampozNDAgKG1heCAjIG9mIHNlbWFwaG9yZXMgcGVyIGlkKQ0K PiCampqampqamppzZW1vcG06IJqampqampqamjEwMCAobWF4ICMgb2Ygb3BlcmF0aW9ucyBwZXIg c2Vtb3AgY2FsbCkNCj4gmpqampqampqac2VtdW1lOiCampqampqampqaNTAgKG1heCAjIG9mIHVu ZG8gZW50cmllcyBwZXIgcHJvY2VzcykNCj4gmpqampqampqac2VtdXN6OiCampqampqampo2MzIg KHNpemUgaW4gYnl0ZXMgb2YgdW5kbyBzdHJ1Y3R1cmUpDQo+IJqampqampqamnNlbXZteDogmpqa mpqamjMyNzY3IChzZW1hcGhvcmUgbWF4aW11bSB2YWx1ZSkNCj4gmpqampqampqac2VtYWVtOiCa mpqampqaMTYzODQgKGFkanVzdCBvbiBleGl0IG1heCB2YWx1ZSkNCj4NCj4gmi0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLQ0KPiCabmZzc3RhdA0KPg0KPiCaQ2xpZW50IEluZm86DQo+IJpScGMgQ291bnRzOg0KPiCa mppHZXRhdHRyIJqaU2V0YXR0ciCamppMb29rdXAgmlJlYWRsaW5rIJqampqaUmVhZCCampqaV3Jp dGUgmpqaQ3JlYXRlIJqamlJlbW92ZQ0KPiCampqampqampowIJqampqampqaMCCampqampqamjAg mpqampqampowIJqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMA0KPiCampqa UmVuYW1lIJqampqaTGluayCamlN5bWxpbmsgmpqamk1rZGlyIJqamppSbWRpciCamlJlYWRkaXIg mlJkaXJQbHVzIJqamkFjY2Vzcw0KPiCampqampqampowIJqampqampqaMCCampqampqamjAgmpqa mpqampowIJqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMA0KPiCampqamk1r bm9kIJqamkZzc3RhdCCamppGc2luZm8gmlBhdGhDb25mIJqamkNvbW1pdA0KPiCaPiCampqampqa mjAgmpqampqampowIJqampqampqaMCCampqampqamjAgmpqampqampowDQo+IJpScGMgSW5mbzoN Cj4gmppUaW1lZE91dCCamkludmFsaWQgWCBSZXBsaWVzIJqaUmV0cmllcyCaUmVxdWVzdHMNCj4g mpqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMCCampqampqamjANCj4gmkNh Y2hlIEluZm86DQo+IJpBdHRyIEhpdHMgmpqaTWlzc2VzIExrdXAgSGl0cyCamppNaXNzZXMgQmlv UiBIaXRzIJqamk1pc3NlcyBCaW9XIEhpdHMgmpqaTWlzc2VzDQo+IJqampqampqamjAgmpqampqa mpowIJqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMCCampqampqamjAgmpqa mpqampowDQo+IJpCaW9STEhpdHMgmpqaTWlzc2VzIEJpb0QgSGl0cyCamppNaXNzZXMgRGlyRSBI aXRzIJqamk1pc3NlcyBBY2NzIEhpdHMgmpqaTWlzc2VzDQo+IJqampqampqamjAgmpqampqampow IJqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMCCampqampqamjAgmpqampqa mpowDQo+DQo+IJpTZXJ2ZXIgSW5mbzoNCj4gmpqaR2V0YXR0ciCamlNldGF0dHIgmpqaTG9va3Vw IJpSZWFkbGluayCampqamlJlYWQgmpqamldyaXRlIJqamkNyZWF0ZSCamppSZW1vdmUNCj4gmpqa mpqampqaMCCampqampqamjAgmpqampqampowIJqampqampqaMCCampqampqamjAgmpqampqampow IJqampqampqaMCCampqampqamjANCj4gmpqamlJlbmFtZSCampqamkxpbmsgmppTeW1saW5rIJqa mppNa2RpciCampqaUm1kaXIgmppSZWFkZGlyIJpSZGlyUGx1cyCamppBY2Nlc3MNCj4gmpqampqa mpqaMCCampqampqamjAgmpqampqampowIJqampqampqaMCCampqampqamjAgmpqampqampowIJqa mpqampqaMCCampqampqamjANCj4gmpqamppNa25vZCCamppGc3N0YXQgmpqaRnNpbmZvIJpQYXRo Q29uZiCamppDb21taXQNCj4gmpqampqampqaMCCampqampqamjAgmpqampqampowIJqampqampqa MCCampqampqamjANCj4gmlNlcnZlciBSZXQtRmFpbGVkDQo+IJqampqampqampqampqampqaMA0K PiCaU2VydmVyIEZhdWx0cw0KPiCampqampqampqampqaMA0KPiCaU2VydmVyIENhY2hlIFN0YXRz Og0KPiCampqaSW5wcm9nIJqampqaSWRlbSCaTm9uLWlkZW0gmpqaTWlzc2VzDQo+IJqampqampqa mjAgmpqampqampowIJqampqampqaMCCampqampqamjANCj4gmlNlcnZlciBXcml0ZSBHYXRoZXJp bmc6DQo+IJqaV3JpdGVPcHMgmldyaXRlUlBDIJqaT3BzYXZlZA0KPiCampqampqampowIJqampqa mpqaMCCampqampqamjANCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCabmV0c3RhdCAtcw0KPg0K PiCadGNwOg0KPiCampqampqampowIHBhY2tldHMgc2VudA0KPiCampqampqampqampqampqamjAg ZGF0YSBwYWNrZXRzICgwIGJ5dGVzKQ0KPiCampqampqampqampqampqamjAgZGF0YSBwYWNrZXRz ICgwIGJ5dGVzKSByZXRyYW5zbWl0dGVkDQo+IJqampqampqampqampqampqaMCBkYXRhIHBhY2tl dHMgdW5uZWNlc3NhcmlseSByZXRyYW5zbWl0dGVkDQo+IJqampqampqampqampqampqaMCByZXNl bmRzIGluaXRpYXRlZCBieSBNVFUgZGlzY292ZXJ5DQo+IJqampqampqampqampqampqaMCBhY2st b25seSBwYWNrZXRzICgwIGRlbGF5ZWQpDQo+IJqampqampqampqampqampqaMCBVUkcgb25seSBw YWNrZXRzDQo+IJqampqampqampqampqampqaMCB3aW5kb3cgcHJvYmUgcGFja2V0cw0KPiCampqa mpqampqampqampqamjAgd2luZG93IHVwZGF0ZSBwYWNrZXRzDQo+IJqampqampqampqampqampqa MCBjb250cm9sIHBhY2tldHMNCj4gmpqampqampqaMCBwYWNrZXRzIHJlY2VpdmVkDQo+IJqampqa mpqampqampqampqaMCBhY2tzIChmb3IgMCBieXRlcykNCj4gmpqampqampqampqampqampowIGR1 cGxpY2F0ZSBhY2tzDQo+IJqampqampqampqampqampqaMCBhY2tzIGZvciB1bnNlbnQgZGF0YQ0K PiCampqampqampqampqampqamjAgcGFja2V0cyAoMCBieXRlcykgcmVjZWl2ZWQgaW4tc2VxdWVu Y2UNCj4gmpqampqampqampqampqampowIGNvbXBsZXRlbHkgZHVwbGljYXRlIHBhY2tldHMgKDAg Ynl0ZXMpDQo+IJqampqampqampqampqampqaMCBvbGQgZHVwbGljYXRlIHBhY2tldHMNCj4gmpqa mpqampqampqampqampowIHBhY2tldHMgd2l0aCBzb21lIGR1cC4gZGF0YSAoMCBieXRlcyBkdXBl ZCkNCj4gmpqampqampqampqampqampowIG91dC1vZi1vcmRlciBwYWNrZXRzICgwIGJ5dGVzKQ0K PiCampqampqampqampqampqamjAgcGFja2V0cyAoMCBieXRlcykgb2YgZGF0YSBhZnRlciB3aW5k b3cNCj4gmpqampqampqampqampqampowIHdpbmRvdyBwcm9iZXMNCj4gmpqampqampqampqampqa mpowIHdpbmRvdyB1cGRhdGUgcGFja2V0cw0KPiCampqampqampqampqampqamjAgcGFja2V0cyBy ZWNlaXZlZCBhZnRlciBjbG9zZQ0KPiCampqampqampqampqampqamjAgZGlzY2FyZGVkIGZvciBi YWQgY2hlY2tzdW1zDQo+IJqampqampqampqampqampqaMCBkaXNjYXJkZWQgZm9yIGJhZCBoZWFk ZXIgb2Zmc2V0IGZpZWxkcw0KPiCampqampqampqampqampqamjAgZGlzY2FyZGVkIGJlY2F1c2Ug cGFja2V0IHRvbyBzaG9ydA0KPiCampqampqampqampqampqamjAgZGlzY2FyZGVkIGR1ZSB0byBt ZW1vcnkgcHJvYmxlbXMNCj4gmpqampqampqaMCBjb25uZWN0aW9uIHJlcXVlc3RzDQo+IJqampqa mpqamjAgY29ubmVjdGlvbiBhY2NlcHRzDQo+IJqampqampqamjAgYmFkIGNvbm5lY3Rpb24gYXR0 ZW1wdHMNCj4gmpqampqampqaMCBsaXN0ZW4gcXVldWUgb3ZlcmZsb3dzDQo+IJqampqampqamjAg aWdub3JlZCBSU1RzIGluIHRoZSB3aW5kb3dzDQo+IJqampqampqamjAgY29ubmVjdGlvbnMgZXN0 YWJsaXNoZWQgKGluY2x1ZGluZyBhY2NlcHRzKQ0KPiCampqampqampoyIGNvbm5lY3Rpb25zIGNs b3NlZCAoaW5jbHVkaW5nIDAgZHJvcHMpDQo+IJqampqampqampqampqampqaMCBjb25uZWN0aW9u cyB1cGRhdGVkIGNhY2hlZCBSVFQgb24gY2xvc2UNCj4gmpqampqampqampqampqampowIGNvbm5l Y3Rpb25zIHVwZGF0ZWQgY2FjaGVkIFJUVCB2YXJpYW5jZSBvbiBjbG9zZQ0KPiCampqampqampqa mpqampqamjAgY29ubmVjdGlvbnMgdXBkYXRlZCBjYWNoZWQgc3N0aHJlc2ggb24gY2xvc2UNCj4g mpqampqampqaMCBlbWJyeW9uaWMgY29ubmVjdGlvbnMgZHJvcHBlZA0KPiCampqampqampowIHNl Z21lbnRzIHVwZGF0ZWQgcnR0IChvZiAwIGF0dGVtcHRzKQ0KPiCampqampqampowIHJldHJhbnNt aXQgdGltZW91dHMNCj4gmpqampqampqampqampqampowIGNvbm5lY3Rpb25zIGRyb3BwZWQgYnkg cmV4bWl0IHRpbWVvdXQNCj4gmpqampqampqaMCBwZXJzaXN0IHRpbWVvdXRzDQo+IJqampqampqa mpqampqampqaMCBjb25uZWN0aW9ucyBkcm9wcGVkIGJ5IHBlcnNpc3QgdGltZW91dA0KPiCampqa mpqampowIENvbm5lY3Rpb25zIChmaW5fd2FpdF8yKSBkcm9wcGVkIGJlY2F1c2Ugb2YgdGltZW91 dA0KPiCampqampqampowIGtlZXBhbGl2ZSB0aW1lb3V0cw0KPiCampqampqampqampqampqamjAg a2VlcGFsaXZlIHByb2JlcyBzZW50DQo+IJqampqampqampqampqampqaMCBjb25uZWN0aW9ucyBk cm9wcGVkIGJ5IGtlZXBhbGl2ZQ0KPiCampqampqampowIGNvcnJlY3QgQUNLIGhlYWRlciBwcmVk aWN0aW9ucw0KPiCampqampqampowIGNvcnJlY3QgZGF0YSBwYWNrZXQgaGVhZGVyIHByZWRpY3Rp b25zDQo+IJqampqampqamjAgc3luY2FjaGUgZW50cmllcyBhZGRlZA0KPiCampqampqampqampqa mpqamjAgcmV0cmFuc21pdHRlZA0KPiCampqampqampqampqampqamjAgZHVwc3luDQo+IJqampqa mpqampqampqampqaMCBkcm9wcGVkDQo+IJqampqampqampqampqampqaMCBjb21wbGV0ZWQNCj4g mpqampqampqampqampqampowIGJ1Y2tldCBvdmVyZmxvdw0KPiCampqampqampqampqampqamjAg Y2FjaGUgb3ZlcmZsb3cNCj4gmpqampqampqampqampqampowIHJlc2V0DQo+IJqampqampqampqa mpqampqaMCBzdGFsZQ0KPiCampqampqampqampqampqamjAgYWJvcnRlZA0KPiCampqampqampqa mpqampqamjAgYmFkYWNrDQo+IJqampqampqampqampqampqaMCB1bnJlYWNoDQo+IJqampqampqa mpqampqampqaMCB6b25lIGZhaWx1cmVzDQo+IJqampqampqamjAgY29va2llcyBzZW50DQo+IJqa mpqampqamjAgY29va2llcyByZWNlaXZlZA0KPiCampqampqampowIGhvc3RjYWNoZSBlbnRyaWVz IGFkZGVkDQo+IJqampqampqampqampqampqaMCBidWNrZXQgb3ZlcmZsb3cNCj4gmpqampqampqa MCBTQUNLIHJlY292ZXJ5IGVwaXNvZGVzDQo+IJqampqampqamjAgc2VnbWVudCByZXhtaXRzIGlu IFNBQ0sgcmVjb3ZlcnkgZXBpc29kZXMNCj4gmpqampqampqaMCBieXRlIHJleG1pdHMgaW4gU0FD SyByZWNvdmVyeSBlcGlzb2Rlcw0KPiCampqampqampowIFNBQ0sgb3B0aW9ucyAoU0FDSyBibG9j a3MpIHJlY2VpdmVkDQo+IJqampqampqamjAgU0FDSyBvcHRpb25zIChTQUNLIGJsb2Nrcykgc2Vu dA0KPiCampqampqampowIFNBQ0sgc2NvcmVib2FyZCBvdmVyZmxvdw0KPiCampqampqampowIHBh Y2tldHMgd2l0aCBFQ04gQ0UgYml0IHNldA0KPiCampqampqampowIHBhY2tldHMgd2l0aCBFQ04g RUNUKDApIGJpdCBzZXQNCj4gmpqampqampqaMCBwYWNrZXRzIHdpdGggRUNOIEVDVCgxKSBiaXQg c2V0DQo+IJqampqampqamjAgc3VjY2Vzc2Z1bCBFQ04gaGFuZHNoYWtlcw0KPiCampqampqampow IHRpbWVzIEVDTiByZWR1Y2VkIHRoZSBjb25nZXN0aW9uIHdpbmRvdw0KPiCadWRwOg0KPiCampqa mpqampowIGRhdGFncmFtcyByZWNlaXZlZA0KPiCampqampqampowIHdpdGggaW5jb21wbGV0ZSBo ZWFkZXINCj4gmpqampqampqaMCB3aXRoIGJhZCBkYXRhIGxlbmd0aCBmaWVsZA0KPiCampqampqa mpowIHdpdGggYmFkIGNoZWNrc3VtDQo+IJqampqampqamjAgd2l0aCBubyBjaGVja3N1bQ0KPiCa mpqampqampowIGRyb3BwZWQgZHVlIHRvIG5vIHNvY2tldA0KPiCampqampqampowIGJyb2FkY2Fz dC9tdWx0aWNhc3QgZGF0YWdyYW1zIHVuZGVsaXZlcmVkDQo+IJqampqampqamjAgZHJvcHBlZCBk dWUgdG8gZnVsbCBzb2NrZXQgYnVmZmVycw0KPiCampqampqampowIG5vdCBmb3IgaGFzaGVkIHBj Yg0KPiCampqampqampowIGRlbGl2ZXJlZA0KPiCampqampqampowIGRhdGFncmFtcyBvdXRwdXQN Cj4gmpqampqampqaMCB0aW1lcyBtdWx0aWNhc3Qgc291cmNlIGZpbHRlciBtYXRjaGVkDQo+IJpp cDoNCj4gmpqampqampqaMCB0b3RhbCBwYWNrZXRzIHJlY2VpdmVkDQo+IJqampqampqamjAgYmFk IGhlYWRlciBjaGVja3N1bXMNCj4gmpqampqampqaMCB3aXRoIHNpemUgc21hbGxlciB0aGFuIG1p bmltdW0NCj4gmpqampqampqaMCB3aXRoIGRhdGEgc2l6ZSA8IGRhdGEgbGVuZ3RoDQo+IJqampqa mpqamjAgd2l0aCBpcCBsZW5ndGggPiBtYXggaXAgcGFja2V0IHNpemUNCj4gmpqampqampqaMCB3 aXRoIGhlYWRlciBsZW5ndGggPCBkYXRhIHNpemUNCj4gmpqampqampqaMCB3aXRoIGRhdGEgbGVu Z3RoIDwgaGVhZGVyIGxlbmd0aA0KPiCampqampqampowIHdpdGggYmFkIG9wdGlvbnMNCj4gmpqa mpqampqaMCB3aXRoIGluY29ycmVjdCB2ZXJzaW9uIG51bWJlcg0KPiCampqampqampowIGZyYWdt ZW50cyByZWNlaXZlZA0KPiCampqampqampowIGZyYWdtZW50cyBkcm9wcGVkIChkdXAgb3Igb3V0 IG9mIHNwYWNlKQ0KPiCampqampqampowIGZyYWdtZW50cyBkcm9wcGVkIGFmdGVyIHRpbWVvdXQN Cj4gmpqampqampqaMCBwYWNrZXRzIHJlYXNzZW1ibGVkIG9rDQo+IJqampqampqamjAgcGFja2V0 cyBmb3IgdGhpcyBob3N0DQo+IJqampqampqamjAgcGFja2V0cyBmb3IgdW5rbm93bi91bnN1cHBv cnRlZCBwcm90b2NvbA0KPiCampqampqampowIHBhY2tldHMgZm9yd2FyZGVkICgwIHBhY2tldHMg ZmFzdCBmb3J3YXJkZWQpDQo+IJqampqampqamjAgcGFja2V0cyBub3QgZm9yd2FyZGFibGUNCj4g mpqampqampqaMCBwYWNrZXRzIHJlY2VpdmVkIGZvciB1bmtub3duIG11bHRpY2FzdCBncm91cA0K PiCampqampqampowIHJlZGlyZWN0cyBzZW50DQo+IJqampqampqamjAgcGFja2V0cyBzZW50IGZy b20gdGhpcyBob3N0DQo+IJqampqampqamjAgcGFja2V0cyBzZW50IHdpdGggZmFicmljYXRlZCBp cCBoZWFkZXINCj4gmpqampqampqaMCBvdXRwdXQgcGFja2V0cyBkcm9wcGVkIGR1ZSB0byBubyBi dWZzLCBldGMuDQo+IJqampqampqamjAgb3V0cHV0IHBhY2tldHMgZGlzY2FyZGVkIGR1ZSB0byBu byByb3V0ZQ0KPiCampqampqampowIG91dHB1dCBkYXRhZ3JhbXMgZnJhZ21lbnRlZA0KPiCampqa mpqampowIGZyYWdtZW50cyBjcmVhdGVkDQo+IJqampqampqamjAgZGF0YWdyYW1zIHRoYXQgY2Fu J3QgYmUgZnJhZ21lbnRlZA0KPiCampqampqampowIHR1bm5lbGluZyBwYWNrZXRzIHRoYXQgY2Fu J3QgZmluZCBnaWYNCj4gmpqampqampqaMCBkYXRhZ3JhbXMgd2l0aCBiYWQgYWRkcmVzcyBpbiBo ZWFkZXINCj4gmmljbXA6DQo+IJqampqampqamjAgY2FsbHMgdG8gaWNtcF9lcnJvcg0KPiCampqa mpqampowIGVycm9ycyBub3QgZ2VuZXJhdGVkIGluIHJlc3BvbnNlIHRvIGFuIGljbXAgbWVzc2Fn ZQ0KPiCampqampqampowIG1lc3NhZ2VzIHdpdGggYmFkIGNvZGUgZmllbGRzDQo+IJqampqampqa mjAgbWVzc2FnZXMgbGVzcyB0aGFuIHRoZSBtaW5pbXVtIGxlbmd0aA0KPiCampqampqampowIG1l c3NhZ2VzIHdpdGggYmFkIGNoZWNrc3VtDQo+IJqampqampqamjAgbWVzc2FnZXMgd2l0aCBiYWQg bGVuZ3RoDQo+IJqampqampqamjAgbXVsdGljYXN0IGVjaG8gcmVxdWVzdHMgaWdub3JlZA0KPiCa mpqampqampowIG11bHRpY2FzdCB0aW1lc3RhbXAgcmVxdWVzdHMgaWdub3JlZA0KPiCampqampqa mpowIG1lc3NhZ2UgcmVzcG9uc2VzIGdlbmVyYXRlZA0KPiCampqampqampowIGludmFsaWQgcmV0 dXJuIGFkZHJlc3Nlcw0KPiCampqampqampowIG5vIHJldHVybiByb3V0ZXMNCj4gmmlnbXA6DQo+ IJqampqampqamjAgbWVzc2FnZXMgcmVjZWl2ZWQNCj4gmpqampqampqaMCBtZXNzYWdlcyByZWNl aXZlZCB3aXRoIHRvbyBmZXcgYnl0ZXMNCj4gmpqampqampqaMCBtZXNzYWdlcyByZWNlaXZlZCB3 aXRoIHdyb25nIFRUTA0KPiCampqampqampowIG1lc3NhZ2VzIHJlY2VpdmVkIHdpdGggYmFkIGNo ZWNrc3VtDQo+IJqampqampqamjAgVjEvVjIgbWVtYmVyc2hpcCBxdWVyaWVzIHJlY2VpdmVkDQo+ IJqampqampqamjAgVjMgbWVtYmVyc2hpcCBxdWVyaWVzIHJlY2VpdmVkDQo+IJqampqampqamjAg bWVtYmVyc2hpcCBxdWVyaWVzIHJlY2VpdmVkIHdpdGggaW52YWxpZCBmaWVsZChzKQ0KPiCampqa mpqampowIGdlbmVyYWwgcXVlcmllcyByZWNlaXZlZA0KPiCampqampqampowIGdyb3VwIHF1ZXJp ZXMgcmVjZWl2ZWQNCj4gmpqampqampqaMCBncm91cC1zb3VyY2UgcXVlcmllcyByZWNlaXZlZA0K PiCampqampqampowIGdyb3VwLXNvdXJjZSBxdWVyaWVzIGRyb3BwZWQNCj4gmpqampqampqaMCBt ZW1iZXJzaGlwIHJlcG9ydHMgcmVjZWl2ZWQNCj4gmpqampqampqaMCBtZW1iZXJzaGlwIHJlcG9y dHMgcmVjZWl2ZWQgd2l0aCBpbnZhbGlkIGZpZWxkKHMpDQo+IJqampqampqamjAgbWVtYmVyc2hp cCByZXBvcnRzIHJlY2VpdmVkIGZvciBncm91cHMgdG8gd2hpY2ggd2UgYmVsb25nDQo+IJqampqa mpqamjAgVjMgcmVwb3J0cyByZWNlaXZlZCB3aXRob3V0IFJvdXRlciBBbGVydA0KPiCampqampqa mpowIG1lbWJlcnNoaXAgcmVwb3J0cyBzZW50DQo+IJphcnA6DQo+IJqampqampqamjAgQVJQIHJl cXVlc3RzIHNlbnQNCj4gmpqampqampqaMCBBUlAgcmVwbGllcyBzZW50DQo+IJqampqampqamjAg QVJQIHJlcXVlc3RzIHJlY2VpdmVkDQo+IJqampqampqamjAgQVJQIHJlcGxpZXMgcmVjZWl2ZWQN Cj4gmpo+IJqampqampowIEFSUCBwYWNrZXRzIHJlY2VpdmVkDQo+IJqampqampqamjAgdG90YWwg cGFja2V0cyBkcm9wcGVkIGR1ZSB0byBubyBBUlAgZW50cnkNCj4gmpqampqampqaMCBBUlAgZW50 cnlzIHRpbWVkIG91dA0KPiCampqampqampowIER1cGxpY2F0ZSBJUHMgc2Vlbg0KPiCaaXA2Og0K PiCampqampqampowIHRvdGFsIHBhY2tldHMgcmVjZWl2ZWQNCj4gmpqampqampqaMCB3aXRoIHNp emUgc21hbGxlciB0aGFuIG1pbmltdW0NCj4gmpqampqampqaMCB3aXRoIGRhdGEgc2l6ZSA8IGRh dGEgbGVuZ3RoDQo+IJqampqampqamjAgd2l0aCBiYWQgb3B0aW9ucw0KPiCampqampqampowIHdp dGggaW5jb3JyZWN0IHZlcnNpb24gbnVtYmVyDQo+IJqampqampqamjAgZnJhZ21lbnRzIHJlY2Vp dmVkDQo+IJqampqampqamjAgZnJhZ21lbnRzIGRyb3BwZWQgKGR1cCBvciBvdXQgb2Ygc3BhY2Up DQo+IJqampqampqamjAgZnJhZ21lbnRzIGRyb3BwZWQgYWZ0ZXIgdGltZW91dA0KPiCampqampqa mpowIGZyYWdtZW50cyB0aGF0IGV4Y2VlZGVkIGxpbWl0DQo+IJqampqampqamjAgcGFja2V0cyBy ZWFzc2VtYmxlZCBvaw0KPiCampqampqampowIHBhY2tldHMgZm9yIHRoaXMgaG9zdA0KPiCampqa mpqampowIHBhY2tldHMgZm9yd2FyZGVkDQo+IJqampqampqamjAgcGFja2V0cyBub3QgZm9yd2Fy ZGFibGUNCj4gmpqampqampqaMCByZWRpcmVjdHMgc2VudA0KPiCampqampqampowIHBhY2tldHMg c2VudCBmcm9tIHRoaXMgaG9zdA0KPiCampqampqampowIHBhY2tldHMgc2VudCB3aXRoIGZhYnJp Y2F0ZWQgaXAgaGVhZGVyDQo+IJqampqampqamjAgb3V0cHV0IHBhY2tldHMgZHJvcHBlZCBkdWUg dG8gbm8gYnVmcywgZXRjLg0KPiCampqampqampowIG91dHB1dCBwYWNrZXRzIGRpc2NhcmRlZCBk dWUgdG8gbm8gcm91dGUNCj4gmpqampqampqaMCBvdXRwdXQgZGF0YWdyYW1zIGZyYWdtZW50ZWQN Cj4gmpqampqampqaMCBmcmFnbWVudHMgY3JlYXRlZA0KPiCampqampqampowIGRhdGFncmFtcyB0 aGF0IGNhbid0IGJlIGZyYWdtZW50ZWQNCj4gmpqampqampqaMCBwYWNrZXRzIHRoYXQgdmlvbGF0 ZWQgc2NvcGUgcnVsZXMNCj4gmpqampqampqaMCBtdWx0aWNhc3QgcGFja2V0cyB3aGljaCB3ZSBk b24ndCBqb2luDQo+IJqampqampqamk1idWYgc3RhdGlzdGljczoNCj4gmpqampqampqampqampqa mpowIG9uZSBtYnVmDQo+IJqampqampqampqampqampqaMCBvbmUgZXh0IG1idWYNCj4gmpqampqa mpqampqampqampowIHR3byBvciBtb3JlIGV4dCBtYnVmDQo+IJqampqampqamjAgcGFja2V0cyB3 aG9zZSBoZWFkZXJzIGFyZSBub3QgY29udGlndW91cw0KPiCampqampqampowIHR1bm5lbGluZyBw YWNrZXRzIHRoYXQgY2FuJ3QgZmluZCBnaWYNCj4gmpqampqampqaMCBwYWNrZXRzIGRpc2NhcmRl ZCBiZWNhdXNlIG9mIHRvbyBtYW55IGhlYWRlcnMNCj4gmpqampqampqaMCBmYWlsdXJlcyBvZiBz b3VyY2UgYWRkcmVzcyBzZWxlY3Rpb24NCj4gmpqampqampqaU291cmNlIGFkZHJlc3NlcyBzZWxl Y3Rpb24gcnVsZSBhcHBsaWVkOg0KPiCaaWNtcDY6DQo+IJqampqampqamjAgY2FsbHMgdG8gaWNt cDZfZXJyb3INCj4gmpqampqampqaMCBlcnJvcnMgbm90IGdlbmVyYXRlZCBpbiByZXNwb25zZSB0 byBhbiBpY21wNiBtZXNzYWdlDQo+IJqampqampqamjAgZXJyb3JzIG5vdCBnZW5lcmF0ZWQgYmVj YXVzZSBvZiByYXRlIGxpbWl0YXRpb24NCj4gmpqampqampqaMCBtZXNzYWdlcyB3aXRoIGJhZCBj b2RlIGZpZWxkcw0KPiCampqampqampowIG1lc3NhZ2VzIDwgbWluaW11bSBsZW5ndGgNCj4gmpqa mpqampqaMCBiYWQgY2hlY2tzdW1zDQo+IJqampqampqamjAgbWVzc2FnZXMgd2l0aCBiYWQgbGVu Z3RoDQo+IJqampqampqamkhpc3RvZ3JhbSBvZiBlcnJvciBtZXNzYWdlcyB0byBiZSBnZW5lcmF0 ZWQ6DQo+IJqampqampqampqampqampqaMCBubyByb3V0ZQ0KPiCampqampqampqampqampqamjAg YWRtaW5pc3RyYXRpdmVseSBwcm9oaWJpdGVkDQo+IJqampqampqampqampqampqaMCBiZXlvbmQg c2NvcGUNCj4gmpqampqampqampqampqampowIGFkZHJlc3MgdW5yZWFjaGFibGUNCj4gmpqampqa mpqampqampqampowIHBvcnQgdW5yZWFjaGFibGUNCj4gmpqampqampqampqampqampowIHBhY2tl dCB0b28gYmlnDQo+IJqampqampqampqampqampqaMCB0aW1lIGV4Y2VlZCB0cmFuc2l0DQo+IJqa mpqampqampqampqampqaMCB0aW1lIGV4Y2VlZCByZWFzc2VtYmx5DQo+IJqampqampqampqampqa mpqaMCBlcnJvbmVvdXMgaGVhZGVyIGZpZWxkDQo+IJqampqampqampqampqampqaMCB1bnJlY29n bml6ZWQgbmV4dCBoZWFkZXINCj4gmpqampqampqampqampqampowIHVucmVjb2duaXplZCBvcHRp b24NCj4gmpqampqampqampqampqampowIHJlZGlyZWN0DQo+IJqampqampqampqampqampqaMCB1 bmtub3duDQo+IJqampqampqamjAgbWVzc2FnZSByZXNwb25zZXMgZ2VuZXJhdGVkDQo+IJqampqa mpqamjAgbWVzc2FnZXMgd2l0aCB0b28gbWFueSBORCBvcHRpb25zDQo+IJqampqampqamjAgbWVz c2FnZXMgd2l0aCBiYWQgTkQgb3B0aW9ucw0KPiCampqampqampowIGJhZCBuZWlnaGJvciBzb2xp Y2l0YXRpb24gbWVzc2FnZXMNCj4gmpqampqampqaMCBiYWQgbmVpZ2hib3IgYWR2ZXJ0aXNlbWVu dCBtZXNzYWdlcw0KPiCampqampqampowIGJhZCByb3V0ZXIgc29saWNpdGF0aW9uIG1lc3NhZ2Vz DQo+IJqampqampqamjAgYmFkIHJvdXRlciBhZHZlcnRpc2VtZW50IG1lc3NhZ2VzDQo+IJqampqa mpqamjAgYmFkIHJlZGlyZWN0IG1lc3NhZ2VzDQo+IJqampqampqamjAgcGF0aCBNVFUgY2hhbmdl cw0KPiCacmlwNjoNCj4gmpqampqampqaMCBtZXNzYWdlcyByZWNlaXZlZA0KPiCampqampqampow IGNoZWNrc3VtIGNhbGN1bGF0aW9ucyBvbiBpbmJvdW5kDQo+IJqampqampqamjAgbWVzc2FnZXMg d2l0aCBiYWQgY2hlY2tzdW0NCj4gmpqampqampqaMCBtZXNzYWdlcyBkcm9wcGVkIGR1ZSB0byBu byBzb2NrZXQNCj4gmpqampqampqaMCBtdWx0aWNhc3QgbWVzc2FnZXMgZHJvcHBlZCBkdWUgdG8g bm8gc29ja2V0DQo+IJqampqampqamjAgbWVzc2FnZXMgZHJvcHBlZCBkdWUgdG8gZnVsbCBzb2Nr ZXQgYnVmZmVycw0KPiCampqampqampowIGRlbGl2ZXJlZA0KPiCampqampqampowIGRhdGFncmFt cyBvdXRwdXQNCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCabmV0c3RhdCAtbQ0KPg0KPiCaMS8y ODMvMjg0IG1idWZzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3RhbCkNCj4gmjE4NDQ2NzQ0MDcz NzA5NTUxNjAyLzE0OC8xMzQvMjU2MDAgbWJ1ZiBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2Fj aGUvdG90YWwvbWF4KQ0KPiCaMC8xNDIgbWJ1ZitjbHVzdGVycyBvdXQgb2YgcGFja2V0IHNlY29u ZGFyeSB6b25lIGluIHVzZSAoY3VycmVudC9jYWNoZSkNCj4gmjAvMC8wLzEyODAwIDRrIChwYWdl IHNpemUpIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVudC9jYWNoZS90b3RhbC9tYXgpDQo+ IJowLzAvMC8xOTIwMCA5ayBqdW1ibyBjbHVzdGVycyBpbiB1c2UgKGN1cnJlbnQvY2FjaGUvdG90 YWwvbWF4KQ0KPiCaMC8wLzAvMTI4MDAgMTZrIGp1bWJvIGNsdXN0ZXJzIGluIHVzZSAoY3VycmVu dC9jYWNoZS90b3RhbC9tYXgpDQo+IJoxODAxNDM5ODUwOTQ4MTk1NksvMzY2Sy8zMzlLIGJ5dGVz IGFsbG9jYXRlZCB0byBuZXR3b3JrIChjdXJyZW50L2NhY2hlL3RvdGFsKQ0KPiCaMC8wLzAgcmVx dWVzdHMgZm9yIG1idWZzIGRlbmllZCAobWJ1ZnMvY2x1c3RlcnMvbWJ1ZitjbHVzdGVycykNCj4g mjAvMC8wIHJlcXVlc3RzIGZvciBqdW1ibyBjbHVzdGVycyBkZW5pZWQgKDRrLzlrLzE2aykNCj4g mjAgcmVxdWVzdHMgZm9yIHNmYnVmcyBkZW5pZWQNCj4gmjAgcmVxdWVzdHMgZm9yIHNmYnVmcyBk ZWxheWVkDQo+IJowIHJlcXVlc3RzIGZvciBJL08gaW5pdGlhdGVkIGJ5IHNlbmRmaWxlDQo+IJow IGNhbGxzIHRvIHByb3RvY29sIGRyYWluIHJvdXRpbmVzDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0N Cj4gmm5ldHN0YXQgLWlkDQo+DQo+IJpOYW1lIJqamk10dSBOZXR3b3JrIJqampqamkFkZHJlc3Mg mpqampqampqampqamklwa3RzIEllcnJzIElkcm9wIJqamk9wa3RzIE9lcnJzIJpDb2xsIERyb3AN Cj4gmmVtMCogmpoxNTAwIDxMaW5rIzE+IJqampqaMDg6MDA6Mjc6NGI6NmY6ZjAgmpqampqamjAg mpqamjAgmpqamjAgmpqampqamjAgmpqamjAgmpqamjAgmpqaMA0KPiCadXNidXMgmpqamjAgPExp bmsjMj4gmpqampqampqampqampqampqampqampqampqampqaMCCampqaMCCampqaMCCampqampqa MCCampqaMCCampqaMCCampowDQo+IJpsbzAgmpoxNjM4NCA8TGluayMzPiCampqampqampqampqa mpqampqampqampqampqampowIJqampowIJqampowIJqampqampowIJqampowIJqampowIJqamjAN Cj4gmmxvMCCamjE2Mzg0IGxvY2FsaG9zdCCampqaOjoxIJqampqampqampqampqampqampqamjAg mpqami0gmpqami0gmpqampqamjAgmpqami0gmpqami0gmpqaLQ0KPiCabG8wIJqaMTYzODQgZmU4 MDo6MSVsbzAgmppmZTgwOjoxIJqampqampqampqampqampqaMCCampqaLSCampqaLSCampqampqa MCCampqaLSCampqaLSCampotDQo+IJpsbzAgmpoxNjM4NCB5b3VyLW5ldCCampqammxvY2FsaG9z dCCampqampqampqampqampowIJqampotIJqampotIJqampqampowIJqampotIJqampotIJqami0N Cj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCabmV0c3RhdCAtYW5yDQo+DQo+IJpSb3V0aW5nIHRh Ymxlcw0KPg0KPiCaSW50ZXJuZXQ6DQo+IJpEZXN0aW5hdGlvbiCampqampqaR2F0ZXdheSCampqa mpqampqamkZsYWdzIJqamlJlZnMgmpqamppVc2Ugmk5ldGlmIEV4cGlyZQ0KPiCaMTI3LjAuMC4x IJqampqampqammxpbmsjMyCampqampqampqamppVSCCampqampqampowIJqampqampowIJqammxv MA0KPg0KPiCaSW50ZXJuZXQ2Og0KPiCaRGVzdGluYXRpb24gmpqampqampqampqampqampqampqa mkdhdGV3YXkgmpqampqampqampqampqampqampqamkZsYWdzIJqampqaTmV0aWYgRXhwaXJlDQo+ IJo6Oi85NiCampqampqampqampqampqampqampqampqampqaOjoxIJqampqampqampqampqampqa mpqampqampqaVUdSUyCampqampqabG8wDQo+IJo6OjEgmpqampqampqampqampqampqampqampqa mpqampqaOjoxIJqampqampqampqampqampqampqampqampqaVUggmpqampqampqabG8wDQo+IJo6 OmZmZmY6MC4wLjAuMC85NiCampqampqampqampqampqaOjoxIJqampqampqampqampqampqampqa mpqampqaVUdSUyCampqampqabG8wDQo+IJpmZTgwOjovMTAgmpqampqampqampqampqampqampqa mpqaOjoxIJqampqampqampqampqampqampqampqampqaVUdSUyCampqampqabG8wDQo+IJpmZTgw OjolbG8wLzY0IJqampqampqampqampqampqampqabGluayMzIJqampqampqampqampqampqampqa mpqaVSCampqampqampqabG8wDQo+IJpmZTgwOjoxJWxvMCCampqampqampqampqampqampqampqa bGluayMzIJqampqampqampqampqampqampqampqaVUhTIJqampqampqabG8wDQo+IJpmZjAxOjol bG8wLzMyIJqampqampqampqampqampqampqaOjoxIJqampqampqampqampqampqampqampqampqa VSCampqampqampqabG8wDQo+IJpmZjAyOjovMTYgmpqampqampqampqampqampqampqampqaOjox IJqampqampqampqampqampqampqampqampqaVUdSUyCampqampqabG8wDQo+IJpmZjAyOjolbG8w LzMyIJqampqampqampqampqampqampqaOjoxIJqampqampqampqampqampqampqampqampqaVSCa mpqampqampqabG8wDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmm5ldHN0YXQgLWFuQQ0KPg0K PiCaQWN0aXZlIEludGVybmV0IGNvbm5lY3Rpb25zIChpbmNsdWRpbmcgc2VydmVycykNCj4gmlRj cGNiIJqampqampqampqaUHJvdG8gUmVjdi1RIFNlbmQtUSBMb2NhbCBBZGRyZXNzIJqampqaRm9y ZWlnbiBBZGRyZXNzIJqamihzdGF0ZSkNCj4gmmZmZmZmZTAwMDI3NDU3YTAgdGNwNCCampqampow IJqampqaMCAxMjcuMC4wLjEuMjUgmpqampqaKi4qIJqampqampqampqampqamkxJU1RFTg0KPiCa QWN0aXZlIFVOSVggZG9tYWluIHNvY2tldHMNCj4gmkFkZHJlc3MgmlR5cGUgmppSZWN2LVEgU2Vu ZC1RIJqamklub2RlIJqamppDb25uIJqamppSZWZzIJpOZXh0cmVmIEFkZHINCj4gmmZmZmZmZTAw MDI2Y2VkMjAgc3RyZWFtIJqampqaMCCampqamjAgZmZmZmZlMDAwMjZiNGQyMCCampqampqaMCCa mpqampqaMCCampqampqaMCAvdmFyL3J1bi9kZXZkLnBpcGUNCj4NCj4gmi0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LQ0KPiCabmV0c3RhdCAtYUwNCj4NCj4gmkN1cnJlbnQgbGlzdGVuIHF1ZXVlIHNpemVzIChxbGVu L2luY3FsZW4vbWF4cWxlbikNCj4gmlByb3RvIExpc3RlbiCampqampqamkxvY2FsIEFkZHJlc3MN Cj4gmnRjcDQgmjAvMC8xMCCampqampqammxvY2FsaG9zdC5zbXRwDQo+IJp1bml4IJowLzAvNCCa mpqampqampovdmFyL3J1bi9kZXZkLnBpcGUNCj4NCj4gmi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiCaZnN0 YXQNCj4NCj4gmlVTRVIgmpqamkNNRCCampqampqamppQSUQgmppGRCBNT1VOVCCampqamklOVU0g TU9ERSCampqampqamlNafERWIFIvVw0KPiCacm9vdCCampqaaXBuYXQgmpqampqaMTA4MyByb290 IC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammlwbmF0 IJqampqamjEwODMgmpp3ZCAvIJqampqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+ IJpyb290IJqampppcG5hdCCampqampoxMDgzIHRleHQgLyCampqampqaMzc4OTU2IC1yLXhyLXhy LXggmpo2Mjg0OCCacg0KPiCacm9vdCCampqaaXBuYXQgmpqampqaMTA4MyBjdHR5IC9kZXYgmpqa mpqampo0MiBjcnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqammlwbmF0IJqampqamjEw ODMgmpqaMCAvZGV2IJqampqampqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpyb290IJqa mpppcG5hdCCampqampoxMDgzIJqamjEgL2RldiCampqampqamjQyIGNydy0tLS0tLS0gmpp0dHl2 MCBydw0KPiCacm9vdCCampqaaXBuYXQgmpqampqaMTA4MyCampoyIC9kZXYgmpqampqampo0MiBj cnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqammlwbmF0IJqampqamjEwODMgmpqaMyAv ZGV2IJqampqampqaODAgY3J3LS0tLS0tLSCammlwbmF0IHJ3DQo+IJpyb290IJqampppcG5hdCCa mpqampoxMDgzIJqamjQgL2RldiCampqampqamjc5IGNydy0tLS0tLS0gmpqammlwbCCacg0KPiCa cm9vdCCampqaaXBuYXQgmpqampqaMTA4MyCampo1IC8gmpqampqamjI2MDUxNjQgLXJ3LXItLXIt LSCampqamjU4IJpyDQo+IJpyb290IJqamppzaCCampqampqampoxMDc4IHJvb3QgLyCampqampqa mpqampoyIGRyd3hyLXhyLXggmpqaMTAyNCCacg0KPiCacm9vdCCampqac2ggmpqampqampqaMTA3 OCCamndkIC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3Qgmpqa mnNoIJqampqampqamjEwNzggdGV4dCAvIJqampqampoxNDIxMzY4IC1yLXhyLXhyLXggmjE0Mjgz MiCacg0KPiCacm9vdCCampqac2ggmpqampqampqaMTA3OCBjdHR5IC9kZXYgmpqampqampo0MiBj cnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqamnNoIJqampqampqamjEwNzggmpqaMCAv ZGV2IJqampqampqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpyb290IJqamppzaCCampqa mpqampoxMDc4IJqamjEgL2RldiCampqampqamjQyIGNydy0tLS0tLS0gmpp0dHl2MCBydw0KPiCa cm9vdCCampqac2ggmpqampqampqaMTA3OCCampoyIC9kZXYgmpqampqampo0MiBjcnctLS0tLS0t IJqadHR5djAgcncNCj4gmnJvb3QgmpqamnNoIJqampqampqamjEwNjYgcm9vdCAvIJqampqampqa mpqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppzaCCampqampqampoxMDY2 IJqad2QgLyCampqampqampqampoyIGRyd3hyLXhyLXggmpqaMTAyNCCacg0KPiCacm9vdCCampqa c2ggmpqampqampqaMTA2NiB0ZXh0IC8gmpqampqamjE0MjEzNjggLXIteHIteHIteCCaMTQyODMy IJpyDQo+IJpyb290IJqamppzaCCampqampqampoxMDY2IGN0dHkgL2RldiCampqampqamjQyIGNy dy0tLS0tLS0gmpp0dHl2MCBydw0KPiCacm9vdCCampqac2ggmpqampqampqaMTA2NiCampowIC9k ZXYgmpqampqampo0MiBjcnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqamnNoIJqampqa mpqamjEwNjYgmpqaMSAvZGV2IJqampqampqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpy b290IJqamppzaCCampqampqampoxMDY2IJqamjIgL2RldiCampqampqamjQyIGNydy0tLS0tLS0g mpp0dHl2MCBydw0KPiCacm9vdCCampqac2ggmpqampqampqaMTA2NiCamjEwIC8gmpqampqamjI2 MDQ5NjAgLXIteHIteHIteCCampqaNTM0IJpyDQo+IJpyb290IJqamppjc2ggmpqampqampoxMDE3 IHJvb3QgLyCampqampqampqampoyIGRyd3hyLXhyLXggmpqaMTAyNCCacg0KPiCacm9vdCCampqa Y3NoIJqampqampqaMTAxNyCamndkIC8gmpqampqamjk2MzgyNSBkcnd4ci14ci14IJqamjEwMjQg mnINCj4gmnJvb3QgmpqammNzaCCampqampqamjEwMTcgdGV4dCAvIJqampqampoxNDIxMTU4IC1y LXhyLXhyLXggmjM2OTA0OCCacg0KPiCacm9vdCCampqaY3NoIJqampqampqaMTAxNyBjdHR5IC9k ZXYgmpqampqampo0MiBjcnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqammNzaCCampqa mpqamjEwMTcgmpoxNSAvZGV2IJqampqampqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpy b290IJqamppjc2ggmpqampqampoxMDE3IJqaMTYgL2RldiCampqampqamjQyIGNydy0tLS0tLS0g mpp0dHl2MCBydw0KPiCacm9vdCCampqaY3NoIJqampqampqaMTAxNyCamjE3IC9kZXYgmpqampqa mpo0MiBjcnctLS0tLS0tIJqadHR5djAgcncNCj4gmnJvb3QgmpqammNzaCCampqampqamjEwMTcg mpoxOCAvZGV2IJqampqampqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpyb290IJqamppj c2ggmpqampqampoxMDE3IJqaMTkgL2RldiCampqampqamjQyIGNydy0tLS0tLS0gmpp0dHl2MCBy dw0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxNiByb290IC8gmpqampqampqampqaMiBkcnd4 ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTYgmpp3ZCAvIJqa mpqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppnZXR0eSCampqa mpoxMDE2IHRleHQgLyCampqampqaNzY1MzAwIC1yLXhyLXhyLXggmpoyNzUwNCCacg0KPiCacm9v dCCampqaZ2V0dHkgmpqampqaMTAxNiBjdHR5IC9kZXYgmpqampqampo0OSBjcnctLS0tLS0tIJqa dHR5djcgcncNCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTYgmpqaMCAvZGV2IJqampqampqa NDkgY3J3LS0tLS0tLSCamnR0eXY3IHJ3DQo+IJpyb290IJqamppnZXR0eSCampqampoxMDE2IJqa mjEgL2RldiCampqampqamjQ5IGNydy0tLS0tLS0gmpp0dHl2NyBydw0KPiCacm9vdCCampqaZ2V0 dHkgmpqampqaMTAxNiCampoyIC9kZXYgmpqampqampo0OSBjcnctLS0tLS0tIJqadHR5djcgcncN Cj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTUgcm9vdCAvIJqampqampqampqamjIgZHJ3eHIt eHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppnZXR0eSCampqampoxMDE1IJqad2QgLyCampqa mpqampqampoyIGRyd3hyLXhyLXggmpqaMTAyNCCacg0KPiCacm9vdCCampqaZ2V0dHkgmpqampqa MTAxNSB0ZXh0IC8gmpqampqamjc2NTMwMCAtci14ci14ci14IJqaMjc1MDQgmnINCj4gmnJvb3Qg mpqammdldHR5IJqampqamjEwMTUgY3R0eSAvZGV2IJqampqampqaNDggY3J3LS0tLS0tLSCamnR0 eXY2IHJ3DQo+IJpyb290IJqamppnZXR0eSCampqampoxMDE1IJqamjAgL2RldiCampqampqamjQ4 IGNydy0tLS0tLS0gmpp0dHl2NiBydw0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxNSCampox IC9kZXYgmpqampqampo0OCBjcnctLS0tLS0tIJqadHR5djYgcncNCj4gmnJvb3QgmpqammdldHR5 IJqampqamjEwMTUgmpqaMiAvZGV2IJqampqampqaNDggY3J3LS0tLS0tLSCamnR0eXY2IHJ3DQo+ IJpyb290IJqamppnZXR0eSCampqampoxMDE0IHJvb3QgLyCampqampqampqampoyIGRyd3hyLXhy LXggmpqaMTAyNCCacg0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxNCCamndkIC8gmpqampqa mpqampqaMiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammdldHR5IJqampqamjEw MTQgdGV4dCAvIJqampqampo3NjUzMDAgLXIteHIteHIteCCamjI3NTA0IJpyDQo+IJpyb290IJqa mppnZXR0eSCampqampoxMDE0IGN0dHkgL2RldiCampqampqamjQ3IGNydy0tLS0tLS0gmpp0dHl2 NSBydw0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxNCCampowIC9kZXYgmpqampqampo0NyBj cnctLS0tLS0tIJqadHR5djUgcncNCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTQgmpqaMSAv ZGV2IJqampqampqaNDcgY3J3LS0tLS0tLSCamnR0eXY1IHJ3DQo+IJpyb290IJqamppnZXR0eSCa mpqampoxMDE0IJqamjIgL2RldiCampqampqamjQ3IGNydy0tLS0tLS0gmpp0dHl2NSBydw0KPiCa cm9vdCCampqaZ2V0dHkgmpqampqaMTAxMyByb290IC8gmpqampqampqampqaMiBkcnd4ci14ci14 IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTMgmpp3ZCAvIJqampqampqa mpqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEz IHRleHQgLyCampqampqaNzY1MzAwIC1yLXhyLXhyLXggmpoyNzUwNCCacg0KPiCacm9vdCCampqa Z2V0dHkgmpqampqaMTAxMyBjdHR5IC9kZXYgmpqampqampo0NiBjcnctLS0tLS0tIJqadHR5djQg cncNCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTMgmpqaMCAvZGV2IJqampqampqaNDYgY3J3 LS0tLS0tLSCamnR0eXY0IHJ3DQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEzIJqamjEgL2Rl diCampqampqamjQ2IGNydy0tLS0tLS0gmpp0dHl2NCBydw0KPiCacm9vdCCampqaZ2V0dHkgmpqa mpqaMTAxMyCampoyIC9kZXYgmpqampqampo0NiBjcnctLS0tLS0tIJqadHR5djQgcncNCj4gmnJv b3QgmpqammdldHR5IJqampqamjEwMTIgcm9vdCAvIJqampqampqampqamjIgZHJ3eHIteHIteCCa mpoxMDI0IJpyDQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEyIJqad2QgLyCampqampqampqa mpoyIGRyd3hyLXhyLXggmpqaMTAyNCCacg0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxMiB0 ZXh0IC8gmpqampqamjc2NTMwMCAtci14ci14ci14IJqaMjc1MDQgmnINCj4gmnJvb3Qgmpqammdl dHR5IJqampqamjEwMTIgY3R0eSAvZGV2IJqampqampqaNDUgY3J3LS0tLS0tLSCamnR0eXYzIHJ3 DQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEyIJqamjAgL2RldiCampqampqamjQ1IGNydy0t LS0tLS0gmpp0dHl2MyBydw0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxMiCampoxIC9kZXYg mpqampqampo0NSBjcnctLS0tLS0tIJqadHR5djMgcncNCj4gmnJvb3QgmpqammdldHR5IJqampqa mjEwMTIgmpqaMiAvZGV2IJqampqampqaNDUgY3J3LS0tLS0tLSCamnR0eXYzIHJ3DQo+IJpyb290 IJqamppnZXR0eSCampqampoxMDExIHJvb3QgLyCampqampqampqampoyIGRyd3hyLXhyLXggmpqa MTAyNCCacg0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxMSCamndkIC8gmpqampqampqampqa MiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTEgdGV4 dCAvIJqampqampo3NjUzMDAgLXIteHIteHIteCCamjI3NTA0IJpyDQo+IJpyb290IJqamppnZXR0 eSCampqampoxMDExIGN0dHkgL2RldiCampqampqamjQ0IGNydy0tLS0tLS0gmpp0dHl2MiBydw0K PiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAxMSCampowIC9kZXYgmpqampqampo0NCBjcnctLS0t LS0tIJqadHR5djIgcncNCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTEgmpqaMSAvZGV2IJqa mpqampqaNDQgY3J3LS0tLS0tLSCamnR0eXYyIHJ3DQo+IJpyb290IJqamppnZXR0eSCampqampox MDExIJqamjIgL2RldiCampqampqamjQ0IGNydy0tLS0tLS0gmpp0dHl2MiBydw0KPiCacm9vdCCa mpqaZ2V0dHkgmpqampqaMTAxMCByb290IC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEw MjQgmnINCj4gmnJvb3QgmpqammdldHR5IJqampqamjEwMTAgmpp3ZCAvIJqampqampqampqamjIg ZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEwIHRleHQg LyCampqampqaNzY1MzAwIC1yLXhyLXhyLXggmpoyNzUwNCCacg0KPiCacm9vdCCampqaZ2V0dHkg mpqampqaMTAxMCBjdHR5IC9kZXYgmpqampqampo0MyBjcnctLS0tLS0tIJqadHR5djEgcncNCj4g mnJvb3QgmpqammdldHR5IJqampqamjEwMTAgmpqaMCAvZGV2IJqampqampqaNDMgY3J3LS0tLS0t LSCamnR0eXYxIHJ3DQo+IJpyb290IJqamppnZXR0eSCampqampoxMDEwIJqamjEgL2RldiCampqa mpqamjQzIGNydy0tLS0tLS0gmpp0dHl2MSBydw0KPiCacm9vdCCampqaZ2V0dHkgmpqampqaMTAx MCCampoyIC9kZXYgmpqampqampo0MyBjcnctLS0tLS0tIJqadHR5djEgcncNCj4gmnJvb3Qgmpqa mmxvZ2luIJqampqamjEwMDkgcm9vdCAvIJqampqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0 IJpyDQo+IJpyb290IJqamppsb2dpbiCampqampoxMDA5IJqad2QgLyCampqampqaMjAzNjQ4MCBk cnd4ci14ci14IJqampo1MTIgmnINCj4gmnJvb3QgmpqammxvZ2luIJqampqamjEwMDkgdGV4dCAv IJqampqampo3NzM1MjIgLXItc3IteHIteCCamjI1MDgwIJpyDQo+IJpyb290IJqamppsb2dpbiCa mpqampoxMDA5IGN0dHkgL2RldiCampqampqamjQyIGNydy0tLS0tLS0gmpp0dHl2MCBydw0KPiCa cm9vdCCampqabG9naW4gmpqampqaMTAwOSCampowIC9kZXYgmpqampqampo0MiBjcnctLS0tLS0t IJqadHR5djAgcncNCj4gmnJvb3QgmpqammxvZ2luIJqampqamjEwMDkgmpqaMSAvZGV2IJqampqa mpqaNDIgY3J3LS0tLS0tLSCamnR0eXYwIHJ3DQo+IJpyb290IJqamppsb2dpbiCampqampoxMDA5 IJqamjIgL2RldiCampqampqamjQyIGNydy0tLS0tLS0gmpp0dHl2MCBydw0KPiCacm9vdCCampqa Y3JvbiCampqampqamjk0NyByb290IC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEwMjQg mnINCj4gmnJvb3QgmpqammNyb24gmpqampqampo5NDcgmpp3ZCAvIJqampqampoyNDYyNzI3IGRy d3hyLXgtLS0gmpqamjUxMiCacg0KPiCacm9vdCCampqaY3JvbiCampqampqamjk0NyB0ZXh0IC8g mpqampqamjc2Mzk4NyAtci14ci14ci14IJqaNDEzNjAgmnINCj4gmnJvb3QgmpqammNyb24gmpqa mpqampo5NDcgmpqaMCAvZGV2IJqampqampqaMjggY3J3LXJ3LXJ3LSCamppudWxsIHJ3DQo+IJpy b290IJqamppjcm9uIJqampqampqaOTQ3IJqamjEgL2RldiCampqampqamjI4IGNydy1ydy1ydy0g mpqabnVsbCBydw0KPiCacm9vdCCampqaY3JvbiCampqampqamjk0NyCampoyIC9kZXYgmpqampqa mpoyOCBjcnctcnctcnctIJqamm51bGwgcncNCj4gmnJvb3QgmpqammNyb24gmpqampqampo5NDcg mpqaMyAvIJqampqampoyNDYyODk2IC1ydy0tLS0tLS0gmpqampqaMyCadw0KPiCac21tc3Agmpqa c2VuZG1haWwgmpqamjk0MSByb290IC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEwMjQg mnINCj4gmnNtbXNwIJqamnNlbmRtYWlsIJqampo5NDEgmpp3ZCAvIJqampqampoyNDYyNzUwIGRy d3hyd3gtLS0gmpqamjUxMiCacg0KPiCac21tc3Agmpqac2VuZG1haWwgmpqamjk0MSB0ZXh0IC8g mpqampqamjc2NTQ0MSAtci14ci1zci14IJo3MDc0OTYgmnINCj4gmj4gc21tc3Agmpqac2VuZG1h aWwgmpqamjk0MSCampowIC9kZXYgmpqampqampoyOCBjcnctcnctcnctIJqamm51bGwgmnINCj4g mnNtbXNwIJqamnNlbmRtYWlsIJqampo5NDEgmpqaMSAvZGV2IJqampqampqaMjggY3J3LXJ3LXJ3 LSCamppudWxsIJp3DQo+IJpzbW1zcCCamppzZW5kbWFpbCCampqaOTQxIJqamjIgL2RldiCampqa mpqamjI4IGNydy1ydy1ydy0gmpqabnVsbCCadw0KPiCac21tc3Agmpqac2VuZG1haWwgmpqamjk0 MSCampozIC8gmpqampqamjI0NjI5MDMgLXJ3LS0tLS0tLSCampqamjQ5IJp3DQo+IJpyb290IJqa mppzZW5kbWFpbCCampqaOTM4IHJvb3QgLyCampqampqampqampoyIGRyd3hyLXhyLXggmpqaMTAy NCCacg0KPiCacm9vdCCampqac2VuZG1haWwgmpqamjkzOCCamndkIC8gmpqampqamjI0NjI3NDcg ZHJ3eHIteHIteCCampqaNTEyIJpyDQo+IJpyb290IJqamppzZW5kbWFpbCCampqaOTM4IHRleHQg LyCampqampqaNzY1NDQxIC1yLXhyLXNyLXggmjcwNzQ5NiCacg0KPiCacm9vdCCampqac2VuZG1h aWwgmpqamjkzOCCampowIC9kZXYgmpqampqampoyOCBjcnctcnctcnctIJqamm51bGwgmnINCj4g mnJvb3QgmpqamnNlbmRtYWlsIJqampo5MzggmpqaMSAvZGV2IJqampqampqaMjggY3J3LXJ3LXJ3 LSCamppudWxsIJp3DQo+IJpyb290IJqamppzZW5kbWFpbCCampqaOTM4IJqamjIgL2RldiCampqa mpqamjI4IGNydy1ydy1ydy0gmpqabnVsbCCadw0KPiCacm9vdCCampqac2VuZG1haWwgmpqamjkz OCCampozKiBpbnRlcm5ldCBzdHJlYW0gdGNwIGZmZmZmZTAwMDI3NDU3YTANCj4gmnJvb3Qgmpqa mnNlbmRtYWlsIJqampo5MzggmpqaNCAvIJqampqampoyNDYyODk4IC1ydy0tLS0tLS0gmpqampo3 OCCadw0KPiCacm9vdCCampqaZGV2ZCCampqampqamjU2MCByb290IC8gmpqampqampqampqaMiBk cnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammRldmQgmpqampqampo1NjAgmpp3ZCAv IJqampqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppkZXZkIJqa mpqampqaNTYwIHRleHQgLyCampqampqaMzc4OTIwIC1yLXhyLXhyLXggmjQyMzMyOCCacg0KPiCa cm9vdCCampqaZGV2ZCCampqampqamjU2MCCampowIC9kZXYgmpqampqampoyOCBjcnctcnctcnct IJqamm51bGwgcncNCj4gmnJvb3QgmpqammRldmQgmpqampqampo1NjAgmpqaMSAvZGV2IJqampqa mpqaMjggY3J3LXJ3LXJ3LSCamppudWxsIHJ3DQo+IJpyb290IJqamppkZXZkIJqampqampqaNTYw IJqamjIgL2RldiCampqampqamjI4IGNydy1ydy1ydy0gmpqabnVsbCBydw0KPiCacm9vdCCampqa ZGV2ZCCampqampqamjU2MCCampozIC9kZXYgmpqampqampqaNiBjcnctLS0tLS0tIJpkZXZjdGwg mnINCj4gmnJvb3QgmpqammRldmQgmpqampqampo1NjAgmpqaNCogbG9jYWwgc3RyZWFtIGZmZmZm ZTAwMDI2Y2VkMjANCj4gmnJvb3QgmpqammRldmQgmpqampqampo1NjAgmpqaNSAvIJqampqampoy NDYyODY3IC1ydy0tLS0tLS0gmpqampqaMyCadw0KPiCacm9vdCCampqaYWRqa2VybnR6IJqamjEw NyByb290IC8gmpqampqampqampqaMiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3Qgmpqa mmFkamtlcm50eiCampoxMDcgmpp3ZCAvIJqampqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0 IJpyDQo+IJpyb290IJqampphZGprZXJudHogmpqaMTA3IHRleHQgLyCampqampqaMzc4ODg5IC1y LXhyLXhyLXggmpqaOTIzMiCacg0KPiCacm9vdCCampqaYWRqa2VybnR6IJqamjEwNyCampowIC9k ZXYgmpqampqampoyOCBjcnctcnctcnctIJqamm51bGwgcncNCj4gmnJvb3QgmpqammFkamtlcm50 eiCampoxMDcgmpqaMSAvZGV2IJqampqampqaMjggY3J3LXJ3LXJ3LSCamppudWxsIHJ3DQo+IJpy b290IJqampphZGprZXJudHogmpqaMTA3IJqamjIgL2RldiCampqampqamjI4IGNydy1ydy1ydy0g mpqabnVsbCBydw0KPiCacm9vdCCampqaaW5pdCCampqampqampqaMSByb290IC8gmpqampqampqa mpqaMiBkcnd4ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3QgmpqammluaXQgmpqampqampqamjEg mpp3ZCAvIJqampqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+IJpyb290IJqamppp bml0IJqampqampqampoxIHRleHQgLyCampqampqaMzc4OTg4IC1yLXhyLXhyLXggmjc3MTU4NCCa cg0KPiCacm9vdCCampqaa2VybmVsIJqampqampqaMCByb290IC8gmpqampqampqampqaMiBkcnd4 ci14ci14IJqamjEwMjQgmnINCj4gmnJvb3Qgmpqammtlcm5lbCCampqampqamjAgmpp3ZCAvIJqa mpqampqampqamjIgZHJ3eHIteHIteCCampoxMDI0IJpyDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0N Cj4gmmRtZXNnDQo+DQo+IJpDb3B5cmlnaHQgKGMpIDE5OTItMjAxMSBUaGUgRnJlZUJTRCBQcm9q ZWN0Lg0KPiCaQ29weXJpZ2h0IChjKSAxOTc5LCAxOTgwLCAxOTgzLCAxOTg2LCAxOTg4LCAxOTg5 LCAxOTkxLCAxOTkyLCAxOTkzLCAxOTk0DQo+IJqampqampqamlRoZSBSZWdlbnRzIG9mIHRoZSBV bml2ZXJzaXR5IG9mIENhbGlmb3JuaWEuIEFsbCByaWdodHMgcmVzZXJ2ZWQuDQo+IJpGcmVlQlNE IGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZyZWVCU0QgRm91bmRhdGlvbi4NCj4g mkZyZWVCU0QgOS4wLVBSRVJFTEVBU0UgIzc6IFdlZCBOb3YgMTYgMTg6MzY6NDQgRUVUIDIwMTEN Cj4gmpqamppyb290QGJzZC5teS5kb21haW46L3Vzci9vYmovdXNyL3NyYy9zeXMvR0VORVJJQyBh bWQ2NA0KPiCaQ1BVOiBBTUQgUGhlbm9tKHRtKSBJSSBYNiAxMDkwVCBQcm9jZXNzb3IgKDMwNTEu NjItTUh6IEs4LWNsYXNzIENQVSkNCj4gmpqaT3JpZ2luID0gIkF1dGhlbnRpY0FNRCIgmklkID0g MHgxMDBmYTAgmkZhbWlseSA9IDEwIJpNb2RlbCA9IGEgmlN0ZXBwaW5nID0gMA0KPiCamppGZWF0 dXJlcz0weDc4M2ZiZmY8RlBVLFZNRSxERSxQU0UsVFNDLE1TUixQQUUsTUNFLENYOCxBUElDLFNF UCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsTU1YLEZYU1IsU1NFLFNTRTI+DQo+IJqamkZl YXR1cmVzMj0weDk8U1NFMyxNT04+DQo+IJqamkFNRCBGZWF0dXJlcz0weGVhMTAwODAwPFNZU0NB TEwsTlgsRkZYU1IsUkRUU0NQLExNLDNETm93ISssM0ROb3chPg0KPiCamppBTUQgRmVhdHVyZXMy PTB4MTE8TEFIRixDUjg+DQo+IJpyZWFsIG1lbW9yeSCaPSAxMDczNjc2Mjg4ICgxMDIzIE1CKQ0K PiCaYXZhaWwgbWVtb3J5ID0gMTAxMTE4NzcxMiAoOTY0IE1CKQ0KPiCaRXZlbnQgdGltZXIgIkxB UElDIiBxdWFsaXR5IDQwMA0KPiCaQUNQSSBBUElDIFRhYmxlOiA8VkJPWCCamlZCT1hBUElDPg0K PiCaV0FSTklORzogVklNQUdFICh2aXJ0dWFsaXplZCBuZXR3b3JrIHN0YWNrKSBpcyBhIGhpZ2hs eSBleHBlcmltZW50YWwgZmVhdHVyZS4NCj4gmmlvYXBpYzA6IENoYW5naW5nIEFQSUMgSUQgdG8g MQ0KPiCaaW9hcGljMCA8VmVyc2lvbiAxLjE+IGlycXMgMC0yMyBvbiBtb3RoZXJib2FyZA0KPiCa a2JkMSBhdCBrYmRtdXgwDQo+IJphY3BpMDogPFZCT1ggVkJPWFhTRFQ+IG9uIG1vdGhlcmJvYXJk DQo+IJphY3BpMDogUG93ZXIgQnV0dG9uIChmaXhlZCkNCj4gmmFjcGkwOiBTbGVlcCBCdXR0b24g KGZpeGVkKQ0KPiCaVGltZWNvdW50ZXIgIkFDUEktZmFzdCIgZnJlcXVlbmN5IDM1Nzk1NDUgSHog cXVhbGl0eSA5MDANCj4gmmFjcGlfdGltZXIwOiA8MzItYml0IHRpbWVyIGF0IDMuNTc5NTQ1TUh6 PiBwb3J0IDB4NDAwOC0weDQwMGIgb24gYWNwaTANCj4gmmNwdTA6IDxBQ1BJIENQVT4gb24gYWNw aTANCj4gmnBjaWIwOiA8QUNQSSBIb3N0LVBDSSBicmlkZ2U+IHBvcnQgMHhjZjgtMHhjZmYgb24g YWNwaTANCj4gmnBjaTA6IDxBQ1BJIFBDSSBidXM+IG9uIHBjaWIwDQo+IJppc2FiMDogPFBDSS1J U0EgYnJpZGdlPiBhdCBkZXZpY2UgMS4wIG9uIHBjaTANCj4gmmlzYTA6IDxJU0EgYnVzPiBvbiBp c2FiMA0KPiCaYXRhcGNpMDogPEludGVsIFBJSVg0IFVETUEzMyBjb250cm9sbGVyPiBwb3J0IDB4 MWYwLTB4MWY3LDB4M2Y2LDB4MTcwLTB4MTc3LDB4Mzc2LDB4ZDAwMC0weGQwMGYgYXQgZGV2aWNl IDEuMSBvbiBwY2kwDQo+IJphdGEwOiA8QVRBIGNoYW5uZWwgMD4gb24gYXRhcGNpMA0KPiCaYXRh MTogPEFUQSBjaGFubmVsIDE+IG9uIGF0YXBjaTANCj4gmnZnYXBjaTA6IDxWR0EtY29tcGF0aWJs ZSBkaXNwbGF5PiBtZW0gMHhlMDAwMDAwMC0weGUwZmZmZmZmIGlycSAxOCBhdCBkZXZpY2UgMi4w IG9uIHBjaTANCj4gmmVtMDogPEludGVsKFIpIFBSTy8xMDAwIExlZ2FjeSBOZXR3b3JrIENvbm5l Y3Rpb24gMS4wLjM+IHBvcnQgMHhkMDEwLTB4ZDAxNyBtZW0gMHhmMDAwMDAwMC0weGYwMDFmZmZm IGlycSAxOSBhdCBkZXZpY2UgMy4wIG9uIHBjaTANCj4gmmVtMDogRXRoZXJuZXQgYWRkcmVzczog MDg6MDA6Mjc6NGI6NmY6ZjANCj4gmnBjaTA6IDxiYXNlIHBlcmlwaGVyYWw+IGF0IGRldmljZSA0 LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkNCj4gmnBjbTA6IDxJbnRlbCBJQ0ggKDgyODAxQUEpPiBw b3J0IDB4ZDEwMC0weGQxZmYsMHhkMjAwLTB4ZDIzZiBpcnEgMjEgYXQgZGV2aWNlIDUuMCBvbiBw Y2kwDQo+IJpwPiBjbTA6IDxTaWdtYVRlbCBTVEFDOTcwMC84My84NCBBQzk3IENvZGVjPg0KPiCa b2hjaTA6IDxPSENJIChnZW5lcmljKSBVU0IgY29udHJvbGxlcj4gbWVtIDB4ZjA4MDQwMDAtMHhm MDgwNGZmZiBpcnEgMjIgYXQgZGV2aWNlIDYuMCBvbiBwY2kwDQo+IJp1c2J1czA6IDxPSENJIChn ZW5lcmljKSBVU0IgY29udHJvbGxlcj4gb24gb2hjaTANCj4gmnBjaTA6IDxicmlkZ2U+IGF0IGRl dmljZSA3LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkNCj4gmmFjcGlfYWNhZDA6IDxBQyBBZGFwdGVy PiBvbiBhY3BpMA0KPiCaYXRrYmRjMDogPEtleWJvYXJkIGNvbnRyb2xsZXIgKGk4MDQyKT4gcG9y dCAweDYwLDB4NjQgaXJxIDEgb24gYWNwaTANCj4gmmF0a2JkMDogPEFUIEtleWJvYXJkPiBpcnEg MSBvbiBhdGtiZGMwDQo+IJprYmQwIGF0IGF0a2JkMA0KPiCaYXRrYmQwOiBbR0lBTlQtTE9DS0VE XQ0KPiCacHNtMDogPFBTLzIgTW91c2U+IGlycSAxMiBvbiBhdGtiZGMwDQo+IJpwc20wOiBbR0lB TlQtTE9DS0VEXQ0KPiCacHNtMDogbW9kZWwgSW50ZWxsaU1vdXNlIEV4cGxvcmVyLCBkZXZpY2Ug SUQgNA0KPiCaYXR0aW1lcjA6IDxBVCB0aW1lcj4gcG9ydCAweDQwLTB4NDMsMHg1MC0weDUzIG9u IGFjcGkwDQo+IJpUaW1lY291bnRlciAiaTgyNTQiIGZyZXF1ZW5jeSAxMTkzMTgyIEh6IHF1YWxp dHkgMA0KPiCaRXZlbnQgdGltZXIgImk4MjU0IiBmcmVxdWVuY3kgMTE5MzE4MiBIeiBxdWFsaXR5 IDEwMA0KPiCac2MwOiA8U3lzdGVtIGNvbnNvbGU+IGF0IGZsYWdzIDB4MTAwIG9uIGlzYTANCj4g mnNjMDogVkdBIDwxNiB2aXJ0dWFsIGNvbnNvbGVzLCBmbGFncz0weDMwMD4NCj4gmnZnYTA6IDxH ZW5lcmljIElTQSBWR0E+IGF0IHBvcnQgMHgzYzAtMHgzZGYgaW9tZW0gMHhhMDAwMC0weGJmZmZm IG9uIGlzYTANCj4gmmF0cnRjMDogPEFUIHJlYWx0aW1lIGNsb2NrPiBhdCBwb3J0IDB4NzAgaXJx IDggb24gaXNhMA0KPiCaRXZlbnQgdGltZXIgIlJUQyIgZnJlcXVlbmN5IDMyNzY4IEh6IHF1YWxp dHkgMA0KPiCacHBjMDogY2Fubm90IHJlc2VydmUgSS9PIHBvcnQgcmFuZ2UNCj4gmlRpbWVjb3Vu dGVycyB0aWNrIGV2ZXJ5IDEwLjAwMCBtc2VjDQo+IJpwY20wOiBtZWFzdXJlZCBhYzk3IGxpbmsg cmF0ZSBhdCA4MDQwODMgSHoNCj4gmnVzYnVzMDogMTJNYnBzIEZ1bGwgU3BlZWQgVVNCIHYxLjAN Cj4gmnVnZW4wLjE6IDxBcHBsZT4gYXQgdXNidXMwDQo+IJp1aHViMDogPEFwcGxlIE9IQ0kgcm9v dCBIVUIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRkciAxPiBvbiB1c2J1czANCj4gmmFk YTAgYXQgYXRhMCBidXMgMCBzY2J1czAgdGFyZ2V0IDAgbHVuIDANCj4gmmFkYTA6IDxWQk9YIEhB UkRESVNLIDEuMD4gQVRBLTYgZGV2aWNlDQo+IJphZGEwOiAzMy4zMDBNQi9zIHRyYW5zZmVycyAo VURNQTIsIFBJTyA2NTUzNmJ5dGVzKQ0KPiCaYWRhMDogNDQ0ODZNQiAoOTExMDczMjggNTEyIGJ5 dGUgc2VjdG9yczogMTZIIDYzUy9UIDE2MzgzQykNCj4gmmFkYTA6IFByZXZpb3VzbHkgd2FzIGtu b3duIGFzIGFkMA0KPiCaY2QwIGF0IGF0YTEgYnVzIDAgc2NidXMxIHRhcmdldCAwIGx1biAwDQo+ IJpjZDA6IDxWQk9YIENELVJPTSAxLjA+IFJlbW92YWJsZSBDRC1ST00gU0NTSS0wIGRldmljZQ0K PiCaY2QwOiAzMy4zMDBNQi9zIHRyYW5zZmVycyAoVURNQTIsIEFUQVBJIDEyYnl0ZXMsIFBJTyA2 NTUzNGJ5dGVzKQ0KPiCaY2QwOiBBdHRlbXB0IHRvIHF1ZXJ5IGRldmljZSBzaXplIGZhaWxlZDog Tk9UIFJFQURZLCBNZWRpdW0gbm90IHByZXNlbnQNCj4gmlRpbWVjb3VudGVyICJUU0MiIGZyZXF1 ZW5jeSAzMDUxNjIxMDA5IEh6IHF1YWxpdHkgODAwDQo+IJpSb290IG1vdW50IHdhaXRpbmcgZm9y OiB1c2J1czANCj4gmnVodWIwOiA4IHBvcnRzIHdpdGggOCByZW1vdmFibGUsIHNlbGYgcG93ZXJl ZA0KPiCaVHJ5aW5nIHRvIG1vdW50IHJvb3QgZnJvbSB1ZnM6L2Rldi9hZGEwcDIgW3J3XS4uLg0K PiCaU2V0dGluZyBob3N0dXVpZDogYjhlNjNiZjQtNzc5ZC00MzZkLWEyMGEtYzM3N2Y2M2ZkNTNi Lg0KPiCaU2V0dGluZyBob3N0aWQ6IDB4ZjljZWRmYzUuDQo+IJpFbnRyb3B5IGhhcnZlc3Rpbmc6 IGludGVycnVwdHMgZXRoZXJuZXQgcG9pbnRfdG9fcG9pbnQga2lja3N0YXJ0Lg0KPiCaU3RhcnRp bmcgZmlsZSBzeXN0ZW0gY2hlY2tzOg0KPiCaL2Rldi9hZGEwcDI6IEZJTEUgU1lTVEVNIENMRUFO OyBTS0lQUElORyBDSEVDS1MNCj4gmi9kZXYvYWRhMHAyOiBjbGVhbiwgNzk5NTU1MSBmcmVlICgz NDM1OSBmcmFncywgOTk1MTQ5IGJsb2NrcywgMC4zJSBmcmFnbWVudGF0aW9uKQ0KPiCaTW91bnRp bmcgbG9jYWwgZmlsZSBzeXN0ZW1zOi4NCj4gmlNldHRpbmcgaG9zdG5hbWU6IGJzZC5teS5kb21h aW4uDQo+IJpTdGFydGluZyBOZXR3b3JrOiBsbzAgZW0wLg0KPiCabG8wOiBmbGFncz04MDQ5PFVQ LExPT1BCQUNLLFJVTk5JTkcsTVVMVElDQVNUPiBtZXRyaWMgMCBtdHUgMTYzODQNCj4gmpqampqa mpqab3B0aW9ucz0zPFJYQ1NVTSxUWENTVU0+DQo+IJqampqampqammluZXQ2IDo6MSBwcmVmaXhs ZW4gMTI4DQo+IJqampqampqammluZXQ2IGZlODA6OjElbG8wIHByZWZpeGxlbiA2NCBzY29wZWlk IDB4Mw0KPiCampqampqampppbmV0IDEyNy4wLjAuMSBuZXRtYXNrIDB4ZmYwMDAwMDANCj4gmpqa mpqampqabmQ2IG9wdGlvbnM9MjE8UEVSRk9STU5VRCxBVVRPX0xJTktMT0NBTD4NCj4gmmVtMDog ZmxhZ3M9ODgwMjxCUk9BRENBU1QsU0lNUExFWCxNVUxUSUNBU1Q+IG1ldHJpYyAwIG10dSAxNTAw DQo+IJqampqampqamm9wdGlvbnM9OWI8UlhDU1VNLFRYQ1NVTSxWTEFOX01UVSxWTEFOX0hXVEFH R0lORyxWTEFOX0hXQ1NVTT4NCj4gmpqampqampqaZXRoZXIgMDg6MDA6Mjc6NGI6NmY6ZjANCj4g mpqampqampqabmQ2IG9wdGlvbnM9Mjk8UEVSRk9STU5VRCxJRkRJU0FCTEVELEFVVE9fTElOS0xP Q0FMPg0KPiCampqampqampptZWRpYTogRXRoZXJuZXQgYXV0b3NlbGVjdCAoMTAwMGJhc2VUIDxm dWxsLWR1cGxleD4pDQo+IJqampqampqamnN0YXR1czogYWN0aXZlDQo+IJpTdGFydGluZyBkZXZk Lg0KPiCaU3RhcnRpbmcgTmV0d29yazogZW0wLg0KPiCaZW0wOiBmbGFncz04ODAyPEJST0FEQ0FT VCxTSU1QTEVYLE1VTFRJQ0FTVD4gbWV0cmljIDAgbXR1IDE1MDANCj4gmpqampqampqab3B0aW9u cz05YjxSWENTVU0sVFhDU1VNLFZMQU5fTVRVLFZMQU5fSFdUQUdHSU5HLFZMQU5fSFdDU1VNPg0K PiCampqampqamppldGhlciAwODowMDoyNzo0Yjo2ZjpmMA0KPiCampqampqamppuZDYgb3B0aW9u cz0yOTxQRVJGT1JNTlVELElGRElTQUJMRUQsQVVUT19MSU5LTE9DQUw+DQo+IJqampqampqamm1l ZGlhOiBFdGhlcm5ldCBhdXRvc2VsZWN0ICgxMDAwYmFzZVQgPGZ1bGwtZHVwbGV4PikNCj4gmpqa mpqampqac3RhdHVzOiBhY3RpdmUNCj4gmlN0YXJ0aW5nIE5ldHdvcms6IHVzYnVzMC4NCj4gmj4g YWRkIG5ldCA6OmZmZmY6MC4wLjAuMDogZ2F0ZXdheSA6OjENCj4gmmFkZCBuZXQgOjowLjAuMC4w OiBnYXRld2F5IDo6MQ0KPiCaYWRkIG5ldCBmZTgwOjo6IGdhdGV3YXkgOjoxDQo+IJphZGQgbmV0 IGZmMDI6OjogZ2F0ZXdheSA6OjENCj4gmkVMRiBsZGNvbmZpZyBwYXRoOiAvbGliIC91c3IvbGli IC91c3IvbGliL2NvbXBhdCAvdXNyL2xvY2FsL2xpYg0KPiCaMzItYml0IGNvbXBhdGliaWxpdHkg bGRjb25maWcgcGF0aDogL3Vzci9saWIzMg0KPiCaQ3JlYXRpbmcgYW5kL29yIHRyaW1taW5nIGxv ZyBmaWxlcy4NCj4gmk5vIGNvcmUgZHVtcHMgZm91bmQuDQo+IJpDbGVhcmluZyAvdG1wIChYIHJl bGF0ZWQpLg0KPiCaVXBkYXRpbmcgbW90ZDouDQo+IJpDb25maWd1cmluZyBzeXNjb25zOiBrZXlt YXAgYmxhbmt0aW1lLg0KPiCaU3RhcnRpbmcgY3Jvbi4NCj4gmlN0YXJ0aW5nIGJhY2tncm91bmQg ZmlsZSBzeXN0ZW0gY2hlY2tzIGluIDYwIHNlY29uZHMuDQo+DQo+IJpXZWQgTm92IDE2IDE4OjQ4 OjUwIEVFVCAyMDExDQo+IJpJUCBGaWx0ZXI6IHY0LjEuMjggaW5pdGlhbGl6ZWQuIJpEZWZhdWx0 ID0gcGFzcyBhbGwsIExvZ2dpbmcgPSBlbmFibGVkDQo+DQo+IJpGYXRhbCB0cmFwIDEyOiBwYWdl IGZhdWx0IHdoaWxlIGluIGtlcm5lbCBtb2RlDQo+IJpjcHVpZCA9IDA7IGFwaWMgaWQgPSAwMA0K PiCaZmF1bHQgdmlydHVhbCBhZGRyZXNzID0gMHgyOA0KPiCaZmF1bHQgY29kZSA9IHN1cGVydmlz b3IgcmVhZCBkYXRhLCBwYWdlIG5vdCBwcmVzZW50DQo+IJppbnN0cnVjdGlvbiBwb2ludGVyID0g MHgyMDoweGZmZmZmZmZmODA4ZTI2MWENCj4gmnN0YWNrIHBvaW50ZXIgmpqampqamj0gMHgyODow eGZmZmZmZjgwMDAzOTM3ZjANCj4gmmZyYW1lIHBvaW50ZXIgmpqampqamj0gMHgyODoweGZmZmZm ZjgwMDAzOTM4MTANCj4gmmNvZGUgc2VnbWVudCA9IGJhc2UgMHgwLCBsaW1pdCAweGZmZmZmLCB0 eXBlIDB4MWINCj4gmpqampqampqampqampqampqampqampqamj0gRFBMIDAsIHByZXMgMSwgbG9u ZyAxLCBkZWYzMiAwLCBncmFuIDENCj4gmnByb2Nlc3NvciBlZmxhZ3MgPSBpbnRlcnJ1cHQgZW5h YmxlZCwgcmVzdW1lLCBJT1BMID0gMA0KPiCaY3VycmVudCBwcm9jZXNzID0gMTA4MyAoaXBuYXQp DQo+IJp0cmFwIG51bWJlciA9IDEyDQo+IJpwYW5pYzogcGFnZSBmYXVsdA0KPiCaY3B1aWQgPSAw DQo+IJpLREI6IHN0YWNrIGJhY2t0cmFjZToNCj4gmiMwIDB4ZmZmZmZmZmY4MDg2OTRhZSBhdCBr ZGJfYmFja3RyYWNlKzB4NWUNCj4gmiMxIDB4ZmZmZmZmZmY4MDgzM2U0NyBhdCBwYW5pYysweDE4 Nw0KPiCaIzIgMHhmZmZmZmZmZjgwYjI5NzQwIGF0IHRyYXBfZmF0YWwrMHgyOTANCj4gmiMzIDB4 ZmZmZmZmZmY4MGIyOWE4OSBhdCB0cmFwX3BmYXVsdCsweDFmOQ0KPiCaIzQgMHhmZmZmZmZmZjgw YjI5ZjRmIGF0IHRyYXArMHgzZGYNCj4gmiM1IDB4ZmZmZmZmZmY4MGIxNDQ3ZiBhdCBjYWxsdHJh cCsweDgNCj4gmiM2IDB4ZmZmZmZmZmY4MTYyZDg2YSBhdCBmcl9yZXNvbHZlbmljKzB4MWENCj4g miM3IDB4ZmZmZmZmZmY4MTYxNjliNSBhdCBuYXRfcmVzb2x2ZXJ1bGUrMHgyNQ0KPiCaIzggMHhm ZmZmZmZmZjgxNjE3OTQzIGF0IGZyX25hdF9pb2N0bCsweGYwMw0KPiCaIzkgMHhmZmZmZmZmZjgw NzU1MTRiIGF0IGRldmZzX2lvY3RsX2YrMHg3Yg0KPiCaIzEwIDB4ZmZmZmZmZmY4MDg3YWJhNSBh dCBrZXJuX2lvY3RsKzB4MTE1DQo+IJojMTEgMHhmZmZmZmZmZjgwODdhZGRkIGF0IHN5c19pb2N0 bCsweGZkDQo+IJojMTIgMHhmZmZmZmZmZjgwYjI5MDMwIGF0IGFtZDY0X3N5c2NhbGwrMHg0NTAN Cj4gmiMxMyAweGZmZmZmZmZmODBiMTQ3NjcgYXQgWGZhc3Rfc3lzY2FsbCsweGY3DQo+IJpVcHRp bWU6IDNtMTdzDQo+IJpEdW1waW5nIDgwIG91dCBvZiAxMDA1IE1COi4uMjAlLi40MCUuLjYwJS4u ODAlLi4xMDAlDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmmtlcm5lbCBjb25maWcNCj4NCj4g mm9wdGlvbnMgQ09ORklHX0FVVE9HRU5FUkFURUQNCj4gmmlkZW50IEdFTkVSSUMNCj4gmm1hY2hp bmUgYW1kNjQNCj4gmmNwdSBIQU1NRVINCj4gmm1ha2VvcHRpb25zIERFQlVHPS1nDQo+IJpvcHRp b25zIFZJTUFHRQ0KPiCab3B0aW9ucyBVU0JfREVCVUcNCj4gmm9wdGlvbnMgQUhfU1VQUE9SVF9B UjU0MTYNCj4gmm9wdGlvbnMgSUVFRTgwMjExX1NVUFBPUlRfTUVTSA0KPiCab3B0aW9ucyBJRUVF ODAyMTFfQU1QRFVfQUdFDQo+IJpvcHRpb25zIElFRUU4MDIxMV9ERUJVRw0KPiCab3B0aW9ucyBT Q19QSVhFTF9NT0RFDQo+IJpvcHRpb25zIEFIRF9SRUdfUFJFVFRZX1BSSU5UDQo+IJpvcHRpb25z IEFIQ19SRUdfUFJFVFRZX1BSSU5UDQo+IJpvcHRpb25zIEFUQV9TVEFUSUNfSUQNCj4gmm9wdGlv bnMgQVRBX0NBTQ0KPiCab3B0aW9ucyBTTVANCj4gmm9wdGlvbnMgS0RCX1RSQUNFDQo+IJpvcHRp b25zIEtEQg0KPiCab3B0aW9ucyBJTkNMVURFX0NPTkZJR19GSUxFDQo+IJpvcHRpb25zIE1BQw0K PiCab3B0aW9ucyBBVURJVA0KPiCab3B0aW9ucyBIV1BNQ19IT09LUw0KPiCab3B0aW9ucyBLQkRf SU5TVEFMTF9DREVWDQo+IJpvcHRpb25zIFBSSU5URl9CVUZSX1NJWkU9MTI4DQo+IJpvcHRpb25z IF9LUE9TSVhfUFJJT1JJVFlfU0NIRURVTElORw0KPiCab3B0aW9ucyBTWVNWU0VNDQo+IJpvcHRp b25zIFNZU1ZNU0cNCj4gmm9wdGlvbnMgU1lTVlNITQ0KPiCab3B0aW9ucyBTVEFDSw0KPiCab3B0 aW9ucyBLVFJBQ0UNCj4gmm9wdGlvbnMgU0NTSV9ERUxBWT01MDAwDQo+IJpvcHRpb25zIENPTVBB VF9GUkVFQlNENw0KPiCab3B0aW9ucyBDT01QQVRfRlJFRUJTRDYNCj4gmm9wdGlvbnMgQ09NUEFU X0ZSRUVCU0Q1DQo+IJpvcHRpb25zIENPTVBBVF9GUkVFQlNENA0KPiCab3B0aW9ucyBDT01QQVRf RlJFRUJTRDMyDQo+IJpvcHRpb25zIEdFT01fTEFCRUwNCj4gmm9wdGlvbnMgR0VPTV9QQVJUX0dQ VA0KPiCab3B0aW9ucyBQU0VVRE9GUw0KPiCab3B0aW9ucyBQUk9DRlMNCj4gmm9wdGlvbnMgQ0Q5 NjYwDQo+IJpvcHRpb25zIE1TRE9TRlMNCj4gmm9wdGlvbnMgTkZTX1JPT1QNCj4gmm9wdGlvbnMg TkZTTE9DS0QNCj4gmm9wdGlvbnMgTkZTRA0KPiCab3B0aW9ucyBORlNDTA0KPiCab3B0aW9ucyBN RF9ST09UDQo+IJpvcHRpb25zIFVGU19HSk9VUk5BTA0KPiCab3B0aW9ucyBVRlNfRElSSEFTSA0K PiCab3B0aW9ucyBVRlNfQUNMDQo+IJpvcHRpb25zIFNPRlRVUERBVEVTDQo+IJpvcHRpb25zIEZG Uw0KPiCab3B0aW9ucyBTQ1RQDQo+IJpvcHRpb25zIElORVQ2DQo+IJpvcHRpb25zIElORVQNCj4g mm9wdGlvbnMgUFJFRU1QVElPTg0KPiCab3B0aW9ucyBTQ0hFRF9VTEUNCj4gmm9wdGlvbnMgTkVX X1BDSUINCj4gmm9wdGlvbnMgR0VPTV9QQVJUX01CUg0KPiCab3B0aW9ucyBHRU9NX1BBUlRfRUJS X0NPTVBBVA0KPiCab3B0aW9ucyBHRU9NX1BBUlRfRUJSDQo+IJpvcHRpb25zIEdFT01fUEFSVF9C U0QNCj4gmmRldmljZSBpc2ENCj4gmmRldmljZSBtZW0NCj4gmmRldmljZSBpbw0KPiCaZGV2aWNl IHVhcnRfbnM4MjUwDQo+IJpkZXZpY2UgY3B1ZnJlcQ0KPiCaZGV2aWNlIGFjcGkNCj4gmmRldmlj ZSBwY2kNCj4gmmRldmljZSBmZGMNCj4gmmRldmljZSBhaGNpDQo+IJpkZXZpY2UgYXRhDQo+IJpk ZXZpY2UgbXZzDQo+IJpkZXZpY2Ugc2lpcw0KPiCaZGV2aWNlIGFoYw0KPiCaZGV2aWNlIGFoZA0K PiCaZGV2aWNlIGVzcA0KPiCaZGV2aWNlIGhwdGlvcA0KPiCaZGV2aWNlIGlzcA0KPiCaZGV2aWNl IG1wdA0KPiCaZGV2aWNlIG1wcw0KPiCaZGV2aWNlIHN5bQ0KPiCaZGV2aWNlIHRybQ0KPiCaZGV2 aWNlIGFkdg0KPiCaZGV2aWNlIGFkdw0KPiCaZGV2aWNlIGFpYw0KPiCaZGV2aWNlIGJ0DQo+IJpk ZXZpY2Ugc2NidXMNCj4gmmRldmljZSBjaA0KPiCaZGV2aWNlIGRhDQo+IJpkZXZpY2Ugc2ENCj4g mmRldmljZSBjZA0KPiCaZGV2aWNlIHBhc3MNCj4gmmRldmljZSBzZXMNCj4gmmRldmljZSBhbXIN Cj4gmmQ+IGV2aWNlIGFyY21zcg0KPiCaZGV2aWNlIGNpc3MNCj4gmmRldmljZSBkcHQNCj4gmmRl dmljZSBocHRtdg0KPiCaZGV2aWNlIGhwdHJyDQo+IJpkZXZpY2UgaWlyDQo+IJpkZXZpY2UgaXBz DQo+IJpkZXZpY2UgbWx5DQo+IJpkZXZpY2UgdHdhDQo+IJpkZXZpY2UgYWFjDQo+IJpkZXZpY2Ug YWFjcA0KPiCaZGV2aWNlIGlkYQ0KPiCaZGV2aWNlIG1maQ0KPiCaZGV2aWNlIG1seA0KPiCaZGV2 aWNlIHR3ZQ0KPiCaZGV2aWNlIHR3cw0KPiCaZGV2aWNlIGF0a2JkYw0KPiCaZGV2aWNlIGF0a2Jk DQo+IJpkZXZpY2UgcHNtDQo+IJpkZXZpY2Uga2JkbXV4DQo+IJpkZXZpY2UgdmdhDQo+IJpkZXZp Y2Ugc3BsYXNoDQo+IJpkZXZpY2Ugc2MNCj4gmmRldmljZSBhZ3ANCj4gmmRldmljZSBjYmINCj4g mmRldmljZSBwY2NhcmQNCj4gmmRldmljZSBjYXJkYnVzDQo+IJpkZXZpY2UgdWFydA0KPiCaZGV2 aWNlIHBwYw0KPiCaZGV2aWNlIHBwYnVzDQo+IJpkZXZpY2UgbHB0DQo+IJpkZXZpY2UgcGxpcA0K PiCaZGV2aWNlIHBwaQ0KPiCaZGV2aWNlIHB1Yw0KPiCaZGV2aWNlIGJ4ZQ0KPiCaZGV2aWNlIGRl DQo+IJpkZXZpY2UgZW0NCj4gmmRldmljZSBpZ2INCj4gmmRldmljZSBpeGdiZQ0KPiCaZGV2aWNl IGxlDQo+IJpkZXZpY2UgdGkNCj4gmmRldmljZSB0eHANCj4gmmRldmljZSB2eA0KPiCaZGV2aWNl IG1paWJ1cw0KPiCaZGV2aWNlIGFlDQo+IJpkZXZpY2UgYWdlDQo+IJpkZXZpY2UgYWxjDQo+IJpk ZXZpY2UgYWxlDQo+IJpkZXZpY2UgYmNlDQo+IJpkZXZpY2UgYmZlDQo+IJpkZXZpY2UgYmdlDQo+ IJpkZXZpY2UgZGMNCj4gmmRldmljZSBldA0KPiCaZGV2aWNlIGZ4cA0KPiCaZGV2aWNlIGptZQ0K PiCaZGV2aWNlIGxnZQ0KPiCaZGV2aWNlIG1zaw0KPiCaZGV2aWNlIG5mZQ0KPiCaZGV2aWNlIG5n ZQ0KPiCaZGV2aWNlIHBjbg0KPiCaZGV2aWNlIHJlDQo+IJpkZXZpY2UgcmwNCj4gmmRldmljZSBz Zg0KPiCaZGV2aWNlIHNnZQ0KPiCaZGV2aWNlIHNpcw0KPiCaZGV2aWNlIHNrDQo+IJpkZXZpY2Ug c3RlDQo+IJpkZXZpY2Ugc3RnZQ0KPiCaZGV2aWNlIHRsDQo+IJpkZXZpY2UgdHgNCj4gmmRldmlj ZSB2Z2UNCj4gmmRldmljZSB2cg0KPiCaZGV2aWNlIHdiDQo+IJpkZXZpY2UgeGwNCj4gmmRldmlj ZSBjcw0KPiCaZGV2aWNlIGVkDQo+IJpkZXZpY2UgZXgNCj4gmmRldmljZSBlcA0KPiCaZGV2aWNl IGZlDQo+IJpkZXZpY2Ugc24NCj4gmmRldmljZSB4ZQ0KPiCaZGV2aWNlIHdsYW4NCj4gmmRldmlj ZSB3bGFuX3dlcA0KPiCaZGV2aWNlIHdsYW5fY2NtcA0KPiCaZGV2aWNlIHdsYW5fdGtpcA0KPiCa ZGV2aWNlIHdsYW5fYW1ycg0KPiCaZGV2aWNlIGFuDQo+IJpkZXZpY2UgYXRoDQo+IJpkZXZpY2Ug YXRoX3BjaQ0KPiCaZGV2aWNlIGF0aF9oYWwNCj4gmmRldmljZSBhdGhfcmF0ZV9zYW1wbGUNCj4g mmRldmljZSBpcHcNCj4gmmRldmljZSBpd2kNCj4gmmRldmljZSBpd24NCj4gmmRldmljZSBtYWxv DQo+IJpkZXZpY2UgbXdsDQo+IJpkZXZpY2UgcmFsDQo+IJpkZXZpY2Ugd2kNCj4gmmRldmljZSB3 cGkNCj4gmmRldmljZSBsb29wDQo+IJpkZXZpY2UgcmFuZG9tDQo+IJpkZXZpY2UgZXRoZXINCj4g mmRldmljZSB2bGFuDQo+IJpkZXZpY2UgdHVuDQo+IJpkZXZpY2UgcHR5DQo+IJpkZXZpY2UgbWQN Cj4gmmRldmljZSBnaWYNCj4gmmRldmljZSBmYWl0aA0KPiCaZGV2aWNlIGZpcm13YXJlDQo+IJpk ZXZpY2UgYnBmDQo+IJpkZXZpY2UgdWhjaQ0KPiCaZGV2aWNlIG9oY2kNCj4gmmRldmljZSBlaGNp DQo+IJpkZXZpY2UgeGhjaQ0KPiCaZGV2aWNlIHVzYg0KPiCaZGV2aWNlIHVoaWQNCj4gmmRldmlj ZSB1a2JkDQo+IJpkZXZpY2UgdWxwdA0KPiCaZGV2aWNlIHVtYXNzDQo+IJpkZXZpY2UgdW1zDQo+ IJpkZXZpY2UgdXJpbw0KPiCaZGV2aWNlIHUzZw0KPiCaZGV2aWNlIHVhcmsNCj4gmmRldmljZSB1 YnNhDQo+IJpkZXZpY2UgdWZ0ZGkNCj4gmmRldmljZSB1aXBhcQ0KPiCaZGV2aWNlIHVwbGNvbQ0K PiCaZGV2aWNlIHVzbGNvbQ0KPiCaZGV2aWNlIHV2aXNvcg0KPiCaZGV2aWNlIHV2c2NvbQ0KPiCa ZGV2aWNlIGF1ZQ0KPiCaZGV2aWNlIGF4ZQ0KPiCaZGV2aWNlIGNkY2UNCj4gmmRldmljZSBjdWUN Cj4gmmRldmljZSBrdWUNCj4gmmRldmljZSBydWUNCj4gmmRldmljZSB1ZGF2DQo+IJpkZXZpY2Ug cnVtDQo+IJpkZXZpY2UgcnVuDQo+IJpkZXZpY2UgdWF0aA0KPiCaZGV2aWNlIHVwZ3QNCj4gmmRl dmljZSB1cmFsDQo+IJpkZXZpY2UgdXJ0dw0KPiCaZGV2aWNlIHp5ZA0KPiCaZGV2aWNlIGZpcmV3 aXJlDQo+IJpkZXZpY2UgZndlDQo+IJpkZXZpY2UgZndpcA0KPiCaZGV2aWNlIGRjb25zDQo+IJpk ZXZpY2UgZGNvbnNfY3JvbQ0KPiCaZGV2aWNlIHNvdW5kDQo+IJpkZXZpY2Ugc25kX2VzMTM3eA0K PiCaZGV2aWNlIHNuZF9oZGENCj4gmmRldmljZSBzbmRfaWNoDQo+IJpkZXZpY2Ugc25kX3VhdWRp bw0KPiCaZGV2aWNlIHNuZF92aWE4MjMzDQo+DQo+IJotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gmmRkYiBj YXB0dXJlIGJ1ZmZlcg0KPg0KPiCaZGRiOiBkZGJfY2FwdHVyZToga3ZtX25saXN0DQo+IJotLS9l bmQgb2YgY3V0Ly0tDQo+DQo+IJpjb3JlLnR4dC4wIGZyb20gZGVhdGggYnkgbWV0aG9kIChiKToN Cj4NCj4gmi0tLS9jdXQvDQo+IJpic2QubXkuZG9tYWluIGR1bXBlZCBjb3JlIC0gc2VlIC92YXIv Y3Jhc2gvdm1jb3JlLjANCj4NCj4gmldlZCBOb3YgMTYgMTg6NDU6MjMgRUVUIDIwMTENCj4NCj4g mkZyZWVCU0QgYnNkLm15LmRvbWFpbiA5LjAtUFJFUkVMRUFTRSBGcmVlQlNEIDkuMC1QUkVSRUxF QVNFICM3OiBXZWQgTm92IDE2IDE4OjM2OjQ0IEVFVCAyMDExIJqamppyb290QGJzZC5teS5kb21h aW46L3Vzci9vYmovdXNyL3NyYy9zeXMvR0VORVJJQyCaYW1kNjQNCj4NCj4gmnBhbmljOiBwYWdl IGZhdWx0DQo+DQo+IJpHTlUgZ2RiIDYuMS4xIFtGcmVlQlNEXQ0KPiCaQ29weXJpZ2h0IDIwMDQg RnJlZSBTb2Z0d2FyZSBGb3VuZGF0aW9uLCBJbmMuDQo+IJpHREIgaXMgZnJlZSBzb2Z0d2FyZSwg Y292ZXJlZCBieSB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UsIGFuZCB5b3UgYXJlDQo+ IJp3ZWxjb21lIHRvIGNoYW5nZSBpdCBhbmQvb3IgZGlzdHJpYnV0ZSBjb3BpZXMgb2YgaXQgdW5k ZXIgY2VydGFpbiBjb25kaXRpb25zLg0KPiCaVHlwZSAic2hvdyBjb3B5aW5nIiB0byBzZWUgdGhl IGNvbmRpdGlvbnMuDQo+IJpUaGVyZSBpcyBhYnNvbHV0ZWx5IG5vIHdhcnJhbnR5IGZvciBHREIu IJpUeXBlICJzaG93IHdhcnJhbnR5IiBmb3IgZGV0YWlscy4NCj4gmlRoaXMgR0RCIHdhcyBjb25m aWd1cmVkIGFzICJhbWQ2NC1tYXJjZWwtZnJlZWJzZCIuLi4NCj4NCj4gmlVucmVhZCBwb3J0aW9u IG9mIHRoZSBrZXJuZWwgbWVzc2FnZSBidWZmZXI6DQo+IJpDb3B5cmlnaHQgKGMpIDE5OTItMjAx MSBUaGUgRnJlZUJTRCBQcm9qZWN0Lg0KPiCaQ29weXJpZ2h0IChjKSAxOTc5LCAxOTgwLCAxOTgz LCAxOTg2LCAxOTg4LCAxOTg5LCAxOTkxLCAxOTkyLCAxOTkzLCAxOTk0DQo+IJqampqampqamlRo ZSBSZWdlbnRzIG9mIHRoZSBVbml2ZXJzaXR5IG9mIENhbGlmb3JuaWEuIEFsbCByaWdodHMgcmVz ZXJ2ZWQuDQo+IJpGcmVlQlNEIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgVGhlIEZyZWVC U0QgRm91bmRhdGlvbi4NCj4gmkZyZWVCU0QgOS4wLVBSRVJFTEVBU0UgIzc6IFdlZCBOb3YgMTYg MTg6MzY6NDQgRUVUIDIwMTENCj4gmpqamppyb290QGJzZC5teS5kb21haW46L3Vzci9vYmovdXNy L3NyYy9zeXMvR0VORVJJQyBhbWQ2NA0KPiCaQ1BVOiBBTUQgUGhlbm9tKHRtKSBJSSBYNiAxMDkw VCBQcm9jZXNzb3IgKDMwNTEuOTAtTUh6IEs4LWNsYXNzIENQVSkNCj4gmpqaT3JpZ2luID0gIkF1 dGhlbnRpY0FNRCIgmklkID0gMHgxMDBmYTAgmkZhbWlseSA9IDEwIJpNb2RlbCA9IGEgmlN0ZXBw aW5nID0gMA0KPiCamppGZWF0dXJlcz0weDc4M2ZiZmY8RlBVLFZNRSxERSxQU0UsVFNDLE1TUixQ QUUsTUNFLENYOCxBUElDLFNFUCxNVFJSLFBHRSxNQ0EsQ01PVixQQVQsUFNFMzYsTU1YLEZYU1Is U1NFLFNTRTI+DQo+IJqamkZlYXR1cmVzMj0weDk8U1NFMyxNT04+DQo+IJqamkFNRCBGZWF0dXJl cz0weGVhMTAwODAwPFNZU0NBTEwsTlgsRkZYU1IsUkRUU0NQLExNLDNETm93ISssM0ROb3chPg0K PiCamppBTUQgRmVhdHVyZXMyPTB4MTE8TEFIRixDUjg+DQo+IJpyZWFsIG1lbW9yeSCaPSAxMDcz Njc2Mjg4ICgxMDIzIE1CKQ0KPiCaYXZhaWwgbWVtb3J5ID0gMTAxMTE4NzcxMiAoOTY0IE1CKQ0K PiCaRXZlbnQgdGltZXIgIkxBUElDIiBxdWFsaXR5IDQwMA0KPiCaQUNQSSBBUElDIFRhYmxlOiA8 VkJPWCCamlZCT1hBUElDPg0KPiCaV0FSTklORzogVklNQUdFICh2aXJ0dWFsaXplZCBuZXR3b3Jr IHN0YWNrKSBpcyBhIGhpZ2hseSBleHBlcmltZW50YWwgZmVhdHVyZS4NCj4gmmlvYXBpYzA6IENo YW5naW5nIEFQSUMgSUQgdG8gMQ0KPiCaaW9hcGljMCA8VmVyc2lvbiAxLjE+IGlycXMgMC0yMyBv biBtb3RoZXJib2FyZA0KPiCaa2JkMSBhdCBrYmRtdXgwDQo+IJphY3BpMDogPFZCT1ggVkJPWFhT RFQ+IG9uIG1vdGhlcmJvYXJkDQo+IJphPiBjcGkwOiBQb3dlciBCdXR0b24gKGZpeGVkKQ0KPiCa YWNwaTA6IFNsZWVwIEJ1dHRvbiAoZml4ZWQpDQo+IJpUaW1lY291bnRlciAiQUNQSS1mYXN0IiBm cmVxdWVuY3kgMzU3OTU0NSBIeiBxdWFsaXR5IDkwMA0KPiCaYWNwaV90aW1lcjA6IDwzMi1iaXQg dGltZXIgYXQgMy41Nzk1NDVNSHo+IHBvcnQgMHg0MDA4LTB4NDAwYiBvbiBhY3BpMA0KPiCaY3B1 MDogPEFDUEkgQ1BVPiBvbiBhY3BpMA0KPiCacGNpYjA6IDxBQ1BJIEhvc3QtUENJIGJyaWRnZT4g cG9ydCAweGNmOC0weGNmZiBvbiBhY3BpMA0KPiCacGNpMDogPEFDUEkgUENJIGJ1cz4gb24gcGNp YjANCj4gmmlzYWIwOiA8UENJLUlTQSBicmlkZ2U+IGF0IGRldmljZSAxLjAgb24gcGNpMA0KPiCa aXNhMDogPElTQSBidXM+IG9uIGlzYWIwDQo+IJphdGFwY2kwOiA8SW50ZWwgUElJWDQgVURNQTMz IGNvbnRyb2xsZXI+IHBvcnQgMHgxZjAtMHgxZjcsMHgzZjYsMHgxNzAtMHgxNzcsMHgzNzYsMHhk MDAwLTB4ZDAwZiBhdCBkZXZpY2UgMS4xIG9uIHBjaTANCj4gmmF0YTA6IDxBVEEgY2hhbm5lbCAw PiBvbiBhdGFwY2kwDQo+IJphdGExOiA8QVRBIGNoYW5uZWwgMT4gb24gYXRhcGNpMA0KPiCadmdh cGNpMDogPFZHQS1jb21wYXRpYmxlIGRpc3BsYXk+IG1lbSAweGUwMDAwMDAwLTB4ZTBmZmZmZmYg aXJxIDE4IGF0IGRldmljZSAyLjAgb24gcGNpMA0KPiCaZW0wOiA8SW50ZWwoUikgUFJPLzEwMDAg TGVnYWN5IE5ldHdvcmsgQ29ubmVjdGlvbiAxLjAuMz4gcG9ydCAweGQwMTAtMHhkMDE3IG1lbSAw eGYwMDAwMDAwLTB4ZjAwMWZmZmYgaXJxIDE5IGF0IGRldmljZSAzLjAgb24gcGNpMA0KPiCaZW0w OiBFdGhlcm5ldCBhZGRyZXNzOiAwODowMDoyNzo0Yjo2ZjpmMA0KPiCacGNpMDogPGJhc2UgcGVy aXBoZXJhbD4gYXQgZGV2aWNlIDQuMCAobm8gZHJpdmVyIGF0dGFjaGVkKQ0KPiCacGNtMDogPElu dGVsIElDSCAoODI4MDFBQSk+IHBvcnQgMHhkMTAwLTB4ZDFmZiwweGQyMDAtMHhkMjNmIGlycSAy MSBhdCBkZXZpY2UgNS4wIG9uIHBjaTANCj4gmnBjbTA6IDxTaWdtYVRlbCBTVEFDOTcwMC84My84 NCBBQzk3IENvZGVjPg0KPiCab2hjaTA6IDxPSENJIChnZW5lcmljKSBVU0IgY29udHJvbGxlcj4g bWVtIDB4ZjA4MDQwMDAtMHhmMDgwNGZmZiBpcnEgMjIgYXQgZGV2aWNlIDYuMCBvbiBwY2kwDQo+ IJp1c2J1czA6IDxPSENJIChnZW5lcmljKSBVU0IgY29udHJvbGxlcj4gb24gb2hjaTANCj4gmnBj aTA6IDxicmlkZ2U+IGF0IGRldmljZSA3LjAgKG5vIGRyaXZlciBhdHRhY2hlZCkNCj4gmmFjcGlf YWNhZDA6IDxBQyBBZGFwdGVyPiBvbiBhY3BpMA0KPiCaYXRrYmRjMDogPEtleWJvYXJkIGNvbnRy b2xsZXIgKGk4MDQyKT4gcG9ydCAweDYwLDB4NjQgaXJxIDEgb24gYWNwaTANCj4gmmF0a2JkMDog PEFUIEtleWJvYXJkPiBpcnEgMSBvbiBhdGtiZGMwDQo+IJprYmQwIGF0IGF0a2JkMA0KPiCaYXRr YmQwOiBbR0lBTlQtTE9DS0VEXQ0KPiCacHNtMDogPFBTLzIgTW91c2U+IGlycSAxMiBvbiBhdGti ZGMwDQo+IJpwc20wOiBbR0lBTlQtTE9DS0VEXQ0KPiCacHNtMDogbW9kZWwgSW50ZWxsaU1vdXNl IEV4cGxvcmVyLCBkZXZpY2UgSUQgNA0KPiCaYXR0aW1lcjA6IDxBVCB0aW1lcj4gcG9ydCAweDQw LTB4NDMsMHg1MC0weDUzIG9uIGFjcGkwDQo+IJo+IFRpbWVjb3VudGVyICJpODI1NCIgZnJlcXVl bmN5IDExOTMxODIgSHogcXVhbGl0eSAwDQo+IJpFdmVudCB0aW1lciAiaTgyNTQiIGZyZXF1ZW5j eSAxMTkzMTgyIEh6IHF1YWxpdHkgMTAwDQo+IJpzYzA6IDxTeXN0ZW0gY29uc29sZT4gYXQgZmxh Z3MgMHgxMDAgb24gaXNhMA0KPiCac2MwOiBWR0EgPDE2IHZpcnR1YWwgY29uc29sZXMsIGZsYWdz PTB4MzAwPg0KPiCadmdhMDogPEdlbmVyaWMgSVNBIFZHQT4gYXQgcG9ydCAweDNjMC0weDNkZiBp b21lbSAweGEwMDAwLTB4YmZmZmYgb24gaXNhMA0KPiCaYXRydGMwOiA8QVQgcmVhbHRpbWUgY2xv Y2s+IGF0IHBvcnQgMHg3MCBpcnEgOCBvbiBpc2EwDQo+IJpFdmVudCB0aW1lciAiUlRDIiBmcmVx dWVuY3kgMzI3NjggSHogcXVhbGl0eSAwDQo+IJpwcGMwOiBjYW5ub3QgcmVzZXJ2ZSBJL08gcG9y dCByYW5nZQ0KPiCaVGltZWNvdW50ZXJzIHRpY2sgZXZlcnkgMTAuMDAwIG1zZWMNCj4gmnBjbTA6 IG1lYXN1cmVkIGFjOTcgbGluayByYXRlIGF0IDEzMTUzNTAgSHoNCj4gmnVzYnVzMDogMTJNYnBz IEZ1bGwgU3BlZWQgVVNCIHYxLjANCj4gmnVnZW4wLjE6IDxBcHBsZT4gYXQgdXNidXMwDQo+IJp1 aHViMDogPEFwcGxlIE9IQ0kgcm9vdCBIVUIsIGNsYXNzIDkvMCwgcmV2IDEuMDAvMS4wMCwgYWRk ciAxPiBvbiB1c2J1czANCj4gmmFkYTAgYXQgYXRhMCBidXMgMCBzY2J1czAgdGFyZ2V0IDAgbHVu IDANCj4gmmFkYTA6IDxWQk9YIEhBUkRESVNLIDEuMD4gQVRBLTYgZGV2aWNlDQo+IJphZGEwOiAz My4zMDBNQi9zIHRyYW5zZmVycyAoVURNQTIsIFBJTyA2NTUzNmJ5dGVzKQ0KPiCaYWRhMDogNDQ0 ODZNQiAoOTExMDczMjggNTEyIGJ5dGUgc2VjdG9yczogMTZIIDYzUy9UIDE2MzgzQykNCj4gmmFk YTA6IFByZXZpb3VzbHkgd2FzIGtub3duIGFzIGFkMA0KPiCaY2QwIGF0IGF0YTEgYnVzIDAgc2Ni dXMxIHRhcmdldCAwIGx1biAwDQo+IJpjZDA6IDxWQk9YIENELVJPTSAxLjA+IFJlbW92YWJsZSBD RC1ST00gU0NTSS0wIGRldmljZQ0KPiCaY2QwOiAzMy4zMDBNQi9zIHRyYW5zZmVycyAoVURNQTIs IEFUQVBJIDEyYnl0ZXMsIFBJTyA2NTUzNGJ5dGVzKQ0KPiCaY2QwOiBBdHRlbXB0IHRvIHF1ZXJ5 IGRldmljZSBzaXplIGZhaWxlZDogTk9UIFJFQURZLCBNZWRpdW0gbm90IHByZXNlbnQNCj4gmlRp bWVjb3VudGVyICJUU0MiIGZyZXF1ZW5jeSAzMDUxODk1NTQ5IEh6IHF1YWxpdHkgODAwDQo+IJpS b290IG1vdW50IHdhaXRpbmcgZm9yOiB1c2J1czANCj4gmnVodWIwOiA4IHBvcnRzIHdpdGggOCBy ZW1vdmFibGUsIHNlbGYgcG93ZXJlZA0KPiCaVHJ5aW5nIHRvIG1vdW50IHJvb3QgZnJvbSB1ZnM6 L2Rldi9hZGEwcDIgW3J3XS4uLg0KPiCaPDExOD5TZXR0aW5nIGhvc3R1dWlkOiBiOGU2M2JmNC03 NzlkLTQzNmQtYTIwYS1jMzc3ZjYzZmQ1M2IuDQo+IJo8MTE4PlNldHRpbmcgaG9zdGlkOiAweGY5 Y2VkZmM1Lg0KPiCaPDExOD5FbnRyb3B5IGhhcnZlc3Rpbmc6IGludGVycnVwdHMgZXRoZXJuZXQg cG9pbnRfdG9fcG9pbnQga2lja3N0YXJ0Lg0KPiCaPDExOD5TdGFydGluZyBmaWxlIHN5c3RlbSBj aGVja3M6DQo+IJo8MTE4Pi9kZXYvYWRhMHAyOiBGSUxFIFNZU1RFTSBDTEVBTjsgU0tJUFBJTkcg Q0hFQ0tTDQo+IJo8MTE4Pi9kZXYvYWRhMHAyOiBjbGVhbiwgNzk5NTU2MiBmcmVlICgzNDM3MCBm cmFncywgOTk1MTQ5IGJsb2NrcywgMC4zJSBmcmFnbWVudGF0aW9uKQ0KPiCaPDExOD5Nb3VudGlu ZyBsb2NhbCBmaWxlIHN5c3RlbXM6Lg0KPiCaPDExOD5TZXR0aW5nIGhvc3RuYW1lOiBic2QubXku ZG9tYWluLg0KPiCaPDExOD5TdGFydGluZyBOZXR3b3JrOiBsbzAgZW0wLg0KPiCaPDExOD5sbzA6 IGZsYWdzPTgwNDk8VVAsTE9PUEJBQ0ssUlVOTklORyxNVUxUSUNBU1Q+IG1ldHJpYyAwIG10dSAx NjM4NA0KPiCaPDExOD4gb3B0aW9ucz0zPFJYQ1NVTSxUWENTVU0+DQo+IJo8MTE4PiBpbmV0NiA6 OjEgcHJlZml4bGVuIDEyOA0KPiCaPDExOD4gaW5ldDYgZmU4MDo6MSVsbzAgcHJlZml4bGVuIDY0 IHNjb3BlaWQgMHgzDQo+IJo8MTE4PiBpbmV0IDEyNy4wLjAuMSBuZXRtYXNrIDB4ZmYwMDAwMDAN Cj4gmjwxMTg+IG5kNiBvcHRpb25zPTIxPFBFUkZPUk1OVUQsQVVUT19MSU5LTE9DQUw+DQo+IJo8 MTE4PmVtMDogZmxhZ3M9ODgwMjxCUk9BRENBU1QsU0lNUExFWCxNVUxUSUNBU1Q+IG1ldHJpYyAw IG10dSAxNTAwDQo+IJo8MTE4PiBvcHRpb25zPTliPFJYQ1NVTSxUWENTVU0sVkxBTl9NVFUsVkxB Tl9IV1RBR0dJTkcsVkxBTl9IV0NTVU0+DQo+IJo8MTE4PiBldGhlciAwODowMDoyNzo0Yjo2Zjpm MA0KPiCaPDExOD4gbmQ2IG9wdGlvbnM9Mjk8UEVSRk9STU5VRCxJRkRJU0FCTEVELEFVVE9fTElO S0xPQ0FMPg0KPiCaPDExOD4gbWVkaWE6IEV0aGVybmV0IGF1dG9zZWxlY3QgKDEwMDBiYXNlVCA8 ZnVsbC1kdXBsZXg+KQ0KPiCaPDExOD4gc3RhdHVzOiBhY3RpdmUNCj4gmjwxMTg+U3RhcnRpbmcg ZGV2ZC4NCj4gmjwxMTg+U3RhcnRpbmcgTmV0d29yazogZW0wLg0KPiCaPDExOD5lbTA6IGZsYWdz PTg4MDI8QlJPQURDQVNULFNJTVBMRVgsTVVMVElDQVNUPiBtZXRyaWMgMCBtdHUgMTUwMA0KPiCa PDExOD4gb3B0aW9ucz05YjxSWENTVU0sVFhDU1VNLFZMQU5fTVRVLFZMQU5fSFdUQUdHSU5HLFZM QU5fSFdDU1VNPg0KPiCaPDExOD4gZXRoZXIgMDg6MDA6Mjc6NGI6NmY6ZjANCj4gmjwxMTg+IG5k NiBvcHRpb25zPTI5PFBFUkZPUk1OVUQsSUZESVNBQkxFRCxBVVRPX0xJTktMT0NBTD4NCj4gmjwx MTg+IG1lZGlhOiBFdGhlcm5ldCBhdXRvc2VsZWN0ICgxMDAwYmFzZVQgPGZ1bGwtZHVwbGV4PikN Cj4gmjwxMTg+IHN0YXR1czogYWN0aXZlDQo+IJo8MTE4PlN0YXJ0aW5nIE5ldHdvcms6IHVzYnVz MC4NCj4gmjwxMTg+YWRkIG5ldCA6OmZmZmY6MC4wLjAuMDogZ2F0ZXdheSA6OjENCj4gmjwxMTg+ YWRkIG5ldCA6OjAuMC4wLjA6IGdhdGV3YXkgOjoxDQo+IJo8MTE4PmFkZCBuZXQgZmU4MDo6OiBn YXRld2F5IDo6MQ0KPiCaPDExOD5hZGQgbmV0IGZmMDI6OjogZ2F0ZXdheSA6OjENCj4gmjwxMTg+ RUxGIGxkY29uZmlnIHBhdGg6IC9saWIgL3Vzci9saWIgL3Vzci9saWIvY29tcGF0IC91c3IvbG9j YWwvbGliDQo+IJo8MTE4PjMyLWJpdCBjb21wYXRpYmlsaXR5IGxkY29uZmlnIHBhdGg6IC91c3Iv bGliMzINCj4gmjwxMTg+Q3JlYXRpbmcgYW5kL29yIHRyaW1taW5nIGxvZyBmaWxlcy4NCj4gmjwx MTg+Tm8gY29yZSBkdW1wcyBmb3VuZC4NCj4gmjwxMTg+Q2xlYXJpbmcgL3RtcCAoWCByZWxhdGVk KS4NCj4gmjwxMTg+VXBkYXRpbmcgbW90ZDouDQo+IJo8MTE4PkNvbmZpZ3VyaW5nIHN5c2NvbnM6 IGtleW1hcCBibGFua3RpbWUuDQo+IJo8MTE4PlN0YXJ0aW5nIGNyb24uDQo+IJo8MTE4PlN0YXJ0 aW5nIGJhY2tncm91bmQgZmlsZSBzeXN0ZW0gY2hlY2tzIGluIDYwIHNlY29uZHMuDQo+IJo8MTE4 Pg0KPiCaPDExOD5XZWQgTm92IDE2IDE4OjM4OjM5IEVFVCAyMDExDQo+DQo+IJpGYXRhbCB0cmFw IDEyOiBwYWdlIGZhdWx0IHdoaWxlIGluIGtlcm5lbCBtb2RlDQo+IJpjcHVpZCA9IDA7IGFwaWMg aWQgPSAwMA0KPiCaZmF1bHQgdmlydHVhbCBhZGRyZXNzID0gMHgzNjANCj4gmmZhdWx0IGNvZGUg PSBzdXBlcnZpc29yIHJlYWQgZGF0YSwgcGFnZSBub3QgcHJlc2VudA0KPiCaaW5zdHJ1Y3Rpb24g cG9pbnRlciA9IDB4MjA6MHhmZmZmZmZmZjgwODI0MDNlDQo+IJpzdGFjayBwb2ludGVyIJqampqa mpo9IDB4Mjg6MHhmZmZmZmY4MDAwMzk5NTUwDQo+IJpmcmFtZSBwb2ludGVyIJqampqampo9IDB4 Mjg6MHhmZmZmZmY4MDAwMzk5NTcwDQo+IJpjb2RlIHNlZ21lbnQgPSBiYXNlIDB4MCwgbGltaXQg MHhmZmZmZiwgdHlwZSAweDFiDQo+IJqampqampqampqampqampqampqampqampo9IERQTCAwLCBw cmVzIDEsIGxvbmcgMSwgZGVmMzIgMCwgZ3JhbiAxDQo+IJpwcm9jZXNzb3IgZWZsYWdzID0gaW50 ZXJydXB0IGVuYWJsZWQsIHJlc3VtZSwgSU9QTCA9IDANCj4gmmN1cnJlbnQgcHJvY2VzcyA9IDEw NTggKGtsZHVubG9hZCkNCj4gmnRyYXAgbnVtYmVyID0gMTINCj4gmnBhbmljOiBwYWdlIGZhdWx0 DQo+IJpjcHVpZCA9IDANCj4gmktEQjogc3RhY2sgYmFja3RyYWNlOg0KPiCaIzAgMHhmZmZmZmZm ZjgwODY5NGFlIGF0IGtkYl9iYWNrdHJhY2UrMHg1ZQ0KPiCaIzEgMHhmZmZmZmZmZjgwODMzZTQ3 IGF0IHBhbmljKzB4MTg3DQo+IJojMiAweGZmZmZmZmZmODBiMjk3NDAgYXQgdHJhcF9mYXRhbCsw eDI5MA0KPiCaIzMgMHhmZmZmZmZmZjgwYjI5YTg5IGF0IHRyYXBfcGZhdWx0KzB4MWY5DQo+IJoj NCAweGZmZmZmZmZmODBiMjlmNGYgYXQgdHJhcCsweDNkZg0KPiCaIzUgMHhmZmZmZmZmZjgwYjE0 NDdmIGF0IGNhbGx0cmFwKzB4OA0KPiCaIzYgMHhmZmZmZmZmZjgwODI0MzgzIGF0IF9tdHhfbG9j a19mbGFncysweDQzDQo+IJojNyAweGZmZmZmZmZmODE2MjkyNDcgYXQgdm5ldF9wZl91bmluaXQr MHg0Nw0KPiCaIzggMHhmZmZmZmZmZjgwOGZkYzM4IGF0IHZuZXRfZGVyZWdpc3Rlcl9zeXN1bmlu aXQrMHg1OA0KPiCaIzkgMHhmZmZmZmZmZjgwODFhMThhIGF0IGxpbmtlcl9maWxlX3VubG9hZCsw eDM0YQ0KPiCaIzEwIDB4ZmZmZmZmZmY4MDgxYWJlOSBhdCBrZXJuX2tsZHVubG9hZCsweDEzOQ0K PiCaIzExIDB4ZmZmZmZmZmY4MGIyOTAzMCBhdCBhbWQ2NF9zeXNjYWxsKzB4NDUwDQo+IJojMTIg MHhmZmZmZmZmZjgwYjE0NzY3IGF0IFhmYXN0X3N5c2NhbGwrMHhmNw0KPiCaVXB0aW1lOiA2bTE1 cw0KPiCaRHVtcGluZyA4MCBvdXQgb2YgMTAwNSBNQjouLjIwJS4uNDAlLi42MCUuLjgwJQ0KPiCa LS0tL2VuZCBvZiBjdXQvLS0t From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 19:55:25 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD4FA106566B for ; Wed, 16 Nov 2011 19:55:25 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 59E728FC1C for ; Wed, 16 Nov 2011 19:55:24 +0000 (UTC) Received: by bkbzs8 with SMTP id zs8so1366709bkb.13 for ; Wed, 16 Nov 2011 11:55:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=t3DdNs7fJcaG4ztcT+Fo3NgejOz0k8QK5cr2d0Y8erM=; b=ec1Uk7n63dTFmsx88hiVknLfWPiENGy0IomYYe+0Borz44BgbrpURYct3IcsoaYVQv 0k2AgQhs0/Ltzyu386Mi8K69XnwVlkusEToqMoM4aov2GHMeLZDSd3puwI/gwD8ZFRg1 q6ONgLz+r4n6RNvb5oM13kG8jr/AJoUU0l1KA= Received: by 10.205.121.20 with SMTP id ga20mr18007990bkc.94.1321473323255; Wed, 16 Nov 2011 11:55:23 -0800 (PST) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id q16sm6357491fae.6.2011.11.16.11.55.21 (version=SSLv3 cipher=OTHER); Wed, 16 Nov 2011 11:55:22 -0800 (PST) Sender: Alexander Motin Message-ID: <4EC4152B.6020404@FreeBSD.org> Date: Wed, 16 Nov 2011 21:55:23 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111112 Thunderbird/8.0 MIME-Version: 1.0 To: Willem Jan Withagen References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 19:55:25 -0000 Hi. On 16.11.2011 18:12, Willem Jan Withagen wrote: > I'm getting these: > > Nov 16 16:40:49 zfs kernel: ata6: port is not ready (timeout 15000ms) > tfd = 00000080 > Nov 16 16:40:49 zfs kernel: ata6: hardware reset timeout > Nov 16 16:41:50 zfs kernel: ata6: port is not ready (timeout 15000ms) > tfd = 00000080 > Nov 16 16:41:50 zfs kernel: ata6: hardware reset timeout > > When inserting the tray with a SSD disk connected to that controller. > > Which is probably due to a BIOS upgrade.... > At least it started after upgrading the BIOS. So I'm asking SuperMicro > for an older version. > > When this happens, the system sometimes panics, haven't written the > details yet down right now. somewhere in get_devices... > > After the panic I really need to powerdown the machine, otherwise it > boots but stalls at finding any disks. It does not just find no disks, > it "freezes" at the point it should report the found disks in the > bios-boot. > So apparently the ata controller are left in a very confused state. > > Why is the controller found at boot, and works as it should. > And why later it just starts generating these hardware resets?? Looking on messages, I would say that you are using AHCI controller with old ata(4) driver. I would recommend you to try new ahci(4) driver. It has better hot-plug support and also supports NCQ and some other features. Note that disks connected to it will be reported as adaX instead of adY. -- Alexander Motin From owner-freebsd-stable@FreeBSD.ORG Wed Nov 16 23:36:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D09D106564A for ; Wed, 16 Nov 2011 23:36:41 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id B73768FC0A for ; Wed, 16 Nov 2011 23:36:40 +0000 (UTC) Received: by faap15 with SMTP id p15so1000138faa.13 for ; Wed, 16 Nov 2011 15:36:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=2ARoAJpE0TiNiGDiF4TZNE86/C7ZIx2qjuqhMP/9DrU=; b=fcyq+VXEmNQ4L83hlH4XPoqIVgnQ4OvSWaYY+VZOE1kc2GtrIVpzV2bKNh3eh2eyOO eXwZka0eQCtoSTKNC2oOQ1fOVlF57zd85d96bZYfegbjouQzNAcJgqaXk4gVEb99B0Aw PzvazLzPP6Wd4OGy5sMtsHTJwEtmlCWRRgor0= MIME-Version: 1.0 Received: by 10.182.150.4 with SMTP id ue4mr7802900obb.74.1321486599104; Wed, 16 Nov 2011 15:36:39 -0800 (PST) Received: by 10.182.76.9 with HTTP; Wed, 16 Nov 2011 15:36:39 -0800 (PST) In-Reply-To: <4EC3838A.9090003@FreeBSD.org> References: <20111116002944.GC24626@glenbarber.us> <4EC3838A.9090003@FreeBSD.org> Date: Wed, 16 Nov 2011 15:36:39 -0800 Message-ID: From: Chuck Tuffli To: Dimitry Andric Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Glen Barber , freebsd-stable@freebsd.org Subject: Re: Possible to build 9-stable kernel on 8.2? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2011 23:36:41 -0000 On Wed, Nov 16, 2011 at 1:34 AM, Dimitry Andric wrote: > On 2011-11-16 01:29, Glen Barber wrote: >> On Tue, Nov 15, 2011 at 11:45:02AM -0800, Chuck Tuffli wrote: > ... >>> ld:/usr/home/ctuffli/dev/releng_9/src/sys/conf/ldscript.amd64:9: syntax= error >>> *** Error code 1 >> You'll need to do 'buildworld' first. > > Actually, doing "make kernel-toolchain" is enough. =A0This builds just th= e > required tools, e.g. binutils, gcc and so on. > Perfect! This is the gem I needed. Thanks to all for the help. ---chuck From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 03:21:21 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05AF4106564A for ; Thu, 17 Nov 2011 03:21:21 +0000 (UTC) (envelope-from janm@transactionware.com) Received: from midgard.transactionware.com (mail2.transactionware.com [203.14.245.36]) by mx1.freebsd.org (Postfix) with SMTP id 89B8E8FC0C for ; Thu, 17 Nov 2011 03:21:19 +0000 (UTC) Received: (qmail 97745 invoked by uid 907); 17 Nov 2011 03:21:16 -0000 Received: from jmmacpro.transactionware.com (HELO jmmacpro.transactionware.com) (192.168.1.33) by midgard.transactionware.com (qpsmtpd/0.82) with ESMTP; Thu, 17 Nov 2011 14:21:16 +1100 Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=iso-8859-1 From: Jan Mikkelsen In-Reply-To: <4EC393E8.9070500@unsane.co.uk> Date: Thu, 17 Nov 2011 14:21:16 +1100 Content-Transfer-Encoding: quoted-printable Message-Id: <75BA89D7-8D63-41BB-A20C-F3A03D40FF74@transactionware.com> References: <4EA9E0C3.5080306@unsane.co.uk> <201111090939.14177.jhb@freebsd.org> <4EBBAE90.8010101@unsane.co.uk> <201111141442.16722.jhb@freebsd.org> <4EC393E8.9070500@unsane.co.uk> To: John Baldwin X-Mailer: Apple Mail (2.1251.1) Cc: FreeBSD Stable Mailing List , Vincent Hoffman Subject: Re: mfi timeouts X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 03:21:21 -0000 On 16/11/2011, at 9:43 PM, Vincent Hoffman wrote: > On 14/11/2011 19:42, John Baldwin wrote: >> On Thursday, November 10, 2011 5:59:28 am Vincent Hoffman wrote: >>> Well the dell has been up for about 19 hours now using MSI, I ran=20 >>> bonnie++ a few times on it and have now stuck it in a permanent loop=20= >>> (will look in from time to time.) Are there any tests you'd like=20 >>> run/info you'd like? >> Actually, can you please test = www.freebsd.org/~jhb/patches/mfi_msi.patch? >> You will have to set the hw.mfi.msi=3D1 tunable to enable MSI = support. This >> is a commit candidate if it works. Thanks. >>=20 > Applied and running with bonnie++ overnight. All good for me at least. Boots for me with hw.mfi.msi=3D1, fails to boot with mw.mfi.msi=3D0, = giving repeated timeout messages pretty much as expected. Won't be able = to put load on it until later tomorrow or next week. Regards, Jan. From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 06:46:28 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 1ED0F106566C; Thu, 17 Nov 2011 06:46:28 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 6EB361513C5; Thu, 17 Nov 2011 06:46:27 +0000 (UTC) Message-ID: <4EC4ADC3.2060604@FreeBSD.org> Date: Wed, 16 Nov 2011 22:46:27 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> In-Reply-To: <20111115100904.GA92795@icarus.home.lan> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 06:46:28 -0000 On 11/15/2011 02:09, Jeremy Chadwick wrote: > On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>> On 11/14/2011 12:31, Doug Barton wrote: >>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>> in a busy web hosting environment I came across the following post: >>>> >>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>> >>>> That basically describes what we're seeing as well, including the >>>> "doesn't happen on Linux" part. >>>> >>>> Does anyone have any ideas about this? >>>> >>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>> so it seems to be something new in 8. >>> >>> Just took a closer look at our ktrace, and actually our pattern is >>> slightly different than the one in that post. In ours the second option >>> is null, but the third is set: >>> >>> 74195 httpd 0.000017 RET sigprocmask 0 >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> 74195 httpd 0.000009 RET sigprocmask 0 >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> 74195 httpd 0.000009 RET sigprocmask 0 >>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>> >>> But repeated hundreds of times in a row. >> >> The calls cannot come from rtld, they are generated by some setjmp() >> invocation. If signal-safety is not needed, sigsetjmp() should be used >> instead. >> >> Quick grep of the apache httpd source shows a single setjmp() in their >> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). > > I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > the Apache folks (not just port committers) may have some insight to > Kostik's findings. Thanks to everyone for the responses. We tried Kostik's suggestion and unfortunately it didn't reduce the number of sigprocmask() calls to a statistically significant degree. Does anyone have any other ideas on ways to debug this? We're sort of running out of things to test. :-/ Given how important (and prevalent) the Apache + FreeBSD combination is, I'm kind of disturbed that we're seeing this performance problem, and if it's something in 8.x that's also in 9.x, it would be better to fix it prior to 9.0-RELEASE. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 07:49:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82D78106564A; Thu, 17 Nov 2011 07:49:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 1DB5E8FC27; Thu, 17 Nov 2011 07:49:14 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAH7nAlE009051 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAH7nAOh011871; Thu, 17 Nov 2011 09:49:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAH7n9PC011870; Thu, 17 Nov 2011 09:49:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 09:49:09 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="LXnoj88cJr3+5tTl" Content-Disposition: inline In-Reply-To: <4EC4ADC3.2060604@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 07:49:15 -0000 --LXnoj88cJr3+5tTl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > On 11/15/2011 02:09, Jeremy Chadwick wrote: > > On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>> On 11/14/2011 12:31, Doug Barton wrote: > >>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i= 386 > >>>> in a busy web hosting environment I came across the following post: > >>>> > >>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/23= 4520.html > >>>> > >>>> That basically describes what we're seeing as well, including the > >>>> "doesn't happen on Linux" part. > >>>> > >>>> Does anyone have any ideas about this? > >>>> > >>>> With incredibly similar stuff running on 7.x we didn't see this prob= lem, > >>>> so it seems to be something new in 8. > >>> > >>> Just took a closer look at our ktrace, and actually our pattern is > >>> slightly different than the one in that post. In ours the second opti= on > >>> is null, but the third is set: > >>> > >>> 74195 httpd 0.000017 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> 74195 httpd 0.000009 RET sigprocmask 0 > >>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>> > >>> But repeated hundreds of times in a row. > >> > >> The calls cannot come from rtld, they are generated by some setjmp() > >> invocation. If signal-safety is not needed, sigsetjmp() should be used > >> instead. > >> > >> Quick grep of the apache httpd source shows a single setjmp() in their > >> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(= ?, 0). > >=20 > > I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > the Apache folks (not just port committers) may have some insight to > > Kostik's findings. >=20 > Thanks to everyone for the responses. We tried Kostik's suggestion and > unfortunately it didn't reduce the number of sigprocmask() calls to a > statistically significant degree. >=20 > Does anyone have any other ideas on ways to debug this? We're sort of > running out of things to test. :-/ >=20 > Given how important (and prevalent) the Apache + FreeBSD combination is, > I'm kind of disturbed that we're seeing this performance problem, and if > it's something in 8.x that's also in 9.x, it would be better to fix it > prior to 9.0-RELEASE. Since my guess appeared to be not useful, the way forward is to identify the location of the call(s) that cause the issue. I suggest compliling at least apache itself, libc, rtld and libthr (if used) with debugging information. Then, attach to the running apache worker with the gdb and set breakpoint on sigprocmask. Several backtraces from the hit breakpoint should give enough data. High-tech solution is to link with libunwind and add code into sigprocmask() to gather the stacks. But I expect that gdb attach is enough. --LXnoj88cJr3+5tTl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7EvHUACgkQC3+MBN1Mb4il/ACdE7xOw5J8y45ceLUICuABv6pc 300Ani90sb6Q2xiFuU75nGw573Ic0LSC =NOqe -----END PGP SIGNATURE----- --LXnoj88cJr3+5tTl-- From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 08:00:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id CAF85106564A; Thu, 17 Nov 2011 08:00:12 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 9CE8515027C; Thu, 17 Nov 2011 07:59:06 +0000 (UTC) Message-ID: <4EC4BECA.5040705@FreeBSD.org> Date: Wed, 16 Nov 2011 23:59:06 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:00:13 -0000 On 11/16/2011 23:49, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>> in a busy web hosting environment I came across the following post: >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>> >>>>>> That basically describes what we're seeing as well, including the >>>>>> "doesn't happen on Linux" part. >>>>>> >>>>>> Does anyone have any ideas about this? >>>>>> >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>> so it seems to be something new in 8. >>>>> >>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>> slightly different than the one in that post. In ours the second option >>>>> is null, but the third is set: >>>>> >>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> >>>>> But repeated hundreds of times in a row. >>>> >>>> The calls cannot come from rtld, they are generated by some setjmp() >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>> instead. >>>> >>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>> >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>> the Apache folks (not just port committers) may have some insight to >>> Kostik's findings. >> >> Thanks to everyone for the responses. We tried Kostik's suggestion and >> unfortunately it didn't reduce the number of sigprocmask() calls to a >> statistically significant degree. >> >> Does anyone have any other ideas on ways to debug this? We're sort of >> running out of things to test. :-/ >> >> Given how important (and prevalent) the Apache + FreeBSD combination is, >> I'm kind of disturbed that we're seeing this performance problem, and if >> it's something in 8.x that's also in 9.x, it would be better to fix it >> prior to 9.0-RELEASE. > > Since my guess appeared to be not useful, Well I wouldn't say that they weren't useful, we eliminated the obvious candidate. So, "not good news" certainly, but not unhelpful. :) > the way forward is to identify > the location of the call(s) that cause the issue. I suggest compliling > at least apache itself, libc, rtld and libthr (if used) with debugging > information. Then, attach to the running apache worker with the gdb and > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > should give enough data. We tried that, and got this: Loaded symbols for /libexec/ld-elf.so.1 0x28183a5d in accept () from /lib/libc.so.7 (gdb) b sigprocmask Breakpoint 1 at 0x282d8f84 (gdb) c Continuing. no thread to satisfy query 0x28183a5d in accept () from /lib/libc.so.7 (gdb) Of course I'm not the world's greatest gdb'er, so maybe there is a better way to do it? > High-tech solution is to link with libunwind and add code into sigprocmask() > to gather the stacks. But I expect that gdb attach is enough. Ok, we'll look into that, thanks. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 08:12:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79DE31065773; Thu, 17 Nov 2011 08:12:18 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 846A18FC17; Thu, 17 Nov 2011 08:12:18 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAH8CBlJ012106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 10:12:11 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAH8CA03011993; Thu, 17 Nov 2011 10:12:10 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAH8CARW011992; Thu, 17 Nov 2011 10:12:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 10:12:10 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8zu4K4iPNav6uMHi" Content-Disposition: inline In-Reply-To: <4EC4BECA.5040705@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:12:19 -0000 --8zu4K4iPNav6uMHi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > On 11/16/2011 23:49, Kostik Belousov wrote: > > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > >> On 11/15/2011 02:09, Jeremy Chadwick wrote: > >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>>>> On 11/14/2011 12:31, Doug Barton wrote: > >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4= i386 > >>>>>> in a busy web hosting environment I came across the following post: > >>>>>> > >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/= 234520.html > >>>>>> > >>>>>> That basically describes what we're seeing as well, including the > >>>>>> "doesn't happen on Linux" part. > >>>>>> > >>>>>> Does anyone have any ideas about this? > >>>>>> > >>>>>> With incredibly similar stuff running on 7.x we didn't see this pr= oblem, > >>>>>> so it seems to be something new in 8. > >>>>> > >>>>> Just took a closer look at our ktrace, and actually our pattern is > >>>>> slightly different than the one in that post. In ours the second op= tion > >>>>> is null, but the third is set: > >>>>> > >>>>> 74195 httpd 0.000017 RET sigprocmask 0 > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>> > >>>>> But repeated hundreds of times in a row. > >>>> > >>>> The calls cannot come from rtld, they are generated by some setjmp() > >>>> invocation. If signal-safety is not needed, sigsetjmp() should be us= ed > >>>> instead. > >>>> > >>>> Quick grep of the apache httpd source shows a single setjmp() in the= ir > >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjm= p(?, 0). > >>> > >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some = of > >>> the Apache folks (not just port committers) may have some insight to > >>> Kostik's findings. > >> > >> Thanks to everyone for the responses. We tried Kostik's suggestion and > >> unfortunately it didn't reduce the number of sigprocmask() calls to a > >> statistically significant degree. > >> > >> Does anyone have any other ideas on ways to debug this? We're sort of > >> running out of things to test. :-/ > >> > >> Given how important (and prevalent) the Apache + FreeBSD combination i= s, > >> I'm kind of disturbed that we're seeing this performance problem, and = if > >> it's something in 8.x that's also in 9.x, it would be better to fix it > >> prior to 9.0-RELEASE. > >=20 > > Since my guess appeared to be not useful, >=20 > Well I wouldn't say that they weren't useful, we eliminated the obvious > candidate. So, "not good news" certainly, but not unhelpful. :) >=20 > > the way forward is to identify > > the location of the call(s) that cause the issue. I suggest compliling > > at least apache itself, libc, rtld and libthr (if used) with debugging > > information. Then, attach to the running apache worker with the gdb and Note this part. > > set breakpoint on sigprocmask. Several backtraces from the hit breakpoi= nt > > should give enough data. >=20 > We tried that, and got this: >=20 > Loaded symbols for /libexec/ld-elf.so.1 > 0x28183a5d in accept () from /lib/libc.so.7 > (gdb) b sigprocmask > Breakpoint 1 at 0x282d8f84 > (gdb) c > Continuing. > no thread to satisfy query > 0x28183a5d in accept () from /lib/libc.so.7 > (gdb) It seems your libc has no debugging information. accept() is the pure syscall wrapper, it cannot call sigprocmask. If gdb catched the PLT trampoline instead of real accept(), we would see the rtld frames. So install libc, libthr and rtld with debug. Also, having debug symbols for apache itself can be useful. >=20 > Of course I'm not the world's greatest gdb'er, so maybe there is a > better way to do it? >=20 > > High-tech solution is to link with libunwind and add code into sigprocm= ask() > > to gather the stacks. But I expect that gdb attach is enough. >=20 > Ok, we'll look into that, thanks. >=20 >=20 > Doug >=20 > --=20 >=20 > "We could put the whole Internet into a book." > "Too practical." >=20 > Breadth of IT experience, and depth of knowledge in the DNS. > Yours for the right price. :) http://SupersetSolutions.com/ --8zu4K4iPNav6uMHi Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7EwdoACgkQC3+MBN1Mb4jgFACgr6STI4yS2ldAPQ1t53RyC6G+ Vz8AniEJ50FpIGKMm+Kv5UgyQod4NgsM =KoyN -----END PGP SIGNATURE----- --8zu4K4iPNav6uMHi-- From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 08:30:32 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C754E106564A; Thu, 17 Nov 2011 08:30:32 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id D5CCE8FC12; Thu, 17 Nov 2011 08:30:31 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Thu, 17 Nov 2011 12:30:28 +0400 Message-ID: <4EC4C623.4090207@masterhost.ru> Date: Thu, 17 Nov 2011 12:30:27 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] Cc: Kostik Belousov , Doug Barton , freebsd-apache@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:30:32 -0000 On 17.11.2011 11:49, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>> in a busy web hosting environment I came across the following post: >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>> >>>>>> That basically describes what we're seeing as well, including the >>>>>> "doesn't happen on Linux" part. >>>>>> >>>>>> Does anyone have any ideas about this? >>>>>> >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>> so it seems to be something new in 8. >>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>> slightly different than the one in that post. In ours the second option >>>>> is null, but the third is set: >>>>> >>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>> >>>>> But repeated hundreds of times in a row. >>>> The calls cannot come from rtld, they are generated by some setjmp() >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>> instead. >>>> >>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>> the Apache folks (not just port committers) may have some insight to >>> Kostik's findings. >> Thanks to everyone for the responses. We tried Kostik's suggestion and >> unfortunately it didn't reduce the number of sigprocmask() calls to a >> statistically significant degree. >> >> Does anyone have any other ideas on ways to debug this? We're sort of >> running out of things to test. :-/ >> >> Given how important (and prevalent) the Apache + FreeBSD combination is, >> I'm kind of disturbed that we're seeing this performance problem, and if >> it's something in 8.x that's also in 9.x, it would be better to fix it >> prior to 9.0-RELEASE. > Since my guess appeared to be not useful, the way forward is to identify > the location of the call(s) that cause the issue. I suggest compliling > at least apache itself, libc, rtld and libthr (if used) with debugging > information. Then, attach to the running apache worker with the gdb and > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > should give enough data. > > High-tech solution is to link with libunwind and add code into sigprocmask() > to gather the stacks. But I expect that gdb attach is enough. I am sorry for repeat (I wrote about it), but what do you think about this hack: diff -u ./rtld_lock.c.orig ./rtld_lock.c --- ./rtld_lock.c.orig 2011-11-15 07:56:14.000000000 +0000 +++ ./rtld_lock.c 2011-11-15 07:54:42.000000000 +0000 @@ -118,7 +118,7 @@ sigset_t tmp_oldsigmask; for ( ; ; ) { - sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); +// sigprocmask(SIG_BLOCK, &fullsigmask, &tmp_oldsigmask); if (atomic_cmpset_acq_int(&l->lock, 0, WAFLAG)) break; sigprocmask(SIG_SETMASK, &tmp_oldsigmask, NULL); @@ -135,7 +135,7 @@ atomic_add_rel_int(&l->lock, -RC_INCR); else { atomic_add_rel_int(&l->lock, -WAFLAG); - sigprocmask(SIG_SETMASK, &oldsigmask, NULL); +// sigprocmask(SIG_SETMASK, &oldsigmask, NULL); } } This is one of source sigprocmask. Look: truss with original /libexec/ld-elf.so.1: #truss true __sysctl(0xbfbfe624,0x2,0xbfbfe62c,0xbfbfe630,0x0,0x0) = 0 (0x0) mmap(0x0,320,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 671633408 (0x28085000) munmap(0x28085000,320) = 0 (0x0) __sysctl(0xbfbfe688,0x2,0x2807be3c,0xbfbfe690,0x0,0x0) = 0 (0x0) mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 671633408 (0x28085000) issetugid(0x28074967,0xbfbfeb4c,0x104,0x0,0x0,0x0) = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or directory' open("/var/run/ld-elf.so.hints",O_RDONLY,00) = 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0\M^E\0\0"...,128) = 128 (0x80) lseek(3,0x80,SEEK_SET) = 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,133) = 133 (0x85) close(3) = 0 (0x0) access("/lib/libc.so.7",0) = 0 (0x0) open("/lib/libc.so.7",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=94234,size=1155172,blksize=16384 }) = 0 (0x0) pread(0x3,0x2807ad80,0x1000,0x0,0x0,0x0) = 4096 (0x1000) mmap(0x0,1159168,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) = 671666176 (0x2808d000) mmap(0x2808d000,1040384,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE,3,0x0) = 671666176 (0x2808d000) mmap(0x2818b000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xfe000) = 672706560 (0x2818b000) mprotect(0x28191000,94208,PROT_READ|PROT_WRITE) = 0 (0x0) close(3) = 0 (0x0) sysarch(0xa,0xbfbfe6f0,0x2804b3cb,0x2807a2f8,0x2805de39,0x2807a2f8) = 0 (0x0) mmap(0x0,64,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,64) = 0 (0x0) mmap(0x0,21896,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,21896) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) __sysctl(0xbfbfe6a4,0x2,0x28192700,0xbfbfe6ac,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGHUP|SIGINT|SIGQUIT|SIGKILL|SIGPIPE|SIGALRM|SIGTERM|SIGURG|SIGSTOP|SIGTSTP|SIGCONT|SIGCHLD|SIGTTIN|SIGTTOU|SIGIO|SIGXCPU|SIGXFSZ|SIGVTALRM|SIGPROF|SIGWINCH|SIGINFO|SIGUSR1|SIGUSR2,0x0) = 0 (0x0) sigprocmask(SIG_SETMASK,0x0,0x0) = 0 (0x0) process exit, rval = 0 and truss after patching: #truss true __sysctl(0xbfbfe5a4,0x2,0xbfbfe5ac,0xbfbfe5b0,0x0,0x0) = 0 (0x0) mmap(0x0,320,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 671633408 (0x28085000) munmap(0x28085000,320) = 0 (0x0) __sysctl(0xbfbfe608,0x2,0x2807be3c,0xbfbfe610,0x0,0x0) = 0 (0x0) mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 671633408 (0x28085000) issetugid(0x28074927,0xbfbfeacc,0x104,0x0,0x0,0x0) = 0 (0x0) open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or directory' open("/var/run/ld-elf.so.hints",O_RDONLY,00) = 3 (0x3) read(3,"Ehnt\^A\0\0\0\M^@\0\0\0B\0\0\0\0"...,128) = 128 (0x80) lseek(3,0x80,SEEK_SET) = 128 (0x80) read(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,66) = 66 (0x42) close(3) = 0 (0x0) access("/lib/libc.so.7",0) = 0 (0x0) open("/lib/libc.so.7",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=12,size=1155172,blksize=16384 }) = 0 (0x0) pread(0x3,0x2807ad80,0x1000,0x0,0x0,0x0) = 4096 (0x1000) mmap(0x0,1159168,PROT_NONE,MAP_PRIVATE|MAP_ANON|MAP_NOCORE,-1,0x0) = 671666176 (0x2808d000) mmap(0x2808d000,1040384,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE,3,0x0) = 671666176 (0x2808d000) mmap(0x2818b000,24576,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED,3,0xfe000) = 672706560 (0x2818b000) mprotect(0x28191000,94208,PROT_READ|PROT_WRITE) = 0 (0x0) close(3) = 0 (0x0) sysarch(0xa,0xbfbfe670,0x2804b3cb,0x2807a2f8,0x2805ddf9,0x2807a2f8) = 0 (0x0) mmap(0x0,64,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,64) = 0 (0x0) mmap(0x0,21896,PROT_READ|PROT_WRITE,MAP_ANON,-1,0x0) = 672825344 (0x281a8000) munmap(0x281a8000,21896) = 0 (0x0) __sysctl(0xbfbfe624,0x2,0x28192700,0xbfbfe62c,0x0,0x0) = 0 (0x0) process exit, rval = 0 I am not expert in the internal structure of ld-elf.so, but I am trying to think there is more effective way for locks. Because now we are doing a few syscalls for each dynamic loaded library in fork time. -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 08:34:58 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 1848F1065670; Thu, 17 Nov 2011 08:34:58 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 5C15817A7EA; Thu, 17 Nov 2011 08:34:44 +0000 (UTC) Message-ID: <4EC4C723.9090606@FreeBSD.org> Date: Thu, 17 Nov 2011 00:34:43 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Daniil Cherednik References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4C623.4090207@masterhost.ru> In-Reply-To: <4EC4C623.4090207@masterhost.ru> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 08:34:58 -0000 On 11/17/2011 00:30, Daniil Cherednik wrote: > I am sorry for repeat (I wrote about it), but what do you think about > this hack: Danill, thanks, and sorry if I wasn't clear before, but the problem we're seeing has a very clear pattern: 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) That the rtld calls don't exhibit. Kostik, thanks for your more detailed response, we'll poke that a bit and report back. Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 09:27:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id F17081065670; Thu, 17 Nov 2011 09:27:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 110C4175F70; Thu, 17 Nov 2011 09:26:49 +0000 (UTC) Message-ID: <4EC4D359.4040406@FreeBSD.org> Date: Thu, 17 Nov 2011 01:26:49 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 09:27:16 -0000 On 11/17/2011 00:12, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >> On 11/16/2011 23:49, Kostik Belousov wrote: >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>> >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>> >>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>> "doesn't happen on Linux" part. >>>>>>>> >>>>>>>> Does anyone have any ideas about this? >>>>>>>> >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>> so it seems to be something new in 8. >>>>>>> >>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>> slightly different than the one in that post. In ours the second option >>>>>>> is null, but the third is set: >>>>>>> >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>> >>>>>>> But repeated hundreds of times in a row. >>>>>> >>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>> instead. >>>>>> >>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>> >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>> the Apache folks (not just port committers) may have some insight to >>>>> Kostik's findings. >>>> >>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>> statistically significant degree. >>>> >>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>> running out of things to test. :-/ >>>> >>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>> prior to 9.0-RELEASE. >>> >>> Since my guess appeared to be not useful, >> >> Well I wouldn't say that they weren't useful, we eliminated the obvious >> candidate. So, "not good news" certainly, but not unhelpful. :) >> >>> the way forward is to identify >>> the location of the call(s) that cause the issue. I suggest compliling >>> at least apache itself, libc, rtld and libthr (if used) with debugging >>> information. Then, attach to the running apache worker with the gdb and > Note this part. Right, we attached to a worker, that's why it's in accept(). :) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. It's not catching there though: Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) info threads Cannot get thread info: invalid key (gdb) Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:19:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4481C106566C for ; Thu, 17 Nov 2011 10:19:01 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id E0ADD8FC0C for ; Thu, 17 Nov 2011 10:19:00 +0000 (UTC) Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20]) by qmta14.westchester.pa.mail.comcast.net with comcast id yAK11h0030SCNGk5EAK1f5; Thu, 17 Nov 2011 10:19:01 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta09.westchester.pa.mail.comcast.net with comcast id yAJy1h00E1t3BNj3VAJymM; Thu, 17 Nov 2011 10:19:00 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DAC05102C1D; Thu, 17 Nov 2011 02:18:56 -0800 (PST) Date: Thu, 17 Nov 2011 02:18:56 -0800 From: Jeremy Chadwick To: Kostik Belousov Message-ID: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111117081210.GN50300@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Daniil Cherednik , Doug Barton , freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:19:01 -0000 On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > > On 11/16/2011 23:49, Kostik Belousov wrote: > > > On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > > >> On 11/15/2011 02:09, Jeremy Chadwick wrote: > > >>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > > >>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > > >>>>> On 11/14/2011 12:31, Doug Barton wrote: > > >>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 > > >>>>>> in a busy web hosting environment I came across the following post: > > >>>>>> > > >>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html > > >>>>>> > > >>>>>> That basically describes what we're seeing as well, including the > > >>>>>> "doesn't happen on Linux" part. > > >>>>>> > > >>>>>> Does anyone have any ideas about this? > > >>>>>> > > >>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, > > >>>>>> so it seems to be something new in 8. > > >>>>> > > >>>>> Just took a closer look at our ktrace, and actually our pattern is > > >>>>> slightly different than the one in that post. In ours the second option > > >>>>> is null, but the third is set: > > >>>>> > > >>>>> 74195 httpd 0.000017 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> 74195 httpd 0.000009 RET sigprocmask 0 > > >>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > > >>>>> > > >>>>> But repeated hundreds of times in a row. > > >>>> > > >>>> The calls cannot come from rtld, they are generated by some setjmp() > > >>>> invocation. If signal-safety is not needed, sigsetjmp() should be used > > >>>> instead. > > >>>> > > >>>> Quick grep of the apache httpd source shows a single setjmp() in their > > >>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). > > >>> > > >>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of > > >>> the Apache folks (not just port committers) may have some insight to > > >>> Kostik's findings. > > >> > > >> Thanks to everyone for the responses. We tried Kostik's suggestion and > > >> unfortunately it didn't reduce the number of sigprocmask() calls to a > > >> statistically significant degree. > > >> > > >> Does anyone have any other ideas on ways to debug this? We're sort of > > >> running out of things to test. :-/ > > >> > > >> Given how important (and prevalent) the Apache + FreeBSD combination is, > > >> I'm kind of disturbed that we're seeing this performance problem, and if > > >> it's something in 8.x that's also in 9.x, it would be better to fix it > > >> prior to 9.0-RELEASE. > > > > > > Since my guess appeared to be not useful, > > > > Well I wouldn't say that they weren't useful, we eliminated the obvious > > candidate. So, "not good news" certainly, but not unhelpful. :) > > > > > the way forward is to identify > > > the location of the call(s) that cause the issue. I suggest compliling > > > at least apache itself, libc, rtld and libthr (if used) with debugging > > > information. Then, attach to the running apache worker with the gdb and > Note this part. > > > > set breakpoint on sigprocmask. Several backtraces from the hit breakpoint > > > should give enough data. > > > > We tried that, and got this: > > > > Loaded symbols for /libexec/ld-elf.so.1 > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) b sigprocmask > > Breakpoint 1 at 0x282d8f84 > > (gdb) c > > Continuing. > > no thread to satisfy query > > 0x28183a5d in accept () from /lib/libc.so.7 > > (gdb) > It seems your libc has no debugging information. > accept() is the pure syscall wrapper, it cannot call sigprocmask. > If gdb catched the PLT trampoline instead of real accept(), we would > see the rtld frames. So install libc, libthr and rtld with debug. > > Also, having debug symbols for apache itself can be useful. I'd also like to point out that enabling debugging symbols in devel/apr1 will be greatly needed here, not just in www/apache*. I'm wondering if maybe this is some sort of pthread "thing" going on. A quick grep -r sigmask of the Apache source turns up some pthread_* bits pertaining to worker. Is Apache build using WITH_THREADS? What about devel/apr1? I don't use worker MPM on any of our boxes, we actually use ITK MPM solely because of the hosting nature of what we do. I've actually never seen worker MPM in use on any *IX machine I've been on or administrated, only prefork. The Apache documentation even mentions that "if you want stability or compatibility, prefork is the choice", while "if you want scalability, worker is a better choice"[1]. These sorts of quotes often shock me given what year it is. :-) [1]: http://httpd.apache.org/docs/2.0/mpm.html -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:24:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D83F2106564A; Thu, 17 Nov 2011 10:24:08 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 750018FC08; Thu, 17 Nov 2011 10:24:08 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:d9a7:15b3:2d22:93e6]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 659E54AC2D; Thu, 17 Nov 2011 14:24:06 +0400 (MSK) Date: Thu, 17 Nov 2011 14:24:01 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <11310157468.20111117142401@serebryakov.spb.ru> To: Kostik Belousov In-Reply-To: <20111117074909.GL50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Daniil Cherednik , Doug Barton , freebsd-apache@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:24:08 -0000 Hello, Kostik. You wrote 17 =ED=EE=FF=E1=F0=FF 2011 =E3., 11:49:09: > High-tech solution is to link with libunwind and add code into sigprocmas= k() > to gather the stacks. But I expect that gdb attach is enough. Proper high-tech solution is to use DTrace. It is very food in such things. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:57:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2387106564A; Thu, 17 Nov 2011 10:57:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 674268FC16; Thu, 17 Nov 2011 10:57:48 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAHAvib6042499 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAHAviYf012457; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAHAviCC012456; Thu, 17 Nov 2011 12:57:44 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 17 Nov 2011 12:57:44 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111117105744.GS50300@deviant.kiev.zoral.com.ua> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pSPXDt+5DZRK1gNs" Content-Disposition: inline In-Reply-To: <4EC4D359.4040406@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:57:49 -0000 --pSPXDt+5DZRK1gNs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Nov 17, 2011 at 01:26:49AM -0800, Doug Barton wrote: > On 11/17/2011 00:12, Kostik Belousov wrote: > > On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: > >> On 11/16/2011 23:49, Kostik Belousov wrote: > >>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: > >>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: > >>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: > >>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: > >>>>>>> On 11/14/2011 12:31, Doug Barton wrote: > >>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-= p4 i386 > >>>>>>>> in a busy web hosting environment I came across the following po= st: > >>>>>>>> > >>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-Octobe= r/234520.html > >>>>>>>> > >>>>>>>> That basically describes what we're seeing as well, including the > >>>>>>>> "doesn't happen on Linux" part. > >>>>>>>> > >>>>>>>> Does anyone have any ideas about this? > >>>>>>>> > >>>>>>>> With incredibly similar stuff running on 7.x we didn't see this = problem, > >>>>>>>> so it seems to be something new in 8. > >>>>>>> > >>>>>>> Just took a closer look at our ktrace, and actually our pattern is > >>>>>>> slightly different than the one in that post. In ours the second = option > >>>>>>> is null, but the third is set: > >>>>>>> > >>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 > >>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) > >>>>>>> > >>>>>>> But repeated hundreds of times in a row. > >>>>>> > >>>>>> The calls cannot come from rtld, they are generated by some setjmp= () > >>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be = used > >>>>>> instead. > >>>>>> > >>>>>> Quick grep of the apache httpd source shows a single setjmp() in t= heir > >>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigset= jmp(?, 0). > >>>>> > >>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Som= e of > >>>>> the Apache folks (not just port committers) may have some insight to > >>>>> Kostik's findings. > >>>> > >>>> Thanks to everyone for the responses. We tried Kostik's suggestion a= nd > >>>> unfortunately it didn't reduce the number of sigprocmask() calls to a > >>>> statistically significant degree. > >>>> > >>>> Does anyone have any other ideas on ways to debug this? We're sort of > >>>> running out of things to test. :-/ > >>>> > >>>> Given how important (and prevalent) the Apache + FreeBSD combination= is, > >>>> I'm kind of disturbed that we're seeing this performance problem, an= d if > >>>> it's something in 8.x that's also in 9.x, it would be better to fix = it > >>>> prior to 9.0-RELEASE. > >>> > >>> Since my guess appeared to be not useful, > >> > >> Well I wouldn't say that they weren't useful, we eliminated the obvious > >> candidate. So, "not good news" certainly, but not unhelpful. :) > >> > >>> the way forward is to identify > >>> the location of the call(s) that cause the issue. I suggest compliling > >>> at least apache itself, libc, rtld and libthr (if used) with debugging > >>> information. Then, attach to the running apache worker with the gdb a= nd > > Note this part. >=20 > Right, we attached to a worker, that's why it's in accept(). :) >=20 > > It seems your libc has no debugging information. > > accept() is the pure syscall wrapper, it cannot call sigprocmask. > > If gdb catched the PLT trampoline instead of real accept(), we would > > see the rtld frames. So install libc, libthr and rtld with debug. >=20 > It's not catching there though: >=20 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) c > Continuing. > no thread to satisfy query > 0x28183b2d in accept () at accept.S:3 > 3 RSYSCALL(accept) > (gdb) info threads > Cannot get thread info: invalid key > (gdb) Err, the other part of my message was that you shall set the breakpoint on sigprocmask. I want to see a backtrace from the breakpoint hit. Several times. The backtrace at the attach time has no use. --pSPXDt+5DZRK1gNs Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7E6KgACgkQC3+MBN1Mb4j5mgCgvbV20mLT2co6NO3NUTQlM8Ub kOQAmwU4tRvdIjYTtMfkfVwUq63h/pLe =pZru -----END PGP SIGNATURE----- --pSPXDt+5DZRK1gNs-- From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 10:58:24 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 564311065680; Thu, 17 Nov 2011 10:58:24 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id E62D68FC15; Thu, 17 Nov 2011 10:58:23 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 50DF3153436; Thu, 17 Nov 2011 11:58:22 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C+yrDGYL2B9p; Thu, 17 Nov 2011 11:58:20 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id 5DC2B153434; Thu, 17 Nov 2011 11:58:20 +0100 (CET) Message-ID: <4EC4E8CB.1000101@digiware.nl> Date: Thu, 17 Nov 2011 11:58:19 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Alexander Motin References: <4EC4152B.6020404@FreeBSD.org> In-Reply-To: <4EC4152B.6020404@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Stable Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 10:58:24 -0000 On 2011-11-16 20:55, Alexander Motin wrote: > Hi. > > On 16.11.2011 18:12, Willem Jan Withagen wrote: >> I'm getting these: >> >> Nov 16 16:40:49 zfs kernel: ata6: port is not ready (timeout 15000ms) >> tfd = 00000080 >> Nov 16 16:40:49 zfs kernel: ata6: hardware reset timeout >> Nov 16 16:41:50 zfs kernel: ata6: port is not ready (timeout 15000ms) >> tfd = 00000080 >> Nov 16 16:41:50 zfs kernel: ata6: hardware reset timeout >> >> When inserting the tray with a SSD disk connected to that controller. >> >> Which is probably due to a BIOS upgrade.... >> At least it started after upgrading the BIOS. So I'm asking SuperMicro >> for an older version. >> >> When this happens, the system sometimes panics, haven't written the >> details yet down right now. somewhere in get_devices... >> >> After the panic I really need to powerdown the machine, otherwise it >> boots but stalls at finding any disks. It does not just find no disks, >> it "freezes" at the point it should report the found disks in the >> bios-boot. >> So apparently the ata controller are left in a very confused state. >> >> Why is the controller found at boot, and works as it should. >> And why later it just starts generating these hardware resets?? > > Looking on messages, I would say that you are using AHCI controller with > old ata(4) driver. I would recommend you to try new ahci(4) driver. It > has better hot-plug support and also supports NCQ and some other > features. Note that disks connected to it will be reported as adaX > instead of adY. Hi Alexander, Thanx for pointing that out. I recompiled the kernel with ahci.. And using GPT for the most part took care of the fact that the underlying devicenames changed.... Only "problem" was swap, which I renamed from ad{6,8} to ada{6,8} but ahci also renumbers.... However on swap that is not much of a problem during booting. the root partition however is running of: zfsboot 4.16G 62.3G 0 0 0 0 mirror 4.16G 62.3G 0 0 0 0 gptid/966bdc14-0b73-11df-a9ff-003048de97cd - - 0 0 0 0 gptid/60be2c5d-4a83-11df-bf4f-003048de97cd - - 0 0 0 0 But they were not labeled as such in GPT, so that sor t of makes sense. And I've seen a lot of discussion on how to try and fix this. But I think that at the moment I will not bother. Performance wise I have the feeling that it has al lot better performance. It was scrubbing a 6,5T filesystem and read io-ops where around 100-200 with ata, but now they are more in the 600-900 range. Let's see how we fare with this setting. --WjW From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 11:03:30 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AF47106564A for ; Thu, 17 Nov 2011 11:03:30 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id D8F2A8FC08 for ; Thu, 17 Nov 2011 11:03:29 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 46D2C153434; Thu, 17 Nov 2011 12:03:29 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gVuTWNTImIzb; Thu, 17 Nov 2011 12:03:27 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id D5DA6153433; Thu, 17 Nov 2011 12:03:26 +0100 (CET) Message-ID: <4EC4E9FE.2060208@digiware.nl> Date: Thu, 17 Nov 2011 12:03:26 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Peter Maloney References: <4EC3E0E4.5010704@digiware.nl> <4EC3F13B.4020700@brockmann-consult.de> In-Reply-To: <4EC3F13B.4020700@brockmann-consult.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 11:03:30 -0000 On 2011-11-16 18:22, Peter Maloney wrote: > Willem, > > I can only guess, but... > > Is AHCI enabled in the bios? If you are not using 'fake-raid' for any > disks, you should [depending on FreeBSD version, HBA, etc.] probably > enable AHCI. Some servers actually come with SATA set in IDE mode. And > if you are using zfs, the controller optimally should not be RAID at > all. And if you have AHCI enabled already, try disabling it (losing hot > swapping ability, and some performance). ACHI is enabled otherwise I cannot used the last set of SATA connectors with this MB. Controller for these connectors is CH9. > What version of FreeBSD are you using? I had a terrible experience with > ZFS on FreeBSD 8.2 release, and 8.2-stable-April2011. I would recommend > upgrading to the latest 8-stable with cvsup. I'm at most 1 month behind on STABLE, since I just upgrade in about that frequency... Might vary a little on the amount of rumbling on the list. > This thread seems related: > http://forums.freebsd.org/showthread.php?t=24189 > > The guy was using 8.2 release, and he downgraded to an old version of > the driver to fix, saying that a patch also existed in 8-stable that > fixes the problem. > > Are you using an expander? No SATA expanders... > What HBA / hard disk controller are you using? A combi of CH9 and ARECA in PCI-X, disks are all exported a single disks. Thanx for the suggestions.... -_WjW > Am 16.11.2011 17:12, schrieb Willem Jan Withagen: >> Hi, >> >> I'm getting these: >> >> Nov 16 16:40:49 zfs kernel: ata6: port is not ready (timeout 15000ms) >> tfd = 00000080 >> Nov 16 16:40:49 zfs kernel: ata6: hardware reset timeout >> Nov 16 16:41:50 zfs kernel: ata6: port is not ready (timeout 15000ms) >> tfd = 00000080 >> Nov 16 16:41:50 zfs kernel: ata6: hardware reset timeout >> >> When inserting the tray with a SSD disk connected to that controller. >> >> Which is probably due to a BIOS upgrade.... >> At least it started after upgrading the BIOS. So I'm asking SuperMicro >> for an older version. >> >> When this happens, the system sometimes panics, haven't written the >> details yet down right now. somewhere in get_devices... >> >> After the panic I really need to powerdown the machine, otherwise it >> boots but stalls at finding any disks. It does not just find no disks, >> it "freezes" at the point it should report the found disks in the >> bios-boot. >> So apparently the ata controller are left in a very confused state. >> >> Why is the controller found at boot, and works as it should. >> And why later it just starts generating these hardware resets?? >> >> --WjW >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 11:20:15 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6D231065670 for ; Thu, 17 Nov 2011 11:20:15 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta15.emeryville.ca.mail.comcast.net (qmta15.emeryville.ca.mail.comcast.net [76.96.27.228]) by mx1.freebsd.org (Postfix) with ESMTP id 90E148FC13 for ; Thu, 17 Nov 2011 11:20:15 +0000 (UTC) Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87]) by qmta15.emeryville.ca.mail.comcast.net with comcast id yBL21h0041smiN4AFBL8NX; Thu, 17 Nov 2011 11:20:08 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta20.emeryville.ca.mail.comcast.net with comcast id yB0u1h0041t3BNj8gB0uMA; Thu, 17 Nov 2011 11:00:55 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 8B4BF102C1D; Thu, 17 Nov 2011 03:20:13 -0800 (PST) Date: Thu, 17 Nov 2011 03:20:13 -0800 From: Jeremy Chadwick To: Willem Jan Withagen Message-ID: <20111117112013.GA42497@icarus.home.lan> References: <4EC3E0E4.5010704@digiware.nl> <4EC3F13B.4020700@brockmann-consult.de> <4EC4E9FE.2060208@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EC4E9FE.2060208@digiware.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Peter Maloney Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 11:20:16 -0000 On Thu, Nov 17, 2011 at 12:03:26PM +0100, Willem Jan Withagen wrote: > On 2011-11-16 18:22, Peter Maloney wrote: > >Willem, > > > >I can only guess, but... > > > >Is AHCI enabled in the bios? If you are not using 'fake-raid' for any > >disks, you should [depending on FreeBSD version, HBA, etc.] probably > >enable AHCI. Some servers actually come with SATA set in IDE mode. And > >if you are using zfs, the controller optimally should not be RAID at > >all. And if you have AHCI enabled already, try disabling it (losing hot > >swapping ability, and some performance). > > ACHI is enabled otherwise I cannot used the last set of SATA > connectors with this MB. Controller for these connectors is CH9. There are two "kinds" of AHCI on FreeBSD -- and I'm speaking strictly about the kernel bits, not AHCI the option ROM/BIOS option: ataahci.ko -- this is "AHCI support using ata(4)" ahci.ko -- this is "AHCI support using CAM(4)" You want the latter, and I can tell you're using the former (if at all). There would be no "ata6" if you were using ahci.ko; it would be called something like ahcichX, indicating "AHCI channel X". Furthermore, because CAM(4) gets used, your disk device names change from adX to adaX. This is expected. Using ataahci.ko results in the disks still being named adX, because it uses ata(4). Hope this helps shed some light on the confusion. Generally speaking you want to be using ahci.ko, mav@ and many others have spent a lot of time working on that and getting it to play nice with CAM -- it's beautiful, and hot-swapping works perfectly on all the Intel ICHxx systems I've tried it on (ICH7R, ICH9R). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 11:26:19 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3B2F1065678 for ; Thu, 17 Nov 2011 11:26:19 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id 642AB8FC14 for ; Thu, 17 Nov 2011 11:26:19 +0000 (UTC) Received: from rack1.digiware.nl (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id BB780153436; Thu, 17 Nov 2011 12:26:18 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E4wrx4ThyB2X; Thu, 17 Nov 2011 12:26:13 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) by mail.digiware.nl (Postfix) with ESMTP id BDAB0153433; Thu, 17 Nov 2011 12:26:13 +0100 (CET) Message-ID: <4EC4EF55.4060204@digiware.nl> Date: Thu, 17 Nov 2011 12:26:13 +0100 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EC3E0E4.5010704@digiware.nl> <4EC3F13B.4020700@brockmann-consult.de> <4EC4E9FE.2060208@digiware.nl> <20111117112013.GA42497@icarus.home.lan> In-Reply-To: <20111117112013.GA42497@icarus.home.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-stable@freebsd.org, Peter Maloney Subject: Re: Trouble with SSD on SATA X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 11:26:19 -0000 On 2011-11-17 12:20, Jeremy Chadwick wrote: > On Thu, Nov 17, 2011 at 12:03:26PM +0100, Willem Jan Withagen wrote: >> On 2011-11-16 18:22, Peter Maloney wrote: >>> Willem, >>> >>> I can only guess, but... >>> >>> Is AHCI enabled in the bios? If you are not using 'fake-raid' for any >>> disks, you should [depending on FreeBSD version, HBA, etc.] probably >>> enable AHCI. Some servers actually come with SATA set in IDE mode. And >>> if you are using zfs, the controller optimally should not be RAID at >>> all. And if you have AHCI enabled already, try disabling it (losing hot >>> swapping ability, and some performance). >> >> ACHI is enabled otherwise I cannot used the last set of SATA >> connectors with this MB. Controller for these connectors is CH9. > > There are two "kinds" of AHCI on FreeBSD -- and I'm speaking strictly > about the kernel bits, not AHCI the option ROM/BIOS option: > > ataahci.ko -- this is "AHCI support using ata(4)" > ahci.ko -- this is "AHCI support using CAM(4)" > > You want the latter, and I can tell you're using the former (if at all). I did 'man ahci', and followed it from there. So now I'm running with ahci. > There would be no "ata6" if you were using ahci.ko; it would be called > something like ahcichX, indicating "AHCI channel X". Furthermore, > because CAM(4) gets used, your disk device names change from adX to > adaX. This is expected. Using ataahci.ko results in the disks still > being named adX, because it uses ata(4). ahci0: port 0x1c70-0x1c77,0x1c64-0x1c67,0x1c68-0x1c6f,0x1c60-0x1c63,0x1c00-0x1c1f mem 0xdf923000-0xdf9237ff irq 17 at device 31.2 on pci0 ahci0: [ITHREAD] ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: at channel 3 on ahci0 ahcich3: [ITHREAD] ahcich4: at channel 4 on ahci0 ahcich4: [ITHREAD] ahcich5: at channel 5 on ahci0 ahcich5: [ITHREAD] > Hope this helps shed some light on the confusion. Generally speaking > you want to be using ahci.ko, mav@ and many others have spent a lot of > time working on that and getting it to play nice with CAM -- it's > beautiful, and hot-swapping works perfectly on all the Intel ICHxx > systems I've tried it on (ICH7R, ICH9R). For the time being I can only concur with you. Scrubbing is way much faster than with the ata driver. --WjW From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 11:32:18 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E74CF106566C; Thu, 17 Nov 2011 11:32:18 +0000 (UTC) (envelope-from tevans.uk@googlemail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D1BE8FC12; Thu, 17 Nov 2011 11:32:18 +0000 (UTC) Received: by vcbfy13 with SMTP id fy13so1402489vcb.13 for ; Thu, 17 Nov 2011 03:32:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=aUWxZO5AMLb7NVvMyZyAuq0p5HCuPfZ5CWO+DNcuUo8=; b=ImBBA3+sjrCdK5ZdsHk2Y3aG5osHKfQ8odxX4viIX9WFOgA8CSL4ERw0K30kuP/Ro7 knSPUYUNBXNuB6po7i97QbCqtJPqTusLaietlRxyFsMhOS9ecNzvc/liU19AOD8cC4fC MqxERxg9WAy5jr9kVymMx1ZlR4SUXVj74n9lQ= MIME-Version: 1.0 Received: by 10.52.38.6 with SMTP id c6mr57265309vdk.73.1321527951777; Thu, 17 Nov 2011 03:05:51 -0800 (PST) Received: by 10.52.182.40 with HTTP; Thu, 17 Nov 2011 03:05:51 -0800 (PST) In-Reply-To: <20111117101856.GA39096@icarus.home.lan> References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <20111117101856.GA39096@icarus.home.lan> Date: Thu, 17 Nov 2011 11:05:51 +0000 Message-ID: From: Tom Evans To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Kostik Belousov , Daniil Cherednik , Doug Barton , freebsd-stable@freebsd.org, freebsd-apache@freebsd.org Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 11:32:19 -0000 On Thu, Nov 17, 2011 at 10:18 AM, Jeremy Chadwick wrote: > I don't use worker MPM on any of our boxes, we actually use ITK MPM > solely because of the hosting nature of what we do. =C2=A0I've actually n= ever > seen worker MPM in use on any *IX machine I've been on or administrated, > only prefork. =C2=A0The Apache documentation even mentions that "if you w= ant > stability or compatibility, prefork is the choice", while "if you want > scalability, worker is a better choice"[1]. =C2=A0These sorts of quotes o= ften > shock me given what year it is. =C2=A0:-) > I've used both worker and event MPMs in production on high volume sites for > 4 years now, running on FreeBSD 7, with no problems. I think you are cherry picking the quotes from httpd's 2.0 documentation, which is actually an old bit of software now - it has just been voted EOL. The current stable (2.2) docs actually say: "sites that need a great deal of scalability can choose to use a threaded MPM like worker or event, while sites requiring stability or compatibility with older software can use a prefork" Event and worker have no issues unless you run non thread safe modules, or modules which use libraries which are not thread safe, eg PHP (more commonly, a PHP extension). Cheers Tom From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 12:39:43 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50225106564A; Thu, 17 Nov 2011 12:39:43 +0000 (UTC) (envelope-from dcherednik@masterhost.ru) Received: from mail.corp.masterhost.ru (wincas02.mail.corp.masterhost.ru [90.156.219.212]) by mx1.freebsd.org (Postfix) with ESMTP id 78C198FC15; Thu, 17 Nov 2011 12:39:42 +0000 (UTC) Received: from kzholnay-pc.masterhost.ru (87.242.97.5) by mail.corp.masterhost.ru (90.156.219.210) with Microsoft SMTP Server (TLS) id 14.1.339.1; Thu, 17 Nov 2011 16:39:40 +0400 Message-ID: <4EC5008B.8050603@masterhost.ru> Date: Thu, 17 Nov 2011 16:39:39 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 MIME-Version: 1.0 To: References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <20111117101856.GA39096@icarus.home.lan> In-Reply-To: <20111117101856.GA39096@icarus.home.lan> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [87.242.97.5] Cc: Kostik Belousov , Doug Barton , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 12:39:43 -0000 On 17.11.2011 14:18, Jeremy Chadwick wrote: > On Thu, Nov 17, 2011 at 10:12:10AM +0200, Kostik Belousov wrote: >> On Wed, Nov 16, 2011 at 11:59:06PM -0800, Doug Barton wrote: >>> On 11/16/2011 23:49, Kostik Belousov wrote: >>>> On Wed, Nov 16, 2011 at 10:46:27PM -0800, Doug Barton wrote: >>>>> On 11/15/2011 02:09, Jeremy Chadwick wrote: >>>>>> On Tue, Nov 15, 2011 at 11:07:45AM +0200, Kostik Belousov wrote: >>>>>>> On Mon, Nov 14, 2011 at 12:51:35PM -0800, Doug Barton wrote: >>>>>>>> On 11/14/2011 12:31, Doug Barton wrote: >>>>>>>>> Trying to track down a load problem we're seeing on 8.2-RELEASE-p4 i386 >>>>>>>>> in a busy web hosting environment I came across the following post: >>>>>>>>> >>>>>>>>> http://lists.freebsd.org/pipermail/freebsd-questions/2011-October/234520.html >>>>>>>>> >>>>>>>>> That basically describes what we're seeing as well, including the >>>>>>>>> "doesn't happen on Linux" part. >>>>>>>>> >>>>>>>>> Does anyone have any ideas about this? >>>>>>>>> >>>>>>>>> With incredibly similar stuff running on 7.x we didn't see this problem, >>>>>>>>> so it seems to be something new in 8. >>>>>>>> Just took a closer look at our ktrace, and actually our pattern is >>>>>>>> slightly different than the one in that post. In ours the second option >>>>>>>> is null, but the third is set: >>>>>>>> >>>>>>>> 74195 httpd 0.000017 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000013 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> 74195 httpd 0.000009 RET sigprocmask 0 >>>>>>>> 74195 httpd 0.000012 CALL sigprocmask(SIG_BLOCK,0,0xbfbf89d4) >>>>>>>> >>>>>>>> But repeated hundreds of times in a row. >>>>>>> The calls cannot come from rtld, they are generated by some setjmp() >>>>>>> invocation. If signal-safety is not needed, sigsetjmp() should be used >>>>>>> instead. >>>>>>> >>>>>>> Quick grep of the apache httpd source shows a single setjmp() in their >>>>>>> copy of pcre. No idea is it to safe to change setjmp() into sigsetjmp(?, 0). >>>>>> I hate cross-posting, but: adding freebsd-apache@ to the list. Some of >>>>>> the Apache folks (not just port committers) may have some insight to >>>>>> Kostik's findings. >>>>> Thanks to everyone for the responses. We tried Kostik's suggestion and >>>>> unfortunately it didn't reduce the number of sigprocmask() calls to a >>>>> statistically significant degree. >>>>> >>>>> Does anyone have any other ideas on ways to debug this? We're sort of >>>>> running out of things to test. :-/ >>>>> >>>>> Given how important (and prevalent) the Apache + FreeBSD combination is, >>>>> I'm kind of disturbed that we're seeing this performance problem, and if >>>>> it's something in 8.x that's also in 9.x, it would be better to fix it >>>>> prior to 9.0-RELEASE. >>>> Since my guess appeared to be not useful, >>> Well I wouldn't say that they weren't useful, we eliminated the obvious >>> candidate. So, "not good news" certainly, but not unhelpful. :) >>> >>>> the way forward is to identify >>>> the location of the call(s) that cause the issue. I suggest compliling >>>> at least apache itself, libc, rtld and libthr (if used) with debugging >>>> information. Then, attach to the running apache worker with the gdb and >> Note this part. >> >>>> set breakpoint on sigprocmask. Several backtraces from the hit breakpoint >>>> should give enough data. >>> We tried that, and got this: >>> >>> Loaded symbols for /libexec/ld-elf.so.1 >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) b sigprocmask >>> Breakpoint 1 at 0x282d8f84 >>> (gdb) c >>> Continuing. >>> no thread to satisfy query >>> 0x28183a5d in accept () from /lib/libc.so.7 >>> (gdb) >> It seems your libc has no debugging information. >> accept() is the pure syscall wrapper, it cannot call sigprocmask. >> If gdb catched the PLT trampoline instead of real accept(), we would >> see the rtld frames. So install libc, libthr and rtld with debug. >> >> Also, having debug symbols for apache itself can be useful. > I'd also like to point out that enabling debugging symbols in devel/apr1 > will be greatly needed here, not just in www/apache*. > > I'm wondering if maybe this is some sort of pthread "thing" going on. A > quick grep -r sigmask of the Apache source turns up some pthread_* bits > pertaining to worker. > > Is Apache build using WITH_THREADS? What about devel/apr1? > > I don't use worker MPM on any of our boxes, we actually use ITK MPM > solely because of the hosting nature of what we do. I've actually never > seen worker MPM in use on any *IX machine I've been on or administrated, > only prefork. The Apache documentation even mentions that "if you want > stability or compatibility, prefork is the choice", while "if you want > scalability, worker is a better choice"[1]. These sorts of quotes often > shock me given what year it is. :-) > > [1]: http://httpd.apache.org/docs/2.0/mpm.html > We use ITK MPM too, but we have big trouble with performance on FreeBSD. Also, I have to say we can`t use keep-alive connection, so apache creates new child for each request. -- С уважением, Daniil Cherednik .masterhost From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 14:54:13 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 187E8106566B for ; Thu, 17 Nov 2011 14:54:13 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (m209-73.dsl.rawbw.com [198.144.209.73]) by mx1.freebsd.org (Postfix) with ESMTP id D54228FC18 for ; Thu, 17 Nov 2011 14:54:12 +0000 (UTC) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.14.5/8.14.5) with ESMTP id pAHEfvFh029381 for ; Thu, 17 Nov 2011 06:41:57 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.14.5/8.14.5/Submit) id pAHEfvgi029380 for stable@freebsd.org; Thu, 17 Nov 2011 06:41:57 -0800 (PST) (envelope-from david) Date: Thu, 17 Nov 2011 06:41:57 -0800 From: David Wolfskill To: stable@freebsd.org Message-ID: <20111117144157.GL1706@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , stable@freebsd.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="AYsPlKobQGgtCvjI" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: ld: kernel.debug: Not enough room for program headers (allocated 5, need 6) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 14:54:13 -0000 --AYsPlKobQGgtCvjI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Color me perplexed. 3 of the 4 kernels I build were fine; the 4th one ... ugh. I'm tracking stable/8 daily & rebuild as often as that (less if there are no changes). I do this on 2 machines: my laptop (which only builds for itself) and a "build machine" (named "freebeast"), which builds GENERIC for itself, as well as kernels ALBERT & JANUS for a couple of other machines. The laptop was fine (for stable/8); it was running: FreeBSD g1-227.catwhisker.org 8.2-STABLE FreeBSD 8.2-STABLE #272 r227447M: = Fri Nov 11 04:07:05 PST 2011 root@g1-227.catwhisker.org:/common/S1/obj/= usr/src/sys/CANARY i386 and is now running: FreeBSD g1-227.catwhisker.org 8.2-STABLE FreeBSD 8.2-STABLE #273 r227611M: = Thu Nov 17 04:16:50 PST 2011 root@g1-227.catwhisker.org:/common/S1/obj/= usr/src/sys/CANARY i386 (The "M" suffix on the GRN is for a patch to sys/conf/newvers.sh so it will recognize my working copy as SVN even though there is no sys/.svn directory, since I'm using subversion-1.7.1.) The build machine is running: FreeBSD freebeast.catwhisker.org 8.2-STABLE FreeBSD 8.2-STABLE #400 r227447= M: Fri Nov 11 04:11:35 PST 2011 root@freebeast.catwhisker.org:/common/S= 1/obj/usr/src/sys/GENERIC i386 it rebuilt GENERIC & ALBERT OK, then on JANUS, the "make buildkernel" terminated with: =2E.. cc -c -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wnested-externs -Wst= rict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual = -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sy= s -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -inclu= de opt_global.h -fno-common -finline-limit=3D8000 --param inline-unit-growt= h=3D100 --param large-function-growth=3D1000 -mno-align-long-strings -mpre= ferred-stack-boundary=3D2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3= -ffreestanding -fstack-protector -Werror hints.c cc -c -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wnested-externs -Wst= rict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual = -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sy= s -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -inclu= de opt_global.h -fno-common -finline-limit=3D8000 --param inline-unit-growt= h=3D100 --param large-function-growth=3D1000 -mno-align-long-strings -mpre= ferred-stack-boundary=3D2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3= -ffreestanding -fstack-protector -Werror vnode_if.c :> hack.c cc -shared -nostdlib hack.c -o hack.So rm -f hack.c MAKE=3D/usr/obj/usr/src/make.i386/make sh /usr/src/sys/conf/newvers.sh GENE= RIC cc -c -O -pipe -std=3Dc99 -g -Wall -Wredundant-decls -Wnested-externs -Wst= rict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual = -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sy= s -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -inclu= de opt_global.h -fno-common -finline-limit=3D8000 --param inline-unit-growt= h=3D100 --param large-function-growth=3D1000 -mno-align-long-strings -mpre= ferred-stack-boundary=3D2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3= -ffreestanding -fstack-protector -Werror vers.c linking kernel.debug ld: kernel.debug: Not enough room for program headers (allocated 5, need 6) ld: final link failed: Bad value *** Error code 1 Stop in /common/S1/obj/usr/src/sys/JANUS. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. freebeast(8.2-S)[3]=20 I'm rather left wondering "room" where, precisely? The other perplexing thing is that JANUS is actually a subset of my laptop's kernel config, and it's been getting built routinely; as of its most recent update, it is running: FreeBSD janus.catwhisker.org 8.2-STABLE FreeBSD 8.2-STABLE #398 r227447M: F= ri Nov 11 04:15:30 PST 2011 root@freebeast.catwhisker.org:/common/S1/ob= j/usr/src/sys/JANUS i386 And I've not changed the JANUS config since date: 2010/04/18 13:04:27; author: david; state: Exp; Here's a copy: # # JANUS -- kernel configuration file for FreeBSD/i386 as a packet filter # include GENERIC # firewall support, for access limiting options IPFIREWALL # options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_VERBOSE #enable logging to syslogd(8) options IPFIREWALL_VERBOSE_LIMIT=3D0 #do not limit verbosity # dummynet for bandwidth limiting (requires IPFIREWALL) options DUMMYNET # divert sockets for natd options IPDIVERT [End of JANUS config] As noted, since my laptop is also exposed to networks I don't control, its kernel is also built with the above options (as well as quite a few more, for support of user-interface, vs. headless server, operation). And it built & runs fine.... Clues? Thanks! Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --AYsPlKobQGgtCvjI Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7FHTQACgkQmprOCmdXAD1JpgCdGw8aSpXA1DvZmuSybh6n/po/ 1bQAn32axWT+XeFP4naZ02fH52cy8b0x =flF7 -----END PGP SIGNATURE----- --AYsPlKobQGgtCvjI-- From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 16:15:00 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B742106566C for ; Thu, 17 Nov 2011 16:15:00 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id D066C8FC13 for ; Thu, 17 Nov 2011 16:14:59 +0000 (UTC) Received: by ywe9 with SMTP id 9so1869923ywe.13 for ; Thu, 17 Nov 2011 08:14:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=3WNugf3Ek1FjEfIu7xz8LE3OcC0lYmUdoZGNoufQeLg=; b=it3+oe8PDj7lWReJ5A6HYNWCuQZ98ZFiK+QXqJa1BJHyi+IUceNbkIbSMG/HPHQL1L yJ9NzPdIIMMTgz0IpGEYyaF/7MImR6hqLyItem5bdN8l7i1Rajy2iAK8ouuj0jH8248T uLuGFUcvnSqNh2pKCVgdqAIlsLuzbFS2d2pEA= MIME-Version: 1.0 Received: by 10.101.95.19 with SMTP id x19mr11458270anl.70.1321545070608; Thu, 17 Nov 2011 07:51:10 -0800 (PST) Sender: artemb@gmail.com Received: by 10.236.208.34 with HTTP; Thu, 17 Nov 2011 07:51:10 -0800 (PST) In-Reply-To: <20111117144157.GL1706@albert.catwhisker.org> References: <20111117144157.GL1706@albert.catwhisker.org> Date: Thu, 17 Nov 2011 07:51:10 -0800 X-Google-Sender-Auth: TOP0E_puYJrLKn5ti7LiJriQn_8 Message-ID: From: Artem Belevich To: David Wolfskill , stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: ld: kernel.debug: Not enough room for program headers (allocated 5, need 6) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 16:15:00 -0000 On Thu, Nov 17, 2011 at 6:41 AM, David Wolfskill wro= te: > MAKE=3D/usr/obj/usr/src/make.i386/make sh /usr/src/sys/conf/newvers.sh GE= NERIC > cc -c -O -pipe =A0-std=3Dc99 -g -Wall -Wredundant-decls -Wnested-externs = -Wstrict-prototypes =A0-Wmissing-prototypes -Wpointer-arith -Winline -Wcast= -qual =A0-Wundef -Wno-pointer-sign -fformat-extensions -nostdinc =A0-I. -I/= usr/src/sys -I/usr/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEAD= ERS -include opt_global.h -fno-common -finline-limit=3D8000 --param inline-= unit-growth=3D100 --param large-function-growth=3D1000 =A0-mno-align-long-s= trings -mpreferred-stack-boundary=3D2 =A0-mno-mmx -mno-3dnow -mno-sse -mno-= sse2 -mno-sse3 -ffreestanding -fstack-protector -Werror =A0vers.c > linking kernel.debug > ld: kernel.debug: Not enough room for program headers (allocated 5, need = 6) > ld: final link failed: Bad value > *** Error code 1 > I'm rather left wondering "room" where, precisely? Room for the program headers at the beginning of the ELF file. Look at sys/conf/ldscript.* and search for SIZEOF_HEADERS. One way to work around the issue is to replace SIZEOF_HEADERS with a fixed value. Try 0x1000. --Artem From owner-freebsd-stable@FreeBSD.ORG Thu Nov 17 18:19:35 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A8A31065738; Thu, 17 Nov 2011 18:19:35 +0000 (UTC) (envelope-from kensmith@buffalo.edu) Received: from localmailA.acsu.buffalo.edu (localmail.buffalo.edu [128.205.5.196]) by mx1.freebsd.org (Postfix) with ESMTP id 5AB1C8FC08; Thu, 17 Nov 2011 18:19:35 +0000 (UTC) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 785DECE47; Thu, 17 Nov 2011 13:19:34 -0500 (EST) Received: from localmailA.acsu.buffalo.edu (localhost [127.0.0.1]) by localmailA.acsu.buffalo.edu (Postfix) with ESMTP id 30491B8DE; Thu, 17 Nov 2011 13:19:31 -0500 (EST) Received: from smtp1.acsu.buffalo.edu (smtp1.acsu.buffalo.edu [128.205.5.253]) by localmailA.acsu.buffalo.edu (Prefixe) with ESMTP id 14EDCD51A; Thu, 17 Nov 2011 13:19:31 -0500 (EST) Received: from [128.205.32.76] (bauer.cse.buffalo.edu [128.205.32.76]) (Authenticated sender: kensmith@buffalo.edu) by smtp1.acsu.buffalo.edu (Postfix) with ESMTPSA id 088AB48EED; Thu, 17 Nov 2011 13:19:31 -0500 (EST) From: Ken Smith To: freebsd-current , freebsd-stable Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-4X3a6ZpsEuSjvxhdzbe6" Date: Thu, 17 Nov 2011 13:19:30 -0500 Message-ID: <1321553970.82271.66.camel@bauer.cse.buffalo.edu> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-PM-EL-Spam-Prob: : 8% Cc: Subject: FreeBSD 9.0-RC2 Available... X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2011 18:19:35 -0000 --=-4X3a6ZpsEuSjvxhdzbe6 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The second of the Release Candidate builds for the 9.0-RELEASE release cycle is now available. Since this is the first release of a brand new branch I cross-post the announcements on both -current and -stable. But just so you know most of the developers active in head and stable/9 pay more attention to the -current mailing list. If you notice problems you can report them through the normal Gnats PR system or on the -current mailing list. At the current plans are for one more RC build, which will be followed by the release. The 9.0-RELEASE cycle will be tracked here: http://wiki.freebsd.org/Releng/9.0TODO NOTE: The location of the FTP install tree and ISOs is the same as it had been for BETA2/BETA3/RC1, though we are still deciding if this will be the layout we switch to for the release. ISO images for the following architectures are available, with pathnames given relative to the top-level of the FTP site: amd64: .../releases/amd64/amd64/ISO-IMAGES/9.0/ i386: .../releases/i386/i386/ISO-IMAGES/9.0/ ia64: .../releases/ia64/ia64/ISO-IMAGES/9.0/ powerpc: .../releases/powerpc/powerpc/ISO-IMAGES/9.0/ powerpc64: .../releases/powerpc/powerpc64/ISO-IMAGES/9.0/ sparc64: .../releases/sparc64/sparc64/ISO-IMAGES/9.0/ MD5/SHA256 checksums are tacked on below. If you would like to use csup/cvsup mechanisms to access the source tree the branch tag to use is now "RELENG_9_0", if you use "." (head) you will get 10-CURRENT. If you would like to access the source tree via SVN it is "svn://svn.freebsd.org/base/releng/9.0/". We still have the nit that the creation of a new SVN branch winds up causing what looks like a check-in of the entire tree in CVS (a side-effect of the svn2cvs exporter) so "mergemaster -F" is your friend if you are using csup/cvsup. FreeBSD Update -------------- The freebsd-update(8) utility supports binary upgrades of i386 and amd64 sy= stems running earlier FreeBSD releases. Systems running 7.[34]-RELEASE, 8.[12]-RELEASE, 9.0-BETA[123], or 9.0-RC1 can upgrade as follows: First, a minor change must be made to the freebsd-update code in order for it to accept file names appearing in FreeBSD 9.0 which contain the '%' and '@' characters; without this change, freebsd-update will error out with the message "The update metadata is correctly signed, but failed an integrity check". # sed -i '' -e 's/=3D_/=3D%@_/' /usr/sbin/freebsd-update Now freebsd-update can fetch bits belonging to 9.0-RC2. During this proces= s freebsd-update will ask for help in merging configuration files. # freebsd-update upgrade -r 9.0-RC2 Due to changes in the way that FreeBSD is packaged on the release media, tw= o complications may arise in this process if upgrading from FreeBSD 7.x or 8.= x: 1. The FreeBSD kernel, which previously could appear in either /boot/kernel or /boot/GENERIC, now only appears as /boot/kernel. As a result, any kerne= l appearing in /boot/GENERIC will be deleted. Please carefully read the outp= ut printed by freebsd-update and confirm that an updated kernel will be placed into /boot/kernel before proceeding beyond this point. 2. The FreeBSD source tree in /usr/src (if present) will be deleted. (Norm= ally freebsd-update will update a source tree, but in this case the changes in release packaging result in freebsd-update not recognizing that the source = tree from the old release and the source tree from the new release correspond to= the same part of FreeBSD.) # freebsd-update install The system must now be rebooted with the newly installed kernel before the non-kernel components are updated. # shutdown -r now After rebooting, freebsd-update needs to be run again to install the new userland components: # freebsd-update install At this point, users of systems being upgraded from FreeBSD 8.2-RELEASE or earlier will be prompted by freebsd-update to rebuild all third-party applications (e.g., ports installed from the ports tree) due to updates in system libraries. After updating installed third-party applications (and again, only if freebsd-update printed a message indicating that this was necessary), run freebsd-update again so that it can delete the old (no longer used) system libraries: # freebsd-update install Finally, reboot into 9.0-RC2: # shutdown -r now Checksums: MD5 (FreeBSD-9.0-RC2-amd64-bootonly.iso) =3D 0165f0a2a1141a4c69413ec0c0b7d7= 54 MD5 (FreeBSD-9.0-RC2-amd64-memstick.img) =3D 84713f2f556cdd58aa18e36093525e= 6c MD5 (FreeBSD-9.0-RC2-amd64-dvd1.iso) =3D 59792b2012e6feff6981d3cf58c0b901 MD5 (FreeBSD-9.0-RC2-i386-bootonly.iso) =3D ed3e7b8ac2fdadd2c41c0d5c8b26943= c MD5 (FreeBSD-9.0-RC2-i386-memstick.img) =3D f396728fbd72c61078a7f9511b0c71f= f MD5 (FreeBSD-9.0-RC2-i386-dvd1.iso) =3D cacc9962fa80a6b9a5067c907f127e8b MD5 (FreeBSD-9.0-RC2-ia64-bootonly.iso) =3D faaf6f0c529b8ec59b9d4252ae666dc= 7 MD5 (FreeBSD-9.0-RC2-ia64-memstick) =3D b937883e7634334bef1ddf3eb1e06ffb MD5 (FreeBSD-9.0-RC2-ia64-release.iso) =3D c1f5623734132ea80a9fa2298262884c MD5 (FreeBSD-9.0-RC2-powerpc-bootonly.iso) =3D 35e667deaa7223e0829677c37621= 63d8 MD5 (FreeBSD-9.0-RC2-powerpc-memstick) =3D 01b06227124fc7f9ad224d0512940ea8 MD5 (FreeBSD-9.0-RC2-powerpc-release.iso) =3D 055e06744fa1b8c584cfda8be2352= 462 MD5 (FreeBSD-9.0-RC2-powerpc64-bootonly.iso) =3D 38d240e6cdfd5f986f0690ba12= 56eb0f MD5 (FreeBSD-9.0-RC2-powerpc64-memstick) =3D b820ed78bce87b69a04e9d473c63ec= fc MD5 (FreeBSD-9.0-RC2-powerpc64-release.iso) =3D 44d550342db5090c1d0fb5f6070= 5cb0c MD5 (FreeBSD-9.0-RC2-sparc64-bootonly.iso) =3D 7ffb3dc55bb02506cbc95d0f7a94= edab MD5 (FreeBSD-9.0-RC2-sparc64-dvd1.iso) =3D fe4b87eff3f3cde33c2908b071e45c0f SHA256 (FreeBSD-9.0-RC2-amd64-bootonly.iso) =3D 81daa3aa92b8eb6f1bf0c6196c1= cae138c881fe53dce22b9fbf2f19a8cf30551 SHA256 (FreeBSD-9.0-RC2-amd64-memstick.img) =3D fda9025bb8c3ee8c3a4f8db3a01= 44c669408707eab3dc7c5e2070342e979e33e SHA256 (FreeBSD-9.0-RC2-amd64-dvd1.iso) =3D d7071da7cf440f79a7368f8a8b26ba9= b6e18dcb3785aec83df866ddf576ef418 SHA256 (FreeBSD-9.0-RC2-i386-bootonly.iso) =3D 200ac2b9e950548c873cba93f3b3= c669e529720522897220d6b6dd438f806490 SHA256 (FreeBSD-9.0-RC2-i386-memstick.img) =3D 7888f9ed58da415a7356810c0776= 8587c6d2b5d4734f9ca47c92bfc10dab1b0a SHA256 (FreeBSD-9.0-RC2-i386-dvd1.iso) =3D 41a1e12496ba5a44dba4c6e6cfb27c6d= d5f49fb62378e05c1a61909a4f29d06c SHA256 (FreeBSD-9.0-RC2-ia64-bootonly.iso) =3D 059a05b4e779f94cb221fcb7ce26= 1db13e45092510b1fbeac7e17d5f2f6c7c36 SHA256 (FreeBSD-9.0-RC2-ia64-memstick) =3D 3e9985eb02c0f8cd8e5d84356048a019= c98d9628e5836ca777426a3abec3f83f SHA256 (FreeBSD-9.0-RC2-ia64-release.iso) =3D 49524a249f72ad6a8a21ba1ad0d1e= 738ddc09342c7086cf1a20c08ad06885539 SHA256 (FreeBSD-9.0-RC2-powerpc-bootonly.iso) =3D 3a584930ecfd772defa3d8687= 9c62199b7a15f6e502bf4809a04a3fc8d10e10e SHA256 (FreeBSD-9.0-RC2-powerpc-memstick) =3D 06b2985c278362801e2c454d72a3a= e0f873ad2e050a058e76f7b6da84f2d4812 SHA256 (FreeBSD-9.0-RC2-powerpc-release.iso) =3D 0402cca90eb4123fbf1d5dc46d= 9a36ba08b38c0e7b5e83a3a5c0cd7cd1095124 SHA256 (FreeBSD-9.0-RC2-powerpc64-bootonly.iso) =3D 7510745f1fbd3f1a4e1d7c9= d0fed39f8ea3e2dc37de029e931f06856729e5183 SHA256 (FreeBSD-9.0-RC2-powerpc64-memstick) =3D a29bf56f7461be20b0fd15e6ff7= ed08ea0a2baabcbf04430d337893113d19a68 SHA256 (FreeBSD-9.0-RC2-powerpc64-release.iso) =3D 54b1e14798839ced94863a5f= b95b08c20a46474854effc022602cad018789b98 SHA256 (FreeBSD-9.0-RC2-sparc64-bootonly.iso) =3D 9e08825e9549d330384817296= d832f9e00288357ccd1b2d6cce99d0f656b096b SHA256 (FreeBSD-9.0-RC2-sparc64-dvd1.iso) =3D f53810ff78e4015833e0ac9e81865= c1abd93c622607d14aa2a74b918d2bc469c --=20 Ken Smith - From there to here, from here to | kensmith@buffalo.edu there, funny things are everywhere. | - Theodor Geisel | --=-4X3a6ZpsEuSjvxhdzbe6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEABECAAYFAk7FUDIACgkQ/G14VSmup/aGgQCcCAytIwLB5y1FwFCmxaO4DoQV PaQAnielWVf9F02a6oRYQWE2qSqgLUw8 =BFfv -----END PGP SIGNATURE----- --=-4X3a6ZpsEuSjvxhdzbe6-- From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 00:05:17 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83B5D106564A; Fri, 18 Nov 2011 00:05:17 +0000 (UTC) (envelope-from slonoman2011@yandex.ru) Received: from forward12.mail.yandex.net (forward12.mail.yandex.net [IPv6:2a02:6b8:0:801::2]) by mx1.freebsd.org (Postfix) with ESMTP id 3C11B8FC08; Fri, 18 Nov 2011 00:05:16 +0000 (UTC) Received: from web146.yandex.ru (web146.yandex.ru [95.108.131.161]) by forward12.mail.yandex.net (Yandex) with ESMTP id 9865FC21DB8; Fri, 18 Nov 2011 04:05:14 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1321574714; bh=LfBwZ5ESfC+eIt13ZxkHz7EguMvGboVMs5E5V8fvbtA=; h=From:To:Subject:MIME-Version:Message-Id:Date: Content-Transfer-Encoding:Content-Type; b=Jt0XG2SGTr5MMhpoNDHskf6g9PwmeHc5B/2vUgXs9o1++h/6fnCX15lAfPSdSi/4e 5RMrcfh3KWDtVmzAQiK2MHutB8YK+dM2PCKa2WquEepxXtAwjJ9UZTWJX33+tGmuxQ tXjuozeHmrDMxF/6qYTP5iRPeR7jlulImSyT+e3M= Received: from localhost (localhost.localdomain [127.0.0.1]) by web146.yandex.ru (Yandex) with ESMTP id 6A4827158001; Fri, 18 Nov 2011 04:05:14 +0400 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1321574714; bh=LfBwZ5ESfC+eIt13ZxkHz7EguMvGboVMs5E5V8fvbtA=; h=From:To:Subject:MIME-Version:Message-Id:Date: Content-Transfer-Encoding:Content-Type; b=Jt0XG2SGTr5MMhpoNDHskf6g9PwmeHc5B/2vUgXs9o1++h/6fnCX15lAfPSdSi/4e 5RMrcfh3KWDtVmzAQiK2MHutB8YK+dM2PCKa2WquEepxXtAwjJ9UZTWJX33+tGmuxQ tXjuozeHmrDMxF/6qYTP5iRPeR7jlulImSyT+e3M= X-Yandex-Spam: 1 Received: from nat140-249-205-109.tvoe.tv (nat140-249-205-109.tvoe.tv [109.205.249.140]) by web146.yandex.ru with HTTP; Fri, 18 Nov 2011 04:05:13 +0400 From: Slono Slono To: freebsd-hackers@freebsd.org,freebsd-stable@freebsd.org MIME-Version: 1.0 Message-Id: <297161321574713@web146.yandex.ru> Date: Fri, 18 Nov 2011 04:05:13 +0400 X-Mailer: Yamail [ http://yandex.ru ] 5.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain Cc: Subject: memory leaks (and some other warning like divison by zero; ) auto reports for FreeBSD source code X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 00:05:17 -0000 Hi This information can be interesting - in most cases really doesn't suffice free() and someone is necessary with commit bit who it can to correct. reported by cppcheck (http://cppcheck.sourceforge.net/): This report is actual for FreeBSD 9.0-PRERELEASE Scan for /usr/src/libexec/: [rtld-elf/rtld.c:1660]: (error) Resource leak: fd [ftpd/ftpd.c:610]: (error) Mismatching allocation and deallocation: fd Scan for /usr/src/lib/ [libarchive/archive_entry_link_resolver.c:240]: (error) Memory leak: le [libarchive/archive_read_open_filename.c:115]: (error) Resource leak: fd [libarchive/archive_read_support_format_tar.c:638]: (error) Buffer access out-of-bounds: header.magic [libarchive/archive_write_disk.c:1767]: (error) Memory leak: le [libc/db/test/btree.tests/main.c:601]: (error) Resource leak: fp [libc/gen/_pthread_stubs.c:218]: (error) Analysis failed. If the code is valid then please report this failure. [libc/mips/gen/makecontext.c:107]: (error) Uninitialized variable: i [libc/net/getifaddrs.c:250]: (error) Invalid deallocation [libc/net/getifaddrs.c:255]: (error) Invalid deallocation [libc/net/nscache.c:118]: (error) Common realloc mistake: 'buffer' nulled but not freed upon failure [libc/net/nscache.c:204]: (error) Common realloc mistake: 'buffer' nulled but not freed upon failure [libc/net/nscache.c:299]: (error) Common realloc mistake: 'buffer' nulled but not freed upon failure [libc/net/nscache.c:375]: (error) Common realloc mistake: 'buffer' nulled but not freed upon failure [libc/quad/qdivrem.c:100]: (error) Division by zero [libc/rpc/clnt_perror.c:301]: (error) Allocation with clnt_spcreateerror, fprintf doesn't release it. [libc/rpc/netnamer.c:331]: (error) Resource leak: fd [libdisk/open_disk.c:89]: (error) Memory leak: d [libdwarf/dwarf_attr.c:49]: (error) Possible null pointer dereference: at - otherwise it is redundant to check if at is null at line 54 [libdwarf/dwarf_init.c:505]: (error) Memory leak: cu [libedit/readline.c:1009]: (error) Possible null pointer dereference: arr - otherwise it is redundant to check if arr is null at line 1006 [libedit/readline.c:1693]: (error) Possible null pointer dereference: pwd - otherwise it is redundant to check if pwd is null at line 1695 [libedit/readline.c:1934]: (error) Memory leak: wbuf [libfetch/file.c:148]: (error) Resource leak: dir [libgssapi/gss_accept_sec_context.c:217]: (error) Possible null pointer dereference: mc - otherwise it is redundant to check if mc is null at line 219 [libgssapi/gss_accept_sec_context.c:220]: (error) Memory leak: ctx [libgssapi/gss_inquire_cred_by_mech.c:68]: (error) Possible null pointer dereference: mcp - otherwise it is redundant to check if mcp is null at line 70 [libgssapi/gss_verify_mic.c:42]: (error) Possible null pointer dereference: ctx - otherwise it is redundant to check if ctx is null at line 46 [libgssapi/gss_wrap_size_limit.c:43]: (error) Possible null pointer dereference: ctx - otherwise it is redundant to check if ctx is null at line 46 [libjail/jail_getid.c:103]: (error) Uninitialized variable: namebuf [libmd/mdXhl.c:63]: (error) Resource leak: f [libncp/ncpl_conn.c:495]: (error) Resource leak: d [libncp/ncpl_file.c:89]: (error) Resource leak: d [libprocstat/libprocstat.c:723]: (error) Memory leak: path [librt/timer.c:106]: (error) Memory leak: timer [libstand/bzipfs.c:194]: (error) Resource leak: rawfd [libstand/qdivrem.c:99]: (error) Division by zero Scan for /usr/src/bin/: [ps/print.c:427]: (error) Memory leak: buf [ps/print.c:457]: (error) Memory leak: buf [sh/jobs.c:825]: (error) Allocation with open, if doesn't release it. [sh/mknodes.c:269]: (error) Resource leak: hfile [sh/mknodes.c:269]: (error) Resource leak: patfile [pax/cache.c:333]: (error) Possible null pointer dereference: ptr - otherwise it is redundant to check if ptr is null at line 345 [pax/cache.c:397]: (error) Possible null pointer dereference: ptr - otherwise it is redundant to check if ptr is null at line 408 Scan for /usr/src/cddl/: contrib/opensolaris/cmd/dtrace/test/cmd/baddof/baddof.c:202]: (error) Deallocating a deallocated pointer: fd [contrib/opensolaris/cmd/lockstat/sym.c:150]: (error) Memory leak: name [contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c:311]: (error) Memory leak: buffer [contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c:503]: (error) Memory leak: buf [contrib/opensolaris/cmd/sgs/tools/common/sgsmsg.c:950]: (error) Common realloc mistake: 'token_buffer' nulled but not freed upon failure [contrib/opensolaris/cmd/zpool/zpool_main.c:4622]: (error) Resource leak: fd [contrib/opensolaris/lib/libdtrace/common/dt_aggregate.c:568]: (error) Memory leak: percpu [contrib/opensolaris/lib/libdtrace/common/dt_cc.c:2117]: (error) Resource leak: dirp [contrib/opensolaris/lib/libdtrace/common/dt_link.c:1735]: (error) Resource leak: fd [contrib/opensolaris/lib/libdtrace/common/dt_strtab.c:257]: (error) Memory leak: hp [contrib/opensolaris/lib/libzfs/common/libzfs_import.c:1006]: (error) Dangerous usage of 'diskname' (strncpy doesn't always 0-terminate it) [contrib/opensolaris/tools/ctf/cvt/dwarf.c:503]: (error) Possible null pointer dereference: loc - otherwise it is redundant to check if loc is null at line 505 Scan for /usr/src/usr.bin/: [column/column.c:272]: (error) Memory leak: lens [cpio/cmdline.c:348]: (error) Memory leak: user [csup/updater.c:1905]: (error) Resource leak: fd [csup/updater.c:1988]: (error) Resource leak: orig [elf2aout/elf2aout.c:154]: (error) Resource leak: fd [env/envopts.c:467]: (error) Memory leak: newstr [finger/lprint.c:306]: (error) Resource leak: fd [grep/regex/tre-fastmatch.c:534]: (error) Memory leak: p [grep/regex/tre-fastmatch.c:537]: (error) Memory leak: wp [grep/regex/tre-fastmatch.c:766]: (error) Memory leak: p [grep/regex/tre-fastmatch.c:769]: (error) Memory leak: wp [gzip/gzip.c:1272]: (error) Resource leak: in [join/join.c:495]: (error) Possible null pointer dereference: olist - otherwise it is redundant to check if olist is null at line 485 [last/last.c:283]: (error) Possible null pointer dereference: tt - otherwise it is redundant to check if tt is null at line 286 [login/login.c:624]: (error) Memory leak: cx [m4/main.c:643]: (error) Common realloc mistake: 'sstack' nulled but not freed upon failure [make/var.c:1764]: (error) Uninitialized variable: error [makewhatis/makewhatis.c:351]: (error) Resource leak: output [msgs/msgs.c:804]: (error) Resource leak: cpfrom [newkey/update.c:273]: (error) Resource leak: rf [newkey/update.c:320]: (error) Memory leak: tmpname [pr/pr.c:211]: (error) Memory leak: obuf [pr/pr.c:362]: (error) Memory leak: buf [pr/pr.c:383]: (error) Memory leak: vc [pr/pr.c:395]: (error) Memory leak: indy [pr/pr.c:669]: (error) Memory leak: buf [sed/compile.c:842]: (error) Memory leak: p [showmount/showmount.c:332]: (error) Memory leak: ep [showmount/showmount.c:341]: (error) Memory leak: gp [tip/tip/cmds.c:129]: (error) Resource leak: fd [truss/amd64-fbsd.c:214]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 218 [truss/amd64-fbsd32.c:217]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 221 [truss/amd64-linux32.c:187]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 191 [truss/i386-fbsd.c:207]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 211 [truss/i386-linux.c:187]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 191 [truss/ia64-fbsd.c:189]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 193 [truss/mips-fbsd.c:234]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 238 [truss/powerpc-fbsd.c:221]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 225 [truss/powerpc64-fbsd.c:209]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 213 [truss/sparc64-fbsd.c:232]: (error) Possible null pointer dereference: sc - otherwise it is redundant to check if sc is null at line 236 [truss/syscalls.c:518]: (error) Common realloc mistake: 'buf' nulled but not freed upon failure [tset/wrterm.c:114]: (error) Memory leak: sep [unvis/unvis.c:69]: (error) Resource leak: fp [vgrind/vgrindefs.c:119]: (error) Resource leak: tf [vgrind/vgrindefs.c:150]: (error) Possible null pointer dereference: q - otherwise it is redundant to check if q is null at line 148 [vis/vis.c:112]: (error) Resource leak: fp [yacc/lr0.c:289]: (error) The given size 102 is mismatching Scan for /usr/src/usr.sbin: [acpi/acpidb/acpidb.c:373]: (error) Resource leak: fd [acpi/acpidump/acpi.c:1199]: (error) Dangerous usage of 'tmpstr' (strncpy doesn't always 0-terminate it) [asf/asf.c:186]: (error) Resource leak: objcopy [asf/asf_prog.c:73]: (error) Resource leak: kldstat [bluetooth/bt3cfw/bt3cfw.c:225]: (error) Resource leak: firmware_file [boot0cfg/boot0cfg.c:337]: (error) Resource leak: fd [bootparamd/bootparamd/bootparamd.c:239]: (error) Resource leak: bpf [bsdinstall/distextract/distextract.c:52]: (error) Memory leak: diststring [bsdinstall/distfetch/distfetch.c:51]: (error) Memory leak: diststring [bsdinstall/partedit/gpart_ops.c:1218]: (error) Possible null pointer dereference: cp - otherwise it is redundant to check if cp is null at line 1222 [bsdinstall/partedit/gpart_ops.c:307]: (error) Resource leak: bootfd [bsdinstall/partedit/gpart_ops.c:950]: (error) Possible null pointer dereference: gc - otherwise it is redundant to check if gc is null at line 952 [bsdinstall/partedit/part_wizard.c:136]: (error) Undefined behavior: variable is used as parameter and destination in s[n]printf(). [bsdinstall/partedit/part_wizard.c:182]: (error) Possible null pointer dereference: pp - otherwise it is redundant to check if pp is null at line 185 [bsdinstall/partedit/part_wizard.c:206]: (error) Possible null pointer dereference: classp - otherwise it is redundant to check if classp is null at line 209 [bsdinstall/partedit/partedit.c:202]: (error) Possible null pointer dereference: md - otherwise it is redundant to check if md is null at line 205 [bsnmpd/modules/snmp_bridge/bridge_sys.c:1249]: (error) Possible null pointer dereference: bp - otherwise it is redundant to check if bp is null at line 1247 [bsnmpd/modules/snmp_hostres/hostres_fs_tbl.c:161]: (error) Possible null pointer dereference: map - otherwise it is redundant to check if map is null at line 164 [bsnmpd/modules/snmp_hostres/hostres_network_tbl.c:182]: (error) Possible null pointer dereference: net - otherwise it is redundant to check if net is null at line 185 [bsnmpd/modules/snmp_hostres/hostres_partition_tbl.c:172]: (error) Possible null pointer dereference: map - otherwise it is redundant to check if map is null at line 175 [bsnmpd/modules/snmp_hostres/hostres_storage_tbl.c:150]: (error) Possible null pointer dereference: map - otherwise it is redundant to check if map is null at line 153 [bsnmpd/tools/libbsnmptools/bsnmptools.c:1084]: (error) Invalid deallocation [config/main.c:506]: (error) Memory leak: p [config/main.c:516]: (error) Allocation with path, moveifchanged doesn't release it. [config/main.c:612]: (error) Allocation with path, unlink doesn't release it. [config/mkmakefile.c:685]: (error) Memory leak: suff [config/mkoptions.c:190]: (error) Possible null pointer dereference: ol - otherwise it is redundant to check if ol is null at line 227 [config/mkoptions.c:220]: (error) Possible null pointer dereference: ol - otherwise it is redundant to check if ol is null at line 227 [config/mkoptions.c:382]: (error) Memory leak: this [cron/lib/compat.c:235]: (error) Memory leak: tmp [crunch/crunchgen/crunchgen.c:733]: (error) Mismatching allocation and deallocation: f [ctm/ctm/ctm_input.c:90]: (error) Memory leak: p [edquota/edquota.c:558]: (error) Resource leak: fd [edquota/edquota.c:736]: (error) Resource leak: fd [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: ''. [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: 'HAVE_POLL_H'. [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: 'IPV6_FAITH'. [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: 'IPV6_V6ONLY'. [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: 'SA_NOCLDWAIT'. [faithd/faithd.c:633]: (error) Invalid number of character (() when these macros are defined: '__KAME__'. [fdcontrol/fdcontrol.c:208]: (error) Resource leak: fd [fdformat/fdformat.c:290]: (error) Resource leak: fd [fdread/fdread.c:295]: (error) Memory leak: trackbuf [fifolog/lib/fifolog_create.c:121]: (error) Resource leak: fd [fwcontrol/fwcontrol.c:154]: (error) Possible null pointer dereference: data - otherwise it is redundant to check if data is null at line 155 [kbdcontrol/kbdcontrol.c:163]: (error) Common realloc mistake: 'buf' nulled but not freed upon failure [lpr/common_source/common.c:186]: (error) Memory leak: q [lpr/common_source/ctlinfo.c:294]: (error) Resource leak: cfile [lpr/lpd/lpd.c:423]: (error) Resource leak: lfd [lpr/lpd/printjob.c:653]: (error) Dereferencing 'fi' after it is deallocated / released [makefs/cd9660.c:2038]: (error) Memory leak: temp [makefs/cd9660.c:2065]: (error) Memory leak: temp [makefs/cd9660.c:701]: (error) Array 'record.ext_attr_length[0]' index 0 out of bounds [makefs/cd9660/cd9660_eltorito.c:233]: (error) Memory leak: temp [makefs/compat/pwcache.c:391]: (error) Possible null pointer dereference: ptr - otherwise it is redundant to check if ptr is null at line 404 [makefs/compat/pwcache.c:455]: (error) Possible null pointer dereference: ptr - otherwise it is redundant to check if ptr is null at line 468 [mixer/mixer.c:183]: (error) Resource leak: baz [mlxcontrol/command.c:580]: (error) Resource leak: fd [mlxcontrol/command.c:634]: (error) Resource leak: fd [mlxcontrol/interface.c:103]: (error) Resource leak: ctrlfd [mlxcontrol/interface.c:262]: (error) Assigning address of local auto-variable to a function parameter. [mlxcontrol/interface.c:263]: (error) Assigning address of local auto-variable to a function parameter. [mlxcontrol/interface.c:264]: (error) Assigning address of local auto-variable to a function parameter. [mptutil/mpt_config.c:113]: (error) Resource leak: vfd [mptutil/mpt_config.c:129]: (error) Resource leak: dfd [mptutil/mpt_config.c:713]: (error) Memory leak: info [ndiscvt/ndiscvt.c:215]: (error) Memory leak: outfile [newsyslog/newsyslog.c:1950]: (error) Possible null pointer dereference: zwork - otherwise it is redundant to check if zwork is null at line 1951 [ngctl/main.c:205]: (error) Resource leak: fp [pciconf/pciconf.c:645]: (error) Resource leak: fd [pciconf/pciconf.c:664]: (error) Resource leak: fd [pkg_install/lib/exec.c:50]: (error) Memory leak: cmd [pkg_install/lib/exec.c:79]: (error) Memory leak: rp [pkg_install/lib/plist.c:592]: (error) Memory leak: cp1 [pkg_install/lib/url.c:115]: (error) Resource leak: pkgfd [pkg_install/updating/main.c:125]: (error) Possible null pointer dereference: curr - otherwise it is redundant to check if curr is null at line 122 [pkg_install/updating/main.c:126]: (error) Possible null pointer dereference: curr - otherwise it is redundant to check if curr is null at line 122 [pkg_install/updating/main.c:220]: (error) Memory leak: dateline [pmcstat/pmcstat_log.c:862]: (error) Possible null pointer dereference: pcm - otherwise it is redundant to check if pcm is null at line 865 [ppp/lqr.c:204]: (error) Possible null pointer dereference: p - otherwise it is redundant to check if p is null at line 207 [pwd_mkdb/pwd_mkdb.c:710]: (error) Resource leak: from_fd [pwd_mkdb/pwd_mkdb.c:710]: (error) Resource leak: to_fd [rpc.ypupdated/update.c:270]: (error) Resource leak: rf [rpc.ypupdated/update.c:317]: (error) Memory leak: tmpname [rpcbind/rpcb_svc_com.c:414]: (error) Resource leak: fd [rpcbind/rpcbind.c:661]: (error) Memory leak: pml [rtadvd/advcap.c:169]: (error) Resource leak: tf [sade/dmenu.c:120]: (error) Memory leak: var [sade/label.c:1231]: (error) Possible null pointer dereference: pi - otherwise it is redundant to check if pi is null at line 1233 [services_mkdb/uniq.c:126]: (error) Memory leak: cline [sicontrol/sicontrol.c:243]: (error) Memory leak: acp [sysinstall/config.c:426]: (error) Resource leak: rcSite [sysinstall/dispatch.c:465]: (error) Memory leak: list [sysinstall/dispatch.c:518]: (error) Memory leak: list [sysinstall/dispatch.c:562]: (error) Common realloc mistake: 'menu' nulled but not freed upon failure [sysinstall/dispatch.c:653]: (error) Memory leak: list [sysinstall/dmenu.c:160]: (error) Memory leak: var [sysinstall/index.c:715]: (error) Possible null pointer dereference: id - otherwise it is redundant to check if id is null at line 719 [sysinstall/label.c:1217]: (error) Possible null pointer dereference: pi - otherwise it is redundant to check if pi is null at line 1219 [sysinstall/media.c:538]: (error) Memory leak: ufsDevice.private [sysinstall/modules.c:168]: (error) Common realloc mistake: 'menu' nulled but not freed upon failure [sysinstall/modules.c:194]: (error) Possible null pointer dereference: menu [sysinstall/modules.c:204]: (error) Resource leak: dir [sysinstall/tcpip.c:284]: (error) Mismatching allocation and deallocation: ifp [sysinstall/tcpip.c:339]: (error) Mismatching allocation and deallocation: ifp [tzsetup/tzsetup.c:609]: (error) Resource leak: fd1 [tzsetup/tzsetup.c:627]: (error) Resource leak: fd2 [ypserv/yp_dnslookup.c:516]: (error) Memory leak: q [ypserv/yp_main.c:364]: (error) Memory leak: sname PS: Other catalogs as contrib, sys weren't checked - probably and it makes sense From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 03:47:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A4EC106566B for ; Fri, 18 Nov 2011 03:47:01 +0000 (UTC) (envelope-from daryl@ci.com.au) Received: from mippet.ci.com.au (mippet.ci.com.au [192.65.182.30]) by mx1.freebsd.org (Postfix) with ESMTP id 45ACB8FC0C for ; Fri, 18 Nov 2011 03:46:59 +0000 (UTC) Received: from mippet.ci.com.au (localhost [127.0.0.1]) by mippet.ci.com.au (8.14.4/8.14.4/CE101231/cmlga) with ESMTP id pAI3ARLH075118 for ; Fri, 18 Nov 2011 14:10:28 +1100 (EST) (envelope-from daryl@mippet.ci.com.au) Received: (from daryl@localhost) by mippet.ci.com.au (8.14.4/8.14.4/Submit) id pAI3ARbZ075115; Fri, 18 Nov 2011 14:10:27 +1100 (EST) (envelope-from daryl) Date: Fri, 18 Nov 2011 14:10:27 +1100 (EST) From: Daryl Sayers Message-Id: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> To: freebsd-stable@freebsd.org X-Scanned-By: MIMEDefang 2.68 on 192.65.182.30 Subject: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 03:47:01 -0000 Can anyone suggest why I am getting poor write performance from my nfs setup. I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with onboard Gb network cards connected to an idle network. The results below show that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It improves if I use async but a smbfs mount still beats it. I am using the same file, source and destinations for all tests. I have tried alternate Network cards with no resulting benefit. oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 1950511+1 records in 1950511+1 records out 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) 1950477+74 records in 1950511+1 records out 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) oguido# mount -t smbfs //gemini/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) Looking at a systat -v on the destination I see that the nfs test does not exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. For the record I get reads of 22Mb/s without and 77Mb/s with async turned on for the nfs mount. A copy of dmesg: ================ Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3fd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3141234688 (2995 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 cryptosoft0: on motherboard acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, bff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 mpt0: port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 mpt0: [ITHREAD] mpt0: MPI Version=1.5.18.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) uhci0: port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus0: on uhci0 uhci1: port 0xe000-0xe01f irq 17 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus1: on uhci1 ehci0: mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus2: EHCI version 1.0 usbus2: on ehci0 pci0: at device 27.0 (no driver attached) pcib2: irq 16 at device 28.0 on pci0 pci5: on pcib2 atapci0: port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pcib3: irq 17 at device 28.1 on pci0 pci4: on pcib3 em0: port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:04:ac:11 pcib4: irq 19 at device 28.3 on pci0 pci3: on pcib4 age0: mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 age0: 1280 Tx FIFO, 2364 Rx FIFO age0: Using 1 MSI messages. miibus0: on age0 atphy0: PHY 0 on miibus0 atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto age0: Ethernet address: 00:1a:92:d2:de:cc age0: [FILTER] pcib5: irq 16 at device 28.4 on pci0 pci2: on pcib5 atapci1: port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 atapci1: [ITHREAD] atapci2: on atapci1 atapci2: [ITHREAD] atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] uhci2: port 0xd480-0xd49f irq 23 at device 29.0 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus3: on uhci2 uhci3: port 0xd800-0xd81f irq 19 at device 29.1 on pci0 uhci3: [ITHREAD] uhci3: LegSup = 0x2f00 usbus4: on uhci3 uhci4: port 0xd880-0xd89f irq 18 at device 29.2 on pci0 uhci4: [ITHREAD] uhci4: LegSup = 0x2f00 usbus5: on uhci4 ehci1: mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus6: EHCI version 1.0 usbus6: on ehci1 pcib6: at device 30.0 on pci0 pci6: on pcib6 vgapci0: mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 fwohci0: port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 fwohci0: [ITHREAD] fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:11:d8:00:01:39:52:d3 fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x1494000 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 fwe0: Ethernet address: 02:11:d8:39:52:d3 fwip0: on firewire0 fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode isab0: at device 31.0 on pci0 isa0: on isab0 atapci3: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 atapci3: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] ata8: on atapci3 ata8: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci4: port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 atapci4: [ITHREAD] ata9: on atapci4 ata9: [ITHREAD] ata10: on atapci4 ata10: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 acd0: DVDR at ata7-slave UDMA100 SATA 1.5Gb/s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub2: 4 ports with 4 removable, self powered uhub6: 6 ports with 6 removable, self powered da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing enabled da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 300.000MB/s transfers da2: Command Queueing enabled da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) SMP: AP CPU #1 Launched! ugen1.2: at usbus1 ugen3.2: at usbus3 ukbd1: on usbus1 ums0: on usbus3 ums0: 3 buttons and [XYZ] coordinates ID=0 kbd2 at ukbd1 uhid0: on usbus1 Trying to mount root from ufs:/dev/da0s1a -- Daryl Sayers Direct: +612 95525510 Corinthian Engineering Office: +612 95525500 Suite 54, Jones Bay Wharf Fax: +612 95525549 26-32 Pirrama Rd email: daryl@ci.com.au Pyrmont NSW 2009 Australia www: http://www.ci.com.au From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 08:01:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 792E6106564A; Fri, 18 Nov 2011 08:01:01 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 8C54114E685; Fri, 18 Nov 2011 08:00:57 +0000 (UTC) Message-ID: <4EC610B9.8080406@FreeBSD.org> Date: Fri, 18 Nov 2011 00:00:57 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17AAF.9050807@FreeBSD.org> <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111117105744.GS50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 08:01:01 -0000 On 11/17/2011 02:57, Kostik Belousov wrote: >> > It's not catching there though: >> > >> > Reading symbols from /libexec/ld-elf.so.1...done. >> > Loaded symbols for /libexec/ld-elf.so.1 >> > 0x28183b2d in accept () at accept.S:3 >> > 3 RSYSCALL(accept) >> > (gdb) c >> > Continuing. >> > no thread to satisfy query >> > 0x28183b2d in accept () at accept.S:3 >> > 3 RSYSCALL(accept) >> > (gdb) info threads >> > Cannot get thread info: invalid key >> > (gdb) > Err, the other part of my message was that you shall set the breakpoint > on sigprocmask. I'm sorry I'm not making myself clear. We are setting the breakpoint on sigprocmask. But, maybe I'm doing it wrong. Can you give precise instructions as to what you want me to do, from the beginning? Sorry to be so dense. > I want to see a backtrace from the breakpoint hit. > Several times. Me too. :) Meanwhile, in response to one of the other questions, we are using mpm_prefork. Also, the particular problem we're seeing does not appear related to fork(). The pattern of sigprocmask() calls is different from the pattern you see with fork(). Doug -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 08:14:41 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32FD2106566C for ; Fri, 18 Nov 2011 08:14:41 +0000 (UTC) (envelope-from bane.ivosev@pmf.uns.ac.rs) Received: from mail.pmf.uns.ac.rs (mail.pmf.uns.ac.rs [147.91.177.13]) by mx1.freebsd.org (Postfix) with ESMTP id 564C88FC1A for ; Fri, 18 Nov 2011 08:14:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.pmf.uns.ac.rs (Postfix) with ESMTP id F38E51AF46F; Fri, 18 Nov 2011 09:14:38 +0100 (CET) X-Virus-Scanned: amavisd-new at pmf.uns.ac.rs Received: from mail.pmf.uns.ac.rs ([127.0.0.1]) by localhost (mail.pmf.uns.ac.rs [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RBgLr7cCJBP3; Fri, 18 Nov 2011 09:14:24 +0100 (CET) Received: from [10.0.4.20] (unknown [10.0.4.20]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bane.ivosev@pmf.uns.ac.rs) by mail.pmf.uns.ac.rs (Postfix) with ESMTPSA id 0CB131AF41A; Fri, 18 Nov 2011 09:14:24 +0100 (CET) Message-ID: <4EC613DF.2000008@pmf.uns.ac.rs> Date: Fri, 18 Nov 2011 09:14:23 +0100 From: Bane Ivosev Organization: PMF DMI User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: Daryl Sayers , freebsd-stable@freebsd.org References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> X-Enigmail-Version: 1.3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 08:14:41 -0000 did you try this ? sysctl -w vfs.nfsrv.async=1 On 11/18/11 04:10, Daryl Sayers wrote: > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ================ > > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff > Features2=0xe3fd > AMD Features=0x20100000 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > mpt0: port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: on uhci0 > uhci1: port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: on uhci1 > ehci0: mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: on ehci0 > pci0: at device 27.0 (no driver attached) > pcib2: irq 16 at device 28.0 on pci0 > pci5: on pcib2 > atapci0: port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: on atapci0 > ata2: [ITHREAD] > ata3: on atapci0 > ata3: [ITHREAD] > pcib3: irq 17 at device 28.1 on pci0 > pci4: on pcib3 > em0: port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: irq 19 at device 28.3 on pci0 > pci3: on pcib4 > age0: mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: on age0 > atphy0: PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: irq 16 at device 28.4 on pci0 > pci2: on pcib5 > atapci1: port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: on atapci2 > ata4: [ITHREAD] > ata5: on atapci2 > ata5: [ITHREAD] > ata6: on atapci1 > ata6: [ITHREAD] > uhci2: port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: on uhci2 > uhci3: port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: on uhci3 > uhci4: port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: on uhci4 > ehci1: mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: on ehci1 > pcib6: at device 30.0 on pci0 > pci6: on pcib6 > vgapci0: mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: on fwohci0 > dcons_crom0: on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci3: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: on atapci3 > ata7: [ITHREAD] > ata8: on atapci3 > ata8: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atapci4: port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: on atapci4 > ata9: [ITHREAD] > ata10: on atapci4 > ata10: [ITHREAD] > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: on ppc0 > plip0: on ppbus0 > plip0: [ITHREAD] > lpt0: on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: on ppbus0 > pmtimer0 on isa0 > orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > ugen2.1: at usbus2 > uhub2: on usbus2 > ugen3.1: at usbus3 > uhub3: on usbus3 > ugen4.1: at usbus4 > uhub4: on usbus4 > ugen5.1: at usbus5 > uhub5: on usbus5 > ugen6.1: at usbus6 > uhub6: on usbus6 > acd0: DVDR at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: at usbus1 > ugen3.2: at usbus3 > ukbd1: on usbus1 > ums0: on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: on usbus1 > Trying to mount root from ufs:/dev/da0s1a > > From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 08:17:45 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2358B106564A for ; Fri, 18 Nov 2011 08:17:45 +0000 (UTC) (envelope-from bane.ivosev@pmf.uns.ac.rs) Received: from mail.pmf.uns.ac.rs (mail.pmf.uns.ac.rs [147.91.177.13]) by mx1.freebsd.org (Postfix) with ESMTP id AE28F8FC14 for ; Fri, 18 Nov 2011 08:17:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.pmf.uns.ac.rs (Postfix) with ESMTP id BB8821AF41A; Fri, 18 Nov 2011 09:17:43 +0100 (CET) X-Virus-Scanned: amavisd-new at pmf.uns.ac.rs Received: from mail.pmf.uns.ac.rs ([127.0.0.1]) by localhost (mail.pmf.uns.ac.rs [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1vioASm-FTvj; Fri, 18 Nov 2011 09:17:13 +0100 (CET) Received: from [10.0.4.20] (unknown [10.0.4.20]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: bane.ivosev@pmf.uns.ac.rs) by mail.pmf.uns.ac.rs (Postfix) with ESMTPSA id CBB791AF46D; Fri, 18 Nov 2011 09:17:13 +0100 (CET) Message-ID: <4EC61489.8080304@pmf.uns.ac.rs> Date: Fri, 18 Nov 2011 09:17:13 +0100 From: Bane Ivosev Organization: PMF DMI User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: Daryl Sayers , freebsd-stable@freebsd.org References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> X-Enigmail-Version: 1.3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 08:17:45 -0000 and if you use zfs also try this zfs set sync=disabled On 11/18/11 04:10, Daryl Sayers wrote: > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. > > > A copy of dmesg: > ================ > > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > Features=0xbfebfbff > Features2=0xe3fd > AMD Features=0x20100000 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 4294967296 (4096 MB) > avail memory = 3141234688 (2995 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, bff00000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > cpu0: on acpi0 > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) > cpu1: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > mpt0: port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.18.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > uhci0: port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup = 0x2f00 > usbus0: on uhci0 > uhci1: port 0xe000-0xe01f irq 17 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup = 0x2f00 > usbus1: on uhci1 > ehci0: mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus2: EHCI version 1.0 > usbus2: on ehci0 > pci0: at device 27.0 (no driver attached) > pcib2: irq 16 at device 28.0 on pci0 > pci5: on pcib2 > atapci0: port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 > atapci0: [ITHREAD] > ata2: on atapci0 > ata2: [ITHREAD] > ata3: on atapci0 > ata3: [ITHREAD] > pcib3: irq 17 at device 28.1 on pci0 > pci4: on pcib3 > em0: port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 > em0: Using an MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:04:ac:11 > pcib4: irq 19 at device 28.3 on pci0 > pci3: on pcib4 > age0: mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > age0: 1280 Tx FIFO, 2364 Rx FIFO > age0: Using 1 MSI messages. > miibus0: on age0 > atphy0: PHY 0 on miibus0 > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto > age0: Ethernet address: 00:1a:92:d2:de:cc > age0: [FILTER] > pcib5: irq 16 at device 28.4 on pci0 > pci2: on pcib5 > atapci1: port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > atapci1: [ITHREAD] > atapci2: on atapci1 > atapci2: [ITHREAD] > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata4: on atapci2 > ata4: [ITHREAD] > ata5: on atapci2 > ata5: [ITHREAD] > ata6: on atapci1 > ata6: [ITHREAD] > uhci2: port 0xd480-0xd49f irq 23 at device 29.0 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup = 0x2f00 > usbus3: on uhci2 > uhci3: port 0xd800-0xd81f irq 19 at device 29.1 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup = 0x2f00 > usbus4: on uhci3 > uhci4: port 0xd880-0xd89f irq 18 at device 29.2 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup = 0x2f00 > usbus5: on uhci4 > ehci1: mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus6: EHCI version 1.0 > usbus6: on ehci1 > pcib6: at device 30.0 on pci0 > pci6: on pcib6 > vgapci0: mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 > fwohci0: port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > fwohci0: [ITHREAD] > fwohci0: OHCI version 1.10 (ROM=1) > fwohci0: No. of Isochronous channels is 4. > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > fwohci0: Phy 1394a available S400, 2 ports. > fwohci0: Link S400, max_rec 2048 bytes. > firewire0: on fwohci0 > dcons_crom0: on firewire0 > dcons_crom0: bus_addr 0x1494000 > fwe0: on firewire0 > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > fwe0: Ethernet address: 02:11:d8:39:52:d3 > fwip0: on firewire0 > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 > fwohci0: Initiate bus reset > fwohci0: fwohci_intr_core: BUS reset > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci3: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 > atapci3: [ITHREAD] > ata7: on atapci3 > ata7: [ITHREAD] > ata8: on atapci3 > ata8: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atapci4: port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 > atapci4: [ITHREAD] > ata9: on atapci4 > ata9: [ITHREAD] > ata10: on atapci4 > ata10: [ITHREAD] > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart0: [FILTER] > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > fdc0: [FILTER] > acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppc0: [ITHREAD] > ppbus0: on ppc0 > plip0: on ppbus0 > plip0: [ITHREAD] > lpt0: on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: on ppbus0 > pmtimer0 on isa0 > orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > ata0: [ITHREAD] > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > ata1: [ITHREAD] > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > firewire0: bus manager 0 > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 480Mbps High Speed USB v2.0 > usbus3: 12Mbps Full Speed USB v1.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 480Mbps High Speed USB v2.0 > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > ugen2.1: at usbus2 > uhub2: on usbus2 > ugen3.1: at usbus3 > uhub3: on usbus3 > ugen4.1: at usbus4 > uhub4: on usbus4 > ugen5.1: at usbus5 > uhub5: on usbus5 > ugen6.1: at usbus6 > uhub6: on usbus6 > acd0: DVDR at ata7-slave UDMA100 SATA 1.5Gb/s > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub3: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub2: 4 ports with 4 removable, self powered > uhub6: 6 ports with 6 removable, self powered > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 300.000MB/s transfers > da2: Command Queueing enabled > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing enabled > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > SMP: AP CPU #1 Launched! > ugen1.2: at usbus1 > ugen3.2: at usbus3 > ukbd1: on usbus1 > ums0: on usbus3 > ums0: 3 buttons and [XYZ] coordinates ID=0 > kbd2 at ukbd1 > uhid0: on usbus1 > Trying to mount root from ufs:/dev/da0s1a > > From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 09:19:42 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 151BF106566B; Fri, 18 Nov 2011 09:19:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id A21F48FC0C; Fri, 18 Nov 2011 09:19:41 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pAI9Jboh001798 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pAI9Jb8P016837; Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pAI9JbX3016836; Fri, 18 Nov 2011 11:19:37 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 18 Nov 2011 11:19:37 +0200 From: Kostik Belousov To: Doug Barton Message-ID: <20111118091937.GA50300@deviant.kiev.zoral.com.ua> References: <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> <4EC610B9.8080406@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZEJo5p0yAM1jsdkh" Content-Disposition: inline In-Reply-To: <4EC610B9.8080406@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 09:19:42 -0000 --ZEJo5p0yAM1jsdkh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 18, 2011 at 12:00:57AM -0800, Doug Barton wrote: > On 11/17/2011 02:57, Kostik Belousov wrote: > >> > It's not catching there though: > >> >=20 > >> > Reading symbols from /libexec/ld-elf.so.1...done. > >> > Loaded symbols for /libexec/ld-elf.so.1 > >> > 0x28183b2d in accept () at accept.S:3 > >> > 3 RSYSCALL(accept) > >> > (gdb) c > >> > Continuing. > >> > no thread to satisfy query > >> > 0x28183b2d in accept () at accept.S:3 > >> > 3 RSYSCALL(accept) > >> > (gdb) info threads > >> > Cannot get thread info: invalid key > >> > (gdb) > > Err, the other part of my message was that you shall set the breakpoint > > on sigprocmask. >=20 > I'm sorry I'm not making myself clear. We are setting the breakpoint on > sigprocmask. But, maybe I'm doing it wrong. Can you give precise > instructions as to what you want me to do, from the beginning? Sorry to > be so dense. Find the pid of the process issuing excessive number of sigprocmask calls. Do $ gdb /usr/local/bin/httpd (gdb) attach (gdb) b _sigprocmask (gdb) c Bah ! Breakpoint fired. (gdb) bt (gdb) c <... Repeat ... > >=20 > > I want to see a backtrace from the breakpoint hit. > > Several times. >=20 > Me too. :) >=20 > Meanwhile, in response to one of the other questions, we are using > mpm_prefork. Also, the particular problem we're seeing does not appear > related to fork(). The pattern of sigprocmask() calls is different from > the pattern you see with fork(). I am sure that your sigprocmask calls do not come from rtld, it is some use of setjmp or sigsetjmp(1), most likely. I am not aware of any significant users of setjmp or sigprocmask in our system libraries. >=20 >=20 > Doug >=20 > --=20 >=20 > "We could put the whole Internet into a book." > "Too practical." >=20 > Breadth of IT experience, and depth of knowledge in the DNS. > Yours for the right price. :) http://SupersetSolutions.com/ --ZEJo5p0yAM1jsdkh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7GIykACgkQC3+MBN1Mb4gPtwCdEwGLoFiLVJsgTwpmRYNkeXd0 yuEAnj5wdYlX5mJ9r/3oWnVWsQviyC5J =tAQC -----END PGP SIGNATURE----- --ZEJo5p0yAM1jsdkh-- From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 13:14:47 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDC69106566B for ; Fri, 18 Nov 2011 13:14:47 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9596B8FC13 for ; Fri, 18 Nov 2011 13:14:47 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 4EB0946B3B; Fri, 18 Nov 2011 08:14:47 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id E7A8B8A051; Fri, 18 Nov 2011 08:14:46 -0500 (EST) From: John Baldwin To: freebsd-stable@freebsd.org Date: Fri, 18 Nov 2011 08:14:42 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> In-Reply-To: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201111180814.42656.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 18 Nov 2011 08:14:47 -0500 (EST) Cc: Daryl Sayers Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 13:14:47 -0000 On Thursday, November 17, 2011 10:10:27 pm Daryl Sayers wrote: > > Can anyone suggest why I am getting poor write performance from my nfs setup. > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with > onboard Gb network cards connected to an idle network. The results below show > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It > improves if I use async but a smbfs mount still beats it. I am using the same > file, source and destinations for all tests. I have tried alternate Network > cards with no resulting benefit. > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > 1950511+1 records in > 1950511+1 records out > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > 1950477+74 records in > 1950511+1 records out > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) > > > oguido# mount -t smbfs //gemini/ufs /mnt > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > 7619+1 records in > 7619+1 records out > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) > > Looking at a systat -v on the destination I see that the nfs test does not > exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. > For the record I get reads of 22Mb/s without and 77Mb/s with async turned on > for the nfs mount. I don't know if it will help with your performance, but I have some patches to allow the NFS server to cluster writes. You can try www.freebsd.org/~jhb/patches/nfs_server_cluster.patch. I've tested it on 8, but it should probably apply fine to 9. -- John Baldwin From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 17:23:49 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A9138106567A for ; Fri, 18 Nov 2011 17:23:49 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 373EC8FC0A for ; Fri, 18 Nov 2011 17:23:48 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsIEACqNxk6DaFvO/2dsb2JhbAA7CA6Ec6JAgRSCYYFyAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggCBQQBGgIEh2IIpGWSBoEwgimDDYIbgRYEiBeKA4IfkUpV X-IronPort-AV: E=Sophos;i="4.69,533,1315195200"; d="scan'208";a="144557803" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Nov 2011 11:54:43 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 197FEB3F20; Fri, 18 Nov 2011 11:54:43 -0500 (EST) Date: Fri, 18 Nov 2011 11:54:43 -0500 (EST) From: Rick Macklem To: Bane Ivosev Message-ID: <917971298.2052425.1321635283084.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4EC61489.8080304@pmf.uns.ac.rs> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-stable@freebsd.org, Daryl Sayers Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 17:23:49 -0000 Bane Ivosev wrote: > and if you use zfs also try this > > zfs set sync=disabled > I know diddly about zfs, but I believe some others have improved zfs performance for NFS writing by moving the ZIL log to a dedicated device, sometimes an SSD. Apparently (again I'm not knowledgible) you do have to be careful what SSD you use and how full you make it, if you want good write performance on the SSD. I should also note that use of these options (vfs.nfsrv.async=1 and the above for zfs) is risky in the sense that recently written data can be lost when a server crashes/reboots because the NFS clients don't know to hold onto the data and re-write it after a server crash/reboot. rick ps: NFS write performance has been an issue since SUN released their first implementation of it in 1985. The "big" server vendors typically solve the problem with lots of non-volatile RAM in the server boxes. (This solution requires server code that specifically knows how to use this non-volatile RAM. Such code is not in the FreeBSD servers.) > On 11/18/11 04:10, Daryl Sayers wrote: > > Can anyone suggest why I am getting poor write performance from my > > nfs setup. > > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus > > mother boards, > > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives > > with > > onboard Gb network cards connected to an idle network. The results > > below show > > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using > > nfs. It > > improves if I use async but a smbfs mount still beats it. I am using > > the same > > file, source and destinations for all tests. I have tried alternate > > Network > > cards with no resulting benefit. > > > > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 > > 1950511+1 records in > > 1950511+1 records out > > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) > > 1950477+74 records in > > 1950511+1 records out > > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) > > (98Mb/s) > > > > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs > > /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) > > (15Mb/s) > > > > > > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async > > gemini:/dsk/ufs /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) > > (19Mb/s) > > > > > > oguido# mount -t smbfs //gemini/ufs /mnt > > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k > > 7619+1 records in > > 7619+1 records out > > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) > > (33Mb/s) > > > > Looking at a systat -v on the destination I see that the nfs test > > does not > > exceed 16KB/t with 100% busy where the other tests reach up to > > 128KB/t. > > For the record I get reads of 22Mb/s without and 77Mb/s with async > > turned on > > for the nfs mount. > > > > > > A copy of dmesg: > > ================ > > > > Copyright (c) 1992-2011 The FreeBSD Project. > > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > > 1994 > > The Regents of the University of California. All rights > > reserved. > > FreeBSD is a registered trademark of The FreeBSD Foundation. > > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 > > root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 > > Timecounter "i8254" frequency 1193182 Hz quality 0 > > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz > > 686-class CPU) > > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = > > 11 > > Features=0xbfebfbff > > Features2=0xe3fd > > AMD Features=0x20100000 > > AMD Features2=0x1 > > TSC: P-state invariant > > real memory = 4294967296 (4096 MB) > > avail memory = 3141234688 (2995 MB) > > ACPI APIC Table: > > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > > FreeBSD/SMP: 1 package(s) x 2 core(s) > > cpu0 (BSP): APIC ID: 0 > > cpu1 (AP): APIC ID: 1 > > ioapic0 irqs 0-23 on motherboard > > kbd1 at kbdmux0 > > cryptosoft0: on motherboard > > acpi0: on motherboard > > acpi0: [ITHREAD] > > acpi0: Power Button (fixed) > > acpi0: reservation of 0, a0000 (3) failed > > acpi0: reservation of 100000, bff00000 (3) failed > > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > > cpu0: on acpi0 > > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be > > 0xB1 (20101013/tbutils-354) > > cpu1: on acpi0 > > pcib0: port 0xcf8-0xcff on acpi0 > > pci0: on pcib0 > > pcib1: irq 16 at device 1.0 on pci0 > > pci1: on pcib1 > > mpt0: port 0x7800-0x78ff mem > > 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on > > pci1 > > mpt0: [ITHREAD] > > mpt0: MPI Version=1.5.18.0 > > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > > mpt0: 0 Active Volumes (2 Max) > > mpt0: 0 Hidden Drive Members (14 Max) > > uhci0: port 0xdc00-0xdc1f > > irq 16 at device 26.0 on pci0 > > uhci0: [ITHREAD] > > uhci0: LegSup = 0x2f00 > > usbus0: on uhci0 > > uhci1: port 0xe000-0xe01f > > irq 17 at device 26.1 on pci0 > > uhci1: [ITHREAD] > > uhci1: LegSup = 0x2f00 > > usbus1: on uhci1 > > ehci0: mem > > 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 > > ehci0: [ITHREAD] > > usbus2: EHCI version 1.0 > > usbus2: on ehci0 > > pci0: at device 27.0 (no driver attached) > > pcib2: irq 16 at device 28.0 on pci0 > > pci5: on pcib2 > > atapci0: port 0xac00-0xac7f mem > > 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on > > pci5 > > atapci0: [ITHREAD] > > ata2: on atapci0 > > ata2: [ITHREAD] > > ata3: on atapci0 > > ata3: [ITHREAD] > > pcib3: irq 17 at device 28.1 on pci0 > > pci4: on pcib3 > > em0: port 0x9c00-0x9c1f > > mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 > > on pci4 > > em0: Using an MSI interrupt > > em0: [FILTER] > > em0: Ethernet address: 00:1b:21:04:ac:11 > > pcib4: irq 19 at device 28.3 on pci0 > > pci3: on pcib4 > > age0: mem > > 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 > > age0: 1280 Tx FIFO, 2364 Rx FIFO > > age0: Using 1 MSI messages. > > miibus0: on age0 > > atphy0: PHY 0 on miibus0 > > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, > > 1000baseT-FDX, 1000baseT-FDX-master, auto > > age0: Ethernet address: 00:1a:92:d2:de:cc > > age0: [FILTER] > > pcib5: irq 16 at device 28.4 on pci0 > > pci2: on pcib5 > > atapci1: port > > 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f > > mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 > > atapci1: [ITHREAD] > > atapci2: on atapci1 > > atapci2: [ITHREAD] > > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported > > ata4: on atapci2 > > ata4: [ITHREAD] > > ata5: on atapci2 > > ata5: [ITHREAD] > > ata6: on atapci1 > > ata6: [ITHREAD] > > uhci2: port 0xd480-0xd49f > > irq 23 at device 29.0 on pci0 > > uhci2: [ITHREAD] > > uhci2: LegSup = 0x2f00 > > usbus3: on uhci2 > > uhci3: port 0xd800-0xd81f > > irq 19 at device 29.1 on pci0 > > uhci3: [ITHREAD] > > uhci3: LegSup = 0x2f00 > > usbus4: on uhci3 > > uhci4: port 0xd880-0xd89f > > irq 18 at device 29.2 on pci0 > > uhci4: [ITHREAD] > > uhci4: LegSup = 0x2f00 > > usbus5: on uhci4 > > ehci1: mem > > 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 > > ehci1: [ITHREAD] > > usbus6: EHCI version 1.0 > > usbus6: on ehci1 > > pcib6: at device 30.0 on pci0 > > pci6: on pcib6 > > vgapci0: mem > > 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff > > irq 22 at device 1.0 on pci6 > > fwohci0: port 0xbc00-0xbc7f mem > > 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 > > fwohci0: [ITHREAD] > > fwohci0: OHCI version 1.10 (ROM=1) > > fwohci0: No. of Isochronous channels is 4. > > fwohci0: EUI64 00:11:d8:00:01:39:52:d3 > > fwohci0: Phy 1394a available S400, 2 ports. > > fwohci0: Link S400, max_rec 2048 bytes. > > firewire0: on fwohci0 > > dcons_crom0: on firewire0 > > dcons_crom0: bus_addr 0x1494000 > > fwe0: on firewire0 > > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 > > fwe0: Ethernet address: 02:11:d8:39:52:d3 > > fwip0: on firewire0 > > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, > > S400, maxrec 2048 > > fwohci0: Initiate bus reset > > fwohci0: fwohci_intr_core: BUS reset > > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, > > CYCLEMASTER mode > > isab0: at device 31.0 on pci0 > > isa0: on isab0 > > atapci3: port > > 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f > > irq 19 at device 31.2 on pci0 > > atapci3: [ITHREAD] > > ata7: on atapci3 > > ata7: [ITHREAD] > > ata8: on atapci3 > > ata8: [ITHREAD] > > pci0: at device 31.3 (no driver attached) > > atapci4: port > > 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f > > irq 19 at device 31.5 on pci0 > > atapci4: [ITHREAD] > > ata9: on atapci4 > > ata9: [ITHREAD] > > ata10: on atapci4 > > ata10: [ITHREAD] > > acpi_button0: on acpi0 > > atrtc0: port 0x70-0x71 irq 8 on acpi0 > > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on > > acpi0 > > uart0: [FILTER] > > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 > > drq 2 on acpi0 > > fdc0: [FILTER] > > acpi_hpet0: iomem 0xfed00000-0xfed003ff > > on acpi0 > > Timecounter "HPET" frequency 14318180 Hz quality 900 > > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on > > acpi0 > > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > > ppc0: FIFO with 16/16/9 bytes threshold > > ppc0: [ITHREAD] > > ppbus0: on ppc0 > > plip0: on ppbus0 > > plip0: [ITHREAD] > > lpt0: on ppbus0 > > lpt0: [ITHREAD] > > lpt0: Interrupt-driven port > > ppi0: on ppbus0 > > pmtimer0 on isa0 > > orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on > > isa0 > > sc0: at flags 0x100 on isa0 > > sc0: VGA <16 virtual consoles, flags=0x300> > > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on > > isa0 > > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 > > ata0: [ITHREAD] > > ata1 at port 0x170-0x177,0x376 irq 15 on isa0 > > ata1: [ITHREAD] > > atkbdc0: at port 0x60,0x64 on isa0 > > atkbd0: irq 1 on atkbdc0 > > kbd0 at atkbd0 > > atkbd0: [GIANT-LOCKED] > > atkbd0: [ITHREAD] > > est0: on cpu0 > > p4tcc0: on cpu0 > > est1: on cpu1 > > p4tcc1: on cpu1 > > Timecounters tick every 1.000 msec > > IPsec: Initialized Security Association Processing. > > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) > > firewire0: bus manager 0 > > usbus0: 12Mbps Full Speed USB v1.0 > > usbus1: 12Mbps Full Speed USB v1.0 > > usbus2: 480Mbps High Speed USB v2.0 > > usbus3: 12Mbps Full Speed USB v1.0 > > usbus4: 12Mbps Full Speed USB v1.0 > > usbus5: 12Mbps Full Speed USB v1.0 > > usbus6: 480Mbps High Speed USB v2.0 > > ugen0.1: at usbus0 > > uhub0: on > > usbus0 > > ugen1.1: at usbus1 > > uhub1: on > > usbus1 > > ugen2.1: at usbus2 > > uhub2: on > > usbus2 > > ugen3.1: at usbus3 > > uhub3: on > > usbus3 > > ugen4.1: at usbus4 > > uhub4: on > > usbus4 > > ugen5.1: at usbus5 > > uhub5: on > > usbus5 > > ugen6.1: at usbus6 > > uhub6: on > > usbus6 > > acd0: DVDR at ata7-slave UDMA100 > > SATA 1.5Gb/s > > uhub0: 2 ports with 2 removable, self powered > > uhub1: 2 ports with 2 removable, self powered > > uhub3: 2 ports with 2 removable, self powered > > uhub4: 2 ports with 2 removable, self powered > > uhub5: 2 ports with 2 removable, self powered > > uhub2: 4 ports with 4 removable, self powered > > uhub6: 6 ports with 6 removable, self powered > > da1 at mpt0 bus 0 scbus0 target 1 lun 0 > > da1: Fixed Direct Access SCSI-5 device > > da1: 300.000MB/s transfers > > da1: Command Queueing enabled > > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da2 at mpt0 bus 0 scbus0 target 2 lun 0 > > da2: Fixed Direct Access SCSI-5 device > > da2: 300.000MB/s transfers > > da2: Command Queueing enabled > > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da0 at mpt0 bus 0 scbus0 target 0 lun 0 > > da0: Fixed Direct Access SCSI-5 device > > da0: 300.000MB/s transfers > > da0: Command Queueing enabled > > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) > > SMP: AP CPU #1 Launched! > > ugen1.2: at usbus1 > > ugen3.2: at usbus3 > > ukbd1: on > > usbus1 > > ums0: > 1.10/1.99, addr 2> on usbus3 > > ums0: 3 buttons and [XYZ] coordinates ID=0 > > kbd2 at ukbd1 > > uhid0: on > > usbus1 > > Trying to mount root from ufs:/dev/da0s1a > > > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to > "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 20:07:51 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id DCB351065673; Fri, 18 Nov 2011 20:07:51 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 5E157160720; Fri, 18 Nov 2011 20:07:51 +0000 (UTC) Message-ID: <4EC6BB17.20706@FreeBSD.org> Date: Fri, 18 Nov 2011 12:07:51 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: Kostik Belousov References: <4EC17F57.5030008@FreeBSD.org> <20111115090745.GO50300@deviant.kiev.zoral.com.ua> <20111115100904.GA92795@icarus.home.lan> <4EC4ADC3.2060604@FreeBSD.org> <20111117074909.GL50300@deviant.kiev.zoral.com.ua> <4EC4BECA.5040705@FreeBSD.org> <20111117081210.GN50300@deviant.kiev.zoral.com.ua> <4EC4D359.4040406@FreeBSD.org> <20111117105744.GS50300@deviant.kiev.zoral.com.ua> <4EC610B9.8080406@FreeBSD.org> <20111118091937.GA50300@deviant.kiev.zoral.com.ua> In-Reply-To: <20111118091937.GA50300@deviant.kiev.zoral.com.ua> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Daniil Cherednik , freebsd-apache@freebsd.org, freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 8.2 + apache == a LOT of sigprocmask X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 20:07:52 -0000 On 11/18/2011 01:19, Kostik Belousov wrote: > On Fri, Nov 18, 2011 at 12:00:57AM -0800, Doug Barton wrote: >> On 11/17/2011 02:57, Kostik Belousov wrote: >>>>> It's not catching there though: >>>>> >>>>> Reading symbols from /libexec/ld-elf.so.1...done. >>>>> Loaded symbols for /libexec/ld-elf.so.1 >>>>> 0x28183b2d in accept () at accept.S:3 >>>>> 3 RSYSCALL(accept) >>>>> (gdb) c >>>>> Continuing. >>>>> no thread to satisfy query >>>>> 0x28183b2d in accept () at accept.S:3 >>>>> 3 RSYSCALL(accept) >>>>> (gdb) info threads >>>>> Cannot get thread info: invalid key >>>>> (gdb) >>> Err, the other part of my message was that you shall set the breakpoint >>> on sigprocmask. >> >> I'm sorry I'm not making myself clear. We are setting the breakpoint on >> sigprocmask. But, maybe I'm doing it wrong. Can you give precise >> instructions as to what you want me to do, from the beginning? Sorry to >> be so dense. > Find the pid of the process issuing excessive number of sigprocmask > calls. Do > $ gdb /usr/local/bin/httpd > (gdb) attach > (gdb) b _sigprocmask > (gdb) c > Bah ! Breakpoint fired. > (gdb) bt > (gdb) c > <... Repeat ... > Right, so we're on the same page at least. I've been abbreviating the output of gdb to make it easier to see the problem, but here is a (nearly) complete transcript: gdb /usr/local/bin/httpd Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd"... (gdb) attach 1380 Attaching to program: /usr/local/bin/httpd, process 1380 Reading symbols from .... (lots of symbol-reading snipped) 3 RSYSCALL(accept) Current language: auto; currently asm (gdb) b _sigprocmask Breakpoint 1 at 0x282d9055: file /usr/src/lib/libthr/thread/thr_sig.c, line 210. (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) (gdb) c Continuing. no thread to satisfy query 0x28183b2d in accept () at accept.S:3 3 RSYSCALL(accept) .... etc. -- "We could put the whole Internet into a book." "Too practical." Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 21:38:08 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 199481065672 for ; Fri, 18 Nov 2011 21:38:08 +0000 (UTC) (envelope-from rnw@niagara.retirementnewsweekly.ca) Received: from mail-07.primus.ca (mail16.primus.ca [216.254.141.183]) by mx1.freebsd.org (Postfix) with ESMTP id 7A4BA8FC15 for ; Fri, 18 Nov 2011 21:38:06 +0000 (UTC) Received: from d72-38-218-3.commercial1.cgocable.net ([72.38.218.3] helo=OWNER-4F95FE0GS) by mail-07.primus.ca with esmtpa (Exim 4.72) (envelope-from ) id 1RRW8L-0002GI-1c for freebsd-stable@freebsd.org; Fri, 18 Nov 2011 16:38:05 -0500 From: "After 50 News Weekly / Niagara" To: "freebsd-stable" MIME-Version: 1.0 Organization: PNC Date: Fri, 18 Nov 2011 16:38:05 -0500 X-Authenticated: grahamp - d72-38-218-3.commercial1.cgocable.net (OWNER-4F95FE0GS) [72.38.218.3] Message-Id: <20111118213808.199481065672@hub.freebsd.org> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: After 50 News Weekly / Niagara X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 21:38:08 -0000 After 50 News Weekly / Niagara =20 Date: Nov 18th, 2011Last Month's Readers:38,058 - This Months Readers:= 9,088=20 LOCAL NEWS Police warn of new fraud schemes The Niagara Regional Police are investigating two separate fraud cases= =2E The first fraud related issue involves people receiving unsolicited ph= one calls during which the person on the line inquires about problems = with the home owner's computer or suggests that their computer may be = infected with several viruses. The caller then directs the home owner = or resident to a website enabling them to connect to the home owner's = computer. Once connected, the caller identifies several issues with th= e computer and offers to repair the computer for a fee and asks the ho= me owner to provide a credit card number. Complaints received by polic= e have ranged from unauthorized charges to credit cards to people find= ing trojans or viruses on their computers after connecting to the site= provided. Police are warning the public to ensure they know who they = are dealing with when responding to unsolicited phone calls and before= giving out credit card or any personal information. Callers associate= d to this complaint are using numerous telephone numbers,=20 =2E.. read more NATIONAL NEWS Harper looks to Asia for free trade deal Prime Minister Stephen Harper is looking East as Canada.=20 =2E.. read more INTERNATIONAL NEWS International Education Week=20 The Council of Ministers of Education, Canada (CMEC)=20 =2E.. read more COLUMNIST Giving to Charity at the Beginning of the Christmas Season=20 Stores are decked with Christmas celebration. Holiday music greets us = when we turn on the radio. Most likely, we are already starting to mak= e our=20 =2E.. read more SPORTS NBA players file antitrust lawsuits=20 After the NBA Player's Association turned down the latest offer from t= he league, some locked-out players have filed=20 =2E.. read more LIFESTYLE ART AS A FORM OF THERAPY Art isn't just a painting you hang on your wall. For the person who pa= inted it, it was cathartic. If you look at a painting, not just the pi= cture but the actual brush strokes and lines,=20 =2E.. read more FEATURES ASK DOREEN=20 What are your qualifications? I have been a licensed realtor since 198= 5, selling and servicing real estate needs in the region =E2=80=93 NOT= L, Niagara Falls, Fort Erie, St. Catharines, both in resales and new h= ome=20 =2E.. read more ENVIRONMENT HOUSEHOLD CLEANERS: KICKING 'EM OLD SCHOOL Winter is coming and you might be thinking about giving the house a th= orough=20 =2E.. read more THIS WEEK IN HISTORY November 21 =E2=80=93 November 25=20 The #1 song this week in 1970 was The Partridge Family's I Think I Lov= e You. November 21 Today is World Television Day! On this day in:=20 =2E.. read more SPIN THE GLOBE Escape winter in Egypt=20 If you are in the market for an escape from the cold Canadian winter, = may I suggest a getaway that has all three S's: sun, sand, and sea? Ta= ke off to beautiful Egypt. Home to 82 million .=20 =2E.. read more FIFTY & FABULOUS TRANSITIONING YOUR SUMMER WARDROBE FOR FALL =20 With the first day of fall behind us, sadly it=E2=80=99s time to put a= way those shorts and sandals and pull out those cable knit sweaters. W= ho says Fall/Winter clothing has to be dull and dreary?! By pairing an= y one these 5 key pieces with your summer wardrobe you can instantly t= ransform 1. Scarves Scarves are an excellent=20 =2E.. read more YOUR RETIREMENT The Holiday Card=20 This may just be the first year I get my holiday cards out in time, if= I act quickly that is. Thankfully I=E2=80=99ve found plenty of gorgeo= us Letterpress cards to choose from. =2E.. read more EDITOR'S LETTER How do you measure a man and his career?=20 Good question, how do you? That was the question asked during the "Far= ewell Regis" celebration that aired Friday, November 18, 2011. After 2= 8 years on the air, Regis Philbin, one of the biggest names in televis= ion hung up his hat and said=20 =2E.. read more After 50 News Weekly is a resource tool for the 50-plus active lifestyle community in Niaga= ra. We provide you with up-to-date information on retirement, news and= local events, all at your fingertips! If there is a topic you would l= ike us to feature or if you have any questions or comments, please Con= tact Us. For our advertising rates, please contact Graham Pomeroy at g= raham@pomeroynewscorp.ca or call directly at 905-988-9339. Sincerely, Victoria Lewis, Director of Operations=20 Please do not reply to this email. Replies to this email will not be r= esponded to or read. If you have any questions or comments, contact us= by email at info@pomeroynewscorp. Cancel your subscription: UNSUBSCRIBE Sends us a news tip: newsroom@after50newsweekly.caTo Submit an Event: events@after50newsweekly.ca From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 00:36:48 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBB5D106564A; Sat, 19 Nov 2011 00:36:48 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5EF0E8FC08; Sat, 19 Nov 2011 00:36:48 +0000 (UTC) Received: by yenl11 with SMTP id l11so4514975yen.13 for ; Fri, 18 Nov 2011 16:36:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qDs+A311AtrBYVMZZPs3TFv6gFrX36qjVFmwRFx8q2E=; b=Si9IeAaMSc4+636zEqdwIdwgD/OMT6sTBvwabCdJ1OOfAgrMEWpt7TZYClXIW58a59 uF8eqn9r3NAL+bSaLl5FUwLZNq9f9BiZwIX9IXedKK1ups1xdQPI957XDG74YlS41NUc E+tq4OCFHFhVitzarbKnoO8d2yGfjwAdAUVEU= MIME-Version: 1.0 Received: by 10.182.41.69 with SMTP id d5mr1192201obl.47.1321663007585; Fri, 18 Nov 2011 16:36:47 -0800 (PST) Received: by 10.182.192.99 with HTTP; Fri, 18 Nov 2011 16:36:47 -0800 (PST) In-Reply-To: <201111180814.42656.jhb@freebsd.org> References: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> <201111180814.42656.jhb@freebsd.org> Date: Fri, 18 Nov 2011 16:36:47 -0800 Message-ID: From: Xin LI To: John Baldwin Content-Type: multipart/mixed; boundary=f46d0444e96b14738804b20ba655 Cc: freebsd-stable@freebsd.org, Daryl Sayers Subject: Re: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 00:36:49 -0000 --f46d0444e96b14738804b20ba655 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, > I don't know if it will help with your performance, but I have some patch= es > to allow the NFS server to cluster writes. =C2=A0You can try > www.freebsd.org/~jhb/patches/nfs_server_cluster.patch. =C2=A0I've tested = it on 8, > but it should probably apply fine to 9. I think 9 would need some changes, I just made them with minimal compile testing, though. Cheers, --=20 Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die --f46d0444e96b14738804b20ba655 Content-Type: text/x-patch; charset=US-ASCII; name="nfs-cluster.diff" Content-Disposition: attachment; filename="nfs-cluster.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gv5vzwo30 SW5kZXg6IHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2Rwb3J0LmMKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lz L2ZzL25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYwkocmV2aXNpb24gMjI3Njg5KQorKysgc3lzL2Zz L25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYwkod29ya2luZyBjb3B5KQpAQCAtOTAsMjAgKzkwLDc4 IEBAIFNZU0NUTF9JTlQoX3Zmc19uZnNkLCBPSURfQVVUTywgaXNzdWVfZGVsZWdhdGlvbnMsCiBT WVNDVExfSU5UKF92ZnNfbmZzZCwgT0lEX0FVVE8sIGVuYWJsZV9sb2NhbGxvY2tzLCBDVExGTEFH X1JXLAogICAgICZuZnNydl9kb2xvY2FsbG9ja3MsIDAsICJFbmFibGUgbmZzZCB0byBhY3F1aXJl IGxvY2FsIGxvY2tzIG9uIGZpbGVzIik7CiAKLSNkZWZpbmUJTlVNX0hFVVJJU1RJQwkJMTAxNwor I2RlZmluZQlNQVhfUkVPUkRFUkVEX1JQQwkxNgorI2RlZmluZQlOVU1fSEVVUklTVElDCQkxMDMx CiAjZGVmaW5lCU5IVVNFX0lOSVQJCTY0CiAjZGVmaW5lCU5IVVNFX0lOQwkJMTYKICNkZWZpbmUJ TkhVU0VfTUFYCQkyMDQ4CiAKIHN0YXRpYyBzdHJ1Y3QgbmZzaGV1ciB7CiAJc3RydWN0IHZub2Rl ICpuaF92cDsJLyogdnAgdG8gbWF0Y2ggKHVucmVmZXJlbmNlZCBwb2ludGVyKSAqLwotCW9mZl90 IG5oX25leHRyOwkJLyogbmV4dCBvZmZzZXQgZm9yIHNlcXVlbnRpYWwgZGV0ZWN0aW9uICovCisJ b2ZmX3QgbmhfbmV4dG9mZjsJLyogbmV4dCBvZmZzZXQgZm9yIHNlcXVlbnRpYWwgZGV0ZWN0aW9u ICovCiAJaW50IG5oX3VzZTsJCS8qIHVzZSBjb3VudCBmb3Igc2VsZWN0aW9uICovCiAJaW50IG5o X3NlcWNvdW50OwkvKiBoZXVyaXN0aWMgKi8KIH0gbmZzaGV1cltOVU1fSEVVUklTVElDXTsKIAog CiAvKgorICogSGV1cmlzdGljIHRvIGRldGVjdCBzZXF1ZW50aWFsIG9wZXJhdGlvbi4KKyAqLwor c3RhdGljIHN0cnVjdCBuZnNoZXVyICoKK25mc3J2X3NlcXVlbnRpYWxfaGV1cmlzdGljKHN0cnVj dCB1aW8gKnVpbywgc3RydWN0IHZub2RlICp2cCkKK3sKKwlzdHJ1Y3QgbmZzaGV1ciAqbmg7CisJ aW50IGhpLCB0cnk7CisKKwkvKiBMb2NhdGUgYmVzdCBjYW5kaWRhdGUuICovCisJdHJ5ID0gMzI7 CisJaGkgPSAoKGludCkodm1fb2Zmc2V0X3QpdnAgLyBzaXplb2Yoc3RydWN0IHZub2RlKSkgJSBO VU1fSEVVUklTVElDOworCW5oID0gJm5mc2hldXJbaGldOworCXdoaWxlICh0cnktLSkgeworCQlp ZiAobmZzaGV1cltoaV0ubmhfdnAgPT0gdnApIHsKKwkJCW5oID0gJm5mc2hldXJbaGldOworCQkJ YnJlYWs7CisJCX0KKwkJaWYgKG5mc2hldXJbaGldLm5oX3VzZSA+IDApCisJCQktLW5mc2hldXJb aGldLm5oX3VzZTsKKwkJaGkgPSAoaGkgKyAxKSAlIE5VTV9IRVVSSVNUSUM7CisJCWlmIChuZnNo ZXVyW2hpXS5uaF91c2UgPCBuaC0+bmhfdXNlKQorCQkJbmggPSAmbmZzaGV1cltoaV07CisJfQor CisJLyogSW5pdGlhbGl6ZSBoaW50IGlmIHRoaXMgaXMgYSBuZXcgZmlsZS4gKi8KKwlpZiAobmgt Pm5oX3ZwICE9IHZwKSB7CisJCW5oLT5uaF92cCA9IHZwOworCQluaC0+bmhfbmV4dG9mZiA9IHVp by0+dWlvX29mZnNldDsKKwkJbmgtPm5oX3VzZSA9IE5IVVNFX0lOSVQ7CisJCWlmICh1aW8tPnVp b19vZmZzZXQgPT0gMCkKKwkJCW5oLT5uaF9zZXFjb3VudCA9IDQ7CisJCWVsc2UKKwkJCW5oLT5u aF9zZXFjb3VudCA9IDE7CisJfQorCisJLyogQ2FsY3VsYXRlIGhldXJpc3RpYy4gKi8KKwlpZiAo KHVpby0+dWlvX29mZnNldCA9PSAwICYmIG5oLT5uaF9zZXFjb3VudCA+IDApIHx8CisJICAgIHVp by0+dWlvX29mZnNldCA9PSBuaC0+bmhfbmV4dG9mZikgeworCQkvKiBTZWUgY29tbWVudHMgaW4g dmZzX3Zub3BzLmM6c2VxdWVudGlhbF9oZXVyaXN0aWMoKS4gKi8KKwkJbmgtPm5oX3NlcWNvdW50 ICs9IGhvd21hbnkodWlvLT51aW9fcmVzaWQsIDE2Mzg0KTsKKwkJaWYgKG5oLT5uaF9zZXFjb3Vu dCA+IElPX1NFUU1BWCkKKwkJCW5oLT5uaF9zZXFjb3VudCA9IElPX1NFUU1BWDsKKwl9IGVsc2Ug aWYgKHFhYnModWlvLT51aW9fb2Zmc2V0IC0gbmgtPm5oX25leHRvZmYpIDw9IE1BWF9SRU9SREVS RURfUlBDICoKKwkgICAgaW1heCh2cC0+dl9tb3VudC0+bW50X3N0YXQuZl9pb3NpemUsIHVpby0+ dWlvX3Jlc2lkKSkgeworCQkvKiBQcm9iYWJseSBhIHJlb3JkZXJlZCBSUEMsIGxlYXZlIHNlcWNv dW50IGFsb25lLiAqLworCX0gZWxzZSBpZiAobmgtPm5oX3NlcWNvdW50ID4gMSkgeworCQluaC0+ bmhfc2VxY291bnQgLz0gMjsKKwl9IGVsc2UgeworCQluaC0+bmhfc2VxY291bnQgPSAwOworCX0K KwluaC0+bmhfdXNlICs9IE5IVVNFX0lOQzsKKwlpZiAobmgtPm5oX3VzZSA+IE5IVVNFX01BWCkK KwkJbmgtPm5oX3VzZSA9IE5IVVNFX01BWDsKKwlyZXR1cm4gKG5oKTsKK30KKworLyoKICAqIEdl dCBhdHRyaWJ1dGVzIGludG8gbmZzdmF0dHIgc3RydWN0dXJlLgogICovCiBpbnQKQEAgLTU2Nyw1 OCArNjI1LDEyIEBAIG5mc3Zub19yZWFkKHN0cnVjdCB2bm9kZSAqdnAsIG9mZl90IG9mZiwgaW50 IGNudCwKIAlpbnQgaTsKIAlzdHJ1Y3QgaW92ZWMgKml2OwogCXN0cnVjdCBpb3ZlYyAqaXYyOwot CWludCBlcnJvciA9IDAsIGxlbiwgbGVmdCwgc2l6LCB0bGVuLCBpb2ZsYWcgPSAwLCBoaSwgdHJ5 ID0gMzI7CisJaW50IGVycm9yID0gMCwgbGVuLCBsZWZ0LCBzaXosIHRsZW4sIGlvZmxhZyA9IDA7 CiAJc3RydWN0IG1idWYgKm0yID0gTlVMTCwgKm0zOwogCXN0cnVjdCB1aW8gaW8sICp1aW9wID0g JmlvOwogCXN0cnVjdCBuZnNoZXVyICpuaDsKIAotCS8qCi0JICogQ2FsY3VsYXRlIHNlcWNvdW50 IGZvciBoZXVyaXN0aWMKLQkgKi8KLQkvKgotCSAqIExvY2F0ZSBiZXN0IGNhbmRpZGF0ZQotCSAq LwotCi0JaGkgPSAoKGludCkodm1fb2Zmc2V0X3QpdnAgLyBzaXplb2Yoc3RydWN0IHZub2RlKSkg JSBOVU1fSEVVUklTVElDOwotCW5oID0gJm5mc2hldXJbaGldOwotCi0Jd2hpbGUgKHRyeS0tKSB7 Ci0JCWlmIChuZnNoZXVyW2hpXS5uaF92cCA9PSB2cCkgewotCQkJbmggPSAmbmZzaGV1cltoaV07 Ci0JCQlicmVhazsKLQkJfQotCQlpZiAobmZzaGV1cltoaV0ubmhfdXNlID4gMCkKLQkJCS0tbmZz aGV1cltoaV0ubmhfdXNlOwotCQloaSA9IChoaSArIDEpICUgTlVNX0hFVVJJU1RJQzsKLQkJaWYg KG5mc2hldXJbaGldLm5oX3VzZSA8IG5oLT5uaF91c2UpCi0JCQluaCA9ICZuZnNoZXVyW2hpXTsK LQl9Ci0KLQlpZiAobmgtPm5oX3ZwICE9IHZwKSB7Ci0JCW5oLT5uaF92cCA9IHZwOwotCQluaC0+ bmhfbmV4dHIgPSBvZmY7Ci0JCW5oLT5uaF91c2UgPSBOSFVTRV9JTklUOwotCQlpZiAob2ZmID09 IDApCi0JCQluaC0+bmhfc2VxY291bnQgPSA0OwotCQllbHNlCi0JCQluaC0+bmhfc2VxY291bnQg PSAxOwotCX0KLQotCS8qCi0JICogQ2FsY3VsYXRlIGhldXJpc3RpYwotCSAqLwotCi0JaWYgKChv ZmYgPT0gMCAmJiBuaC0+bmhfc2VxY291bnQgPiAwKSB8fCBvZmYgPT0gbmgtPm5oX25leHRyKSB7 Ci0JCWlmICgrK25oLT5uaF9zZXFjb3VudCA+IElPX1NFUU1BWCkKLQkJCW5oLT5uaF9zZXFjb3Vu dCA9IElPX1NFUU1BWDsKLQl9IGVsc2UgaWYgKG5oLT5uaF9zZXFjb3VudCA+IDEpIHsKLQkJbmgt Pm5oX3NlcWNvdW50ID0gMTsKLQl9IGVsc2UgewotCQluaC0+bmhfc2VxY291bnQgPSAwOwotCX0K LQluaC0+bmhfdXNlICs9IE5IVVNFX0lOQzsKLQlpZiAobmgtPm5oX3VzZSA+IE5IVVNFX01BWCkK LQkJbmgtPm5oX3VzZSA9IE5IVVNFX01BWDsKKwluaCA9IG5mc3J2X3NlcXVlbnRpYWxfaGV1cmlz dGljKHVpb3AsIHZwKTsKIAlpb2ZsYWcgfD0gbmgtPm5oX3NlcWNvdW50IDw8IElPX1NFUVNISUZU OwogCiAJbGVuID0gbGVmdCA9IE5GU01fUk5EVVAoY250KTsKQEAgLTY3Miw2ICs2ODQsNyBAQCBu ZnN2bm9fcmVhZChzdHJ1Y3Qgdm5vZGUgKnZwLCBvZmZfdCBvZmYsIGludCBjbnQsCiAJCSptcHAg PSBOVUxMOwogCQlnb3RvIG91dDsKIAl9CisJbmgtPm5oX25leHRvZmYgPSB1aW9wLT51aW9fb2Zm c2V0OwogCXRsZW4gPSBsZW4gLSB1aW9wLT51aW9fcmVzaWQ7CiAJY250ID0gY250IDwgdGxlbiA/ IGNudCA6IHRsZW47CiAJdGxlbiA9IE5GU01fUk5EVVAoY250KTsKQEAgLTcwMCw2ICs3MTMsNyBA QCBuZnN2bm9fd3JpdGUoc3RydWN0IHZub2RlICp2cCwgb2ZmX3Qgb2ZmLCBpbnQgcmV0bAogCXN0 cnVjdCBpb3ZlYyAqaXY7CiAJaW50IGlvZmxhZ3MsIGVycm9yOwogCXN0cnVjdCB1aW8gaW8sICp1 aW9wID0gJmlvOworCXN0cnVjdCBuZnNoZXVyICpuaDsKIAogCU1BTExPQyhpdnAsIHN0cnVjdCBp b3ZlYyAqLCBjbnQgKiBzaXplb2YgKHN0cnVjdCBpb3ZlYyksIE1fVEVNUCwKIAkgICAgTV9XQUlU T0spOwpAQCAtNzMzLDcgKzc0NywxMSBAQCBuZnN2bm9fd3JpdGUoc3RydWN0IHZub2RlICp2cCwg b2ZmX3Qgb2ZmLCBpbnQgcmV0bAogCXVpb3AtPnVpb19zZWdmbGcgPSBVSU9fU1lTU1BBQ0U7CiAJ TkZTVUlPUFJPQyh1aW9wLCBwKTsKIAl1aW9wLT51aW9fb2Zmc2V0ID0gb2ZmOworCW5oID0gbmZz cnZfc2VxdWVudGlhbF9oZXVyaXN0aWModWlvcCwgdnApOworCWlvZmxhZ3MgfD0gbmgtPm5oX3Nl cWNvdW50IDw8IElPX1NFUVNISUZUOwogCWVycm9yID0gVk9QX1dSSVRFKHZwLCB1aW9wLCBpb2Zs YWdzLCBjcmVkKTsKKwlpZiAoZXJyb3IgPT0gMCkKKwkJbmgtPm5oX25leHRvZmYgPSB1aW9wLT51 aW9fb2Zmc2V0OwogCUZSRUUoKGNhZGRyX3QpaXYsIE1fVEVNUCk7CiAKIAlORlNFWElUQ09ERShl cnJvcik7CkluZGV4OiBzeXMvbmZzc2VydmVyL25mc19zZXJ2LmMKPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gc3lz L25mc3NlcnZlci9uZnNfc2Vydi5jCShyZXZpc2lvbiAyMjc2ODkpCisrKyBzeXMvbmZzc2VydmVy L25mc19zZXJ2LmMJKHdvcmtpbmcgY29weSkKQEAgLTEwNywxNCArMTA3LDE1IEBAIEZFQVRVUkUo bmZzc2VydmVyLCAiTkZTIHNlcnZlciIpOwogCiAjZGVmaW5lIE1BWF9DT01NSVRfQ09VTlQJKDEw MjQgKiAxMDI0KQogCi0jZGVmaW5lIE5VTV9IRVVSSVNUSUMJCTEwMTcKKyNkZWZpbmUJTUFYX1JF T1JERVJFRF9SUEMJMTYKKyNkZWZpbmUgTlVNX0hFVVJJU1RJQwkJMTAzMQogI2RlZmluZSBOSFVT RV9JTklUCQk2NAogI2RlZmluZSBOSFVTRV9JTkMJCTE2CiAjZGVmaW5lIE5IVVNFX01BWAkJMjA0 OAogCiBzdGF0aWMgc3RydWN0IG5mc2hldXIgewogCXN0cnVjdCB2bm9kZSAqbmhfdnA7CS8qIHZw IHRvIG1hdGNoICh1bnJlZmVyZW5jZWQgcG9pbnRlcikgKi8KLQlvZmZfdCBuaF9uZXh0cjsJCS8q IG5leHQgb2Zmc2V0IGZvciBzZXF1ZW50aWFsIGRldGVjdGlvbiAqLworCW9mZl90IG5oX25leHRv ZmY7CS8qIG5leHQgb2Zmc2V0IGZvciBzZXF1ZW50aWFsIGRldGVjdGlvbiAqLwogCWludCBuaF91 c2U7CQkvKiB1c2UgY291bnQgZm9yIHNlbGVjdGlvbiAqLwogCWludCBuaF9zZXFjb3VudDsJLyog aGV1cmlzdGljICovCiB9IG5mc2hldXJbTlVNX0hFVVJJU1RJQ107CkBAIC0xODcsNiArMTg4LDYz IEBAIG5mc3J2X2xvY2tlZHBhaXJfbmQoaW50IHZmczEsIHN0cnVjdCBuYW1laWRhdGEgKm5kCiB9 CiAKIC8qCisgKiBIZXVyaXN0aWMgdG8gZGV0ZWN0IHNlcXVlbnRpYWwgb3BlcmF0aW9uLgorICov CitzdGF0aWMgc3RydWN0IG5mc2hldXIgKgorbmZzcnZfc2VxdWVudGlhbF9oZXVyaXN0aWMoc3Ry dWN0IHVpbyAqdWlvLCBzdHJ1Y3Qgdm5vZGUgKnZwKQoreworCXN0cnVjdCBuZnNoZXVyICpuaDsK KwlpbnQgaGksIHRyeTsKKworCS8qIExvY2F0ZSBiZXN0IGNhbmRpZGF0ZS4gKi8KKwl0cnkgPSAz MjsKKwloaSA9ICgoaW50KSh2bV9vZmZzZXRfdCl2cCAvIHNpemVvZihzdHJ1Y3Qgdm5vZGUpKSAl IE5VTV9IRVVSSVNUSUM7CisJbmggPSAmbmZzaGV1cltoaV07CisJd2hpbGUgKHRyeS0tKSB7CisJ CWlmIChuZnNoZXVyW2hpXS5uaF92cCA9PSB2cCkgeworCQkJbmggPSAmbmZzaGV1cltoaV07CisJ CQlicmVhazsKKwkJfQorCQlpZiAobmZzaGV1cltoaV0ubmhfdXNlID4gMCkKKwkJCS0tbmZzaGV1 cltoaV0ubmhfdXNlOworCQloaSA9IChoaSArIDEpICUgTlVNX0hFVVJJU1RJQzsKKwkJaWYgKG5m c2hldXJbaGldLm5oX3VzZSA8IG5oLT5uaF91c2UpCisJCQluaCA9ICZuZnNoZXVyW2hpXTsKKwl9 CisKKwkvKiBJbml0aWFsaXplIGhpbnQgaWYgdGhpcyBpcyBhIG5ldyBmaWxlLiAqLworCWlmIChu aC0+bmhfdnAgIT0gdnApIHsKKwkJbmgtPm5oX3ZwID0gdnA7CisJCW5oLT5uaF9uZXh0b2ZmID0g dWlvLT51aW9fb2Zmc2V0OworCQluaC0+bmhfdXNlID0gTkhVU0VfSU5JVDsKKwkJaWYgKHVpby0+ dWlvX29mZnNldCA9PSAwKQorCQkJbmgtPm5oX3NlcWNvdW50ID0gNDsKKwkJZWxzZQorCQkJbmgt Pm5oX3NlcWNvdW50ID0gMTsKKwl9CisKKwkvKiBDYWxjdWxhdGUgaGV1cmlzdGljLiAqLworCWlm ICgodWlvLT51aW9fb2Zmc2V0ID09IDAgJiYgbmgtPm5oX3NlcWNvdW50ID4gMCkgfHwKKwkgICAg dWlvLT51aW9fb2Zmc2V0ID09IG5oLT5uaF9uZXh0b2ZmKSB7CisJCS8qIFNlZSBjb21tZW50cyBp biB2ZnNfdm5vcHMuYzpzZXF1ZW50aWFsX2hldXJpc3RpYygpLiAqLworCQluaC0+bmhfc2VxY291 bnQgKz0gaG93bWFueSh1aW8tPnVpb19yZXNpZCwgMTYzODQpOworCQlpZiAobmgtPm5oX3NlcWNv dW50ID4gSU9fU0VRTUFYKQorCQkJbmgtPm5oX3NlcWNvdW50ID0gSU9fU0VRTUFYOworCX0gZWxz ZSBpZiAocWFicyh1aW8tPnVpb19vZmZzZXQgLSBuaC0+bmhfbmV4dG9mZikgPD0gTUFYX1JFT1JE RVJFRF9SUEMgKgorCSAgICBpbWF4KHZwLT52X21vdW50LT5tbnRfc3RhdC5mX2lvc2l6ZSwgdWlv LT51aW9fcmVzaWQpKSB7CisJCS8qIFByb2JhYmx5IGEgcmVvcmRlcmVkIFJQQywgbGVhdmUgc2Vx Y291bnQgYWxvbmUuICovCisJfSBlbHNlIGlmIChuaC0+bmhfc2VxY291bnQgPiAxKSB7CisJCW5o LT5uaF9zZXFjb3VudCAvPSAyOworCX0gZWxzZSB7CisJCW5oLT5uaF9zZXFjb3VudCA9IDA7CisJ fQorCW5oLT5uaF91c2UgKz0gTkhVU0VfSU5DOworCWlmIChuaC0+bmhfdXNlID4gTkhVU0VfTUFY KQorCQluaC0+bmhfdXNlID0gTkhVU0VfTUFYOworCXJldHVybiAobmgpOworfQorCisvKgogICog bmZzIHYzIGFjY2VzcyBzZXJ2aWNlCiAgKi8KIGludApAQCAtODQzLDcgKzkwMSw2IEBAIG5mc3J2 X3JlYWQoc3RydWN0IG5mc3J2X2Rlc2NyaXB0ICpuZnNkLCBzdHJ1Y3QgbmZzCiAJLyoKIAkgKiBD YWxjdWxhdGUgYnl0ZSBjb3VudCB0byByZWFkCiAJICovCi0KIAlpZiAob2ZmID49IHZhcC0+dmFf c2l6ZSkKIAkJY250ID0gMDsKIAllbHNlIGlmICgob2ZmICsgcmVxbGVuKSA+IHZhcC0+dmFfc2l6 ZSkKQEAgLTg1MSw2MSArOTA4LDYgQEAgbmZzcnZfcmVhZChzdHJ1Y3QgbmZzcnZfZGVzY3JpcHQg Km5mc2QsIHN0cnVjdCBuZnMKIAllbHNlCiAJCWNudCA9IHJlcWxlbjsKIAotCS8qCi0JICogQ2Fs Y3VsYXRlIHNlcWNvdW50IGZvciBoZXVyaXN0aWMKLQkgKi8KLQotCXsKLQkJaW50IGhpOwotCQlp bnQgdHJ5ID0gMzI7Ci0KLQkJLyoKLQkJICogTG9jYXRlIGJlc3QgY2FuZGlkYXRlCi0JCSAqLwot Ci0JCWhpID0gKChpbnQpKHZtX29mZnNldF90KXZwIC8gc2l6ZW9mKHN0cnVjdCB2bm9kZSkpICUg TlVNX0hFVVJJU1RJQzsKLQkJbmggPSAmbmZzaGV1cltoaV07Ci0KLQkJd2hpbGUgKHRyeS0tKSB7 Ci0JCQlpZiAobmZzaGV1cltoaV0ubmhfdnAgPT0gdnApIHsKLQkJCQluaCA9ICZuZnNoZXVyW2hp XTsKLQkJCQlicmVhazsKLQkJCX0KLQkJCWlmIChuZnNoZXVyW2hpXS5uaF91c2UgPiAwKQotCQkJ CS0tbmZzaGV1cltoaV0ubmhfdXNlOwotCQkJaGkgPSAoaGkgKyAxKSAlIE5VTV9IRVVSSVNUSUM7 Ci0JCQlpZiAobmZzaGV1cltoaV0ubmhfdXNlIDwgbmgtPm5oX3VzZSkKLQkJCQluaCA9ICZuZnNo ZXVyW2hpXTsKLQkJfQotCi0JCWlmIChuaC0+bmhfdnAgIT0gdnApIHsKLQkJCW5oLT5uaF92cCA9 IHZwOwotCQkJbmgtPm5oX25leHRyID0gb2ZmOwotCQkJbmgtPm5oX3VzZSA9IE5IVVNFX0lOSVQ7 Ci0JCQlpZiAob2ZmID09IDApCi0JCQkJbmgtPm5oX3NlcWNvdW50ID0gNDsKLQkJCWVsc2UKLQkJ CQluaC0+bmhfc2VxY291bnQgPSAxOwotCQl9Ci0KLQkJLyoKLQkJICogQ2FsY3VsYXRlIGhldXJp c3RpYwotCQkgKi8KLQotCQlpZiAoKG9mZiA9PSAwICYmIG5oLT5uaF9zZXFjb3VudCA+IDApIHx8 IG9mZiA9PSBuaC0+bmhfbmV4dHIpIHsKLQkJCWlmICgrK25oLT5uaF9zZXFjb3VudCA+IElPX1NF UU1BWCkKLQkJCQluaC0+bmhfc2VxY291bnQgPSBJT19TRVFNQVg7Ci0JCX0gZWxzZSBpZiAobmgt Pm5oX3NlcWNvdW50ID4gMSkgewotCQkJbmgtPm5oX3NlcWNvdW50ID0gMTsKLQkJfSBlbHNlIHsK LQkJCW5oLT5uaF9zZXFjb3VudCA9IDA7Ci0JCX0KLQkJbmgtPm5oX3VzZSArPSBOSFVTRV9JTkM7 Ci0JCWlmIChuaC0+bmhfdXNlID4gTkhVU0VfTUFYKQotCQkJbmgtPm5oX3VzZSA9IE5IVVNFX01B WDsKLQkJaW9mbGFnIHw9IG5oLT5uaF9zZXFjb3VudCA8PCBJT19TRVFTSElGVDsKLSAgICAgICAg fQotCiAJbmZzbV9yZXBseShORlNYX1BPU1RPUE9SRkFUVFIodjMpICsgMyAqIE5GU1hfVU5TSUdO RUQrbmZzbV9ybmR1cChjbnQpKTsKIAlpZiAodjMpIHsKIAkJdGwgPSBuZnNtX2J1aWxkKHVfaW50 MzJfdCAqLCBORlNYX1YzRkFUVFIgKyA0ICogTkZTWF9VTlNJR05FRCk7CkBAIC05NjMsOSArOTY1 LDExIEBAIG5mc3J2X3JlYWQoc3RydWN0IG5mc3J2X2Rlc2NyaXB0ICpuZnNkLCBzdHJ1Y3QgbmZz CiAJCXVpb3AtPnVpb19yZXNpZCA9IGxlbjsKIAkJdWlvcC0+dWlvX3J3ID0gVUlPX1JFQUQ7CiAJ CXVpb3AtPnVpb19zZWdmbGcgPSBVSU9fU1lTU1BBQ0U7CisJCW5oID0gbmZzcnZfc2VxdWVudGlh bF9oZXVyaXN0aWModWlvcCwgdnApOworCQlpb2ZsYWcgfD0gbmgtPm5oX3NlcWNvdW50IDw8IElP X1NFUVNISUZUOwogCQllcnJvciA9IFZPUF9SRUFEKHZwLCB1aW9wLCBJT19OT0RFTE9DS0VEIHwg aW9mbGFnLCBjcmVkKTsKLQkJb2ZmID0gdWlvcC0+dWlvX29mZnNldDsKLQkJbmgtPm5oX25leHRy ID0gb2ZmOworCQlpZiAoZXJyb3IgPT0gMCkKKwkJCW5oLT5uaF9uZXh0b2ZmID0gdWlvcC0+dWlv X29mZnNldDsKIAkJZnJlZSgoY2FkZHJfdClpdjIsIE1fVEVNUCk7CiAJCWlmIChlcnJvciB8fCAo Z2V0cmV0ID0gVk9QX0dFVEFUVFIodnAsIHZhcCwgY3JlZCkpKSB7CiAJCQlpZiAoIWVycm9yKQpA QCAtMTAzMCw2ICsxMDM0LDcgQEAgbmZzcnZfd3JpdGUoc3RydWN0IG5mc3J2X2Rlc2NyaXB0ICpu ZnNkLCBzdHJ1Y3QgbmYKIAlpbnQgdjMgPSAobmZzZC0+bmRfZmxhZyAmIE5EX05GU1YzKTsKIAlz dHJ1Y3QgbWJ1ZiAqbWIsICptcmVxOwogCXN0cnVjdCB2bm9kZSAqdnAgPSBOVUxMOworCXN0cnVj dCBuZnNoZXVyICpuaDsKIAluZnNmaF90IG5maDsKIAlmaGFuZGxlX3QgKmZocDsKIAlzdHJ1Y3Qg dWlvIGlvLCAqdWlvcCA9ICZpbzsKQEAgLTExNzAsNyArMTE3NSwxMSBAQCBuZnNydl93cml0ZShz dHJ1Y3QgbmZzcnZfZGVzY3JpcHQgKm5mc2QsIHN0cnVjdCBuZgogCSAgICB1aW9wLT51aW9fc2Vn ZmxnID0gVUlPX1NZU1NQQUNFOwogCSAgICB1aW9wLT51aW9fdGQgPSBOVUxMOwogCSAgICB1aW9w LT51aW9fb2Zmc2V0ID0gb2ZmOworCSAgICBuaCA9IG5mc3J2X3NlcXVlbnRpYWxfaGV1cmlzdGlj KHVpb3AsIHZwKTsKKwkgICAgaW9mbGFncyB8PSBuaC0+bmhfc2VxY291bnQgPDwgSU9fU0VRU0hJ RlQ7CiAJICAgIGVycm9yID0gVk9QX1dSSVRFKHZwLCB1aW9wLCBpb2ZsYWdzLCBjcmVkKTsKKwkg ICAgaWYgKGVycm9yID09IDApCisJCSAgICBuaC0+bmhfbmV4dG9mZiA9IHVpb3AtPnVpb19vZmZz ZXQ7CiAJICAgIC8qIFVubG9ja2VkIHdyaXRlLiAqLwogCSAgICBuZnNydnN0YXRzLnNydnZvcF93 cml0ZXMrKzsKIAkgICAgZnJlZSgoY2FkZHJfdClpdiwgTV9URU1QKTsK --f46d0444e96b14738804b20ba655-- From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 00:29:02 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 796A4106566C for ; Sat, 19 Nov 2011 00:29:02 +0000 (UTC) (envelope-from darren780@yahoo.com) Received: from nm22.bullet.mail.sp2.yahoo.com (nm22.bullet.mail.sp2.yahoo.com [98.139.91.92]) by mx1.freebsd.org (Postfix) with SMTP id 49A3D8FC0C for ; Sat, 19 Nov 2011 00:29:02 +0000 (UTC) Received: from [98.139.91.67] by nm22.bullet.mail.sp2.yahoo.com with NNFMP; 19 Nov 2011 00:16:08 -0000 Received: from [98.139.91.7] by tm7.bullet.mail.sp2.yahoo.com with NNFMP; 19 Nov 2011 00:16:08 -0000 Received: from [127.0.0.1] by omp1007.mail.sp2.yahoo.com with NNFMP; 19 Nov 2011 00:16:07 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 991212.5173.bm@omp1007.mail.sp2.yahoo.com Received: (qmail 7209 invoked by uid 60001); 19 Nov 2011 00:16:07 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1321661767; bh=VqA7e0iKQ8UtmZM5ZP1J0JAPeoYl08j95sMvH9jDai4=; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=HRrsFmwWyXM4ltm+dOvpxtH1TezF3ghuPXy/setJSH8b/mOPj74BYqEcHCtpB9tQGxXVvQbZYR4ObPt3SUIlrzexI2Rkf2L8VB7YI8pZ1OBURu2k4LGJMJBDEzQ0ZwwDSgHSBoyN5JNwHdaeezd/RjMNC3rLH1YVLYGjnk41bLU= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1kjw5gH2GV0dYy4lv/2aCdDhahv30zLaiLO0FD+wHgoTjlh070nnJwUFN7F1Z7qIwrx6YhjY9O07K8GLEAbvskMndNbzk2Qfz6/0X/DbetABZjRhoUArToQViI/VKdeXeKL++uTg3uuZMyMRry25XiOj9zTuXOgKuj2cHmtNSlc=; X-YMail-OSG: pyXxxdUVM1mqHTa9sz3kJYINSWuk0Ru4vpMWzpxHqLjvGyb gcsVCuHitPP9AcfhjmRx7C2Q5859CBa8e0wImNLRtLfCbnXb5.aMcW.Q9Wpg I9yD6RWGr4MrVpwrb5i02oJC75FQi4NVN1PAPzOdfmkhfZqQKsTFTzntLAbq 90525HQGcej01Qfe9.STrh8yX6R6hjLN7hdBmBRZIn5QkAZBXCBJN3CS95jG zf3PSqqScuUOZgsg2XHSZmI4gHPpwcj.RsDuKFtofOCV2JTpFQzgqOk4tNhf vkAKXJcsQPbtUQ7c2uhGJZhDfYAi9o7l7_IbERwGPg4dJ9qKNUMGbO3MIS9c 9UdtAsv5Ohrb9dWy80ptmBnQiFtuLekMpuEgiKMBLexN2qBs6eBxBV5foWtE YrgCSI5Ikhhl2KC.M.JcS6hvQ7cu8JuMKt5zZPDGfizJqcAMI5.jlN_zDCHd k2kaq2BtkNcJqrTS0ianBIp38f1M.mVHmuz4lFjVuy26DwunxlEsOYh5nRxc _cdWNVkDgpE5gt903TS1w_DXzCLeOZ9BZXesehsPq3KrCXMgnjCtPdah1qwT Ug435WkuiOguk7ecmrNJnvc123WWlS0U9ewqW5PuPd1Kq Received: from [70.40.190.165] by web112010.mail.gq1.yahoo.com via HTTP; Fri, 18 Nov 2011 16:16:07 PST X-Mailer: YahooMailWebService/0.8.115.325013 References: <1311586834.9079.YahooMailClassic@web112004.mail.gq1.yahoo.com> <1311643303.55807.YahooMailRC@web112011.mail.gq1.yahoo.com> Message-ID: <1321661767.89155.YahooMailNeo@web112010.mail.gq1.yahoo.com> Date: Fri, 18 Nov 2011 16:16:07 -0800 (PST) From: "Mr. Darren" To: freebsd In-Reply-To: MIME-Version: 1.0 X-Mailman-Approved-At: Sat, 19 Nov 2011 02:20:35 +0000 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Fw: if_run X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "Mr. Darren" List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 00:29:02 -0000 Been using this with stable for a few months.=A0 thought I would share.=A0 = works like a dream.=0A=0A-Darren=0A=0A----- Forwarded Message -----=0AFrom:= PseudoCylon =0ATo: Mr. Darren =0ASent: Tuesday, July 26, 2011 4:17:32 AM=0ASubject: Re: if_run=0A =0AI= t seems it uses different IDs.=0A=0Aadd a following line somewhere in /usr/= src/sys/dev/usb/usbdevs (The=0Aorder doesn't matter as long as it's bellow = bunch of entries begin=0Awith "vendor".)=0A=0Aproduct=A0=A0=A0 CISCOLINKSYS= =A0=A0=A0 AE1000=A0=A0=A0 0x002f=A0=A0=A0 Cisco AE1000=0A=0A=0Aand insert f= ollowing to line 150 of /usr/src/sys/dev/usb/wlan/if_run.c=0A(The order doe= sn't matter as long as it is in struct usb_device_id=0Arun_devs[].)=0A=0A= =A0=A0=A0 RUN_DEV(CISCOLINKSYS,=A0=A0=A0 AE1000),=0A=0A=0ARecompile and loa= d the driver. If you have compiled the driver into=0Athe kernel (using GENE= RIC kernel), you need to recompile whole kernel.=0ABut, with KERNFAST optio= n,=0A# make buildkernel KERNCONF=3DGENERIC KERNFAST=3D1=0Ait would take onl= y a few minutes. Then, reboot.=0A=0ALet me know if it works. I will submit = the new IDs.=0A=0A=0AAK=0A=0AOn Mon, Jul 25, 2011 at 7:21 PM, Mr. Darren wrote:=0A> I thought so.=A0 However this seems to be th= e only mention in dmesg=0A> ugen1.3: at usbus1=0A> FreeBSD .ed.sh= awcable.net 8.2-STABLE FreeBSD 8.2-STABLE #1: Mon Jul 25=0A> 01:36:49 UTC 2= 011=A0=A0=A0=A0 root@.ed.shawcable.net:/usr/obj/usr/src/sys/GENERIC=0A> amd= 64=0A> on reconnect of usb=0A> Jul 25 19:20:49=A0 root: Unknown USB device:= vendor 0x13b1 product 0x002f bus=0A> uhub1=0A> Jul 25 19:20:49=A0 kernel: = ugen1.3: at usbus1=0A>=0A>=0A> Darren Johnston=0A>=0A> __________= ______________________=0A> From: PseudoCylon =0A> = To: Mr. Darren =0A> Sent: Mon, July 25, 2011 11:09:33 = AM=0A> Subject: Re: if_run=0A>=0A> On Mon, Jul 25, 2011 at 3:40 AM, Mr. Dar= ren wrote:=0A>>=0A>> I noticed your work on the rt357= 2 if_run and was wondering if it can apply=0A>> to cisco linksys ae1000?=0A= >>=0A>=0A> It seems ae1000 also uses rt3572.=0A> http://www.wikidevi.com/wi= ki/Linksys_AE1000=0A> If that true, if_run should work on ae1000 as well.= =0A>=0A>=0A> AK=0A> From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 09:05:34 2011 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 898FA106566B for ; Sat, 19 Nov 2011 09:05:34 +0000 (UTC) (envelope-from matheus@eternamente.info) Received: from phoenix.eternamente.info (phoenix.eternamente.info [109.169.62.232]) by mx1.freebsd.org (Postfix) with ESMTP id 6675B8FC08 for ; Sat, 19 Nov 2011 09:05:34 +0000 (UTC) Received: by phoenix.eternamente.info (Postfix, from userid 80) id BF00F1CC6E; Sat, 19 Nov 2011 02:20:44 -0300 (BRT) Received: from 187.115.177.208 (SquirrelMail authenticated user matheus) by eternamente.info with HTTP; Sat, 19 Nov 2011 03:20:44 -0200 Message-ID: <1c312a399e23d546f05f5e8396763323.squirrel@eternamente.info> Date: Sat, 19 Nov 2011 03:20:44 -0200 From: "Nenhum_de_Nos" To: stable@freebsd.org User-Agent: SquirrelMail/1.4.21 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: Subject: mount GPT from Windows 7 in FreeBSD 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 09:05:34 -0000 hail, I have two disks, and the one holding Windows appears just as ada1 or ad8, despite fdisk shows all partitions fine. I tried to kldload geom_gpt_something, but it says it was already loaded. I couldn't mount using mount_ntfs, so I would use fuse to have ability to write on it :) is there a way to solve this ? two disks. One just FreeBSD, the other just Windows. thanks, matheus -- We will call you cygnus, The God of balance you shall be A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? http://en.wikipedia.org/wiki/Posting_style -- Pregando o ódio a Oi desde 2007 We will call you cygnus, The God of balance you shall be A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? http://en.wikipedia.org/wiki/Posting_style From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 09:30:36 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BE5D106564A for ; Sat, 19 Nov 2011 09:30:36 +0000 (UTC) (envelope-from matheus@eternamente.info) Received: from phoenix.eternamente.info (phoenix.eternamente.info [109.169.62.232]) by mx1.freebsd.org (Postfix) with ESMTP id 258D78FC08 for ; Sat, 19 Nov 2011 09:30:35 +0000 (UTC) Received: by phoenix.eternamente.info (Postfix, from userid 80) id 794101CC6C; Fri, 18 Nov 2011 21:55:10 -0300 (BRT) Received: from 187.115.177.208 (SquirrelMail authenticated user matheus) by eternamente.info with HTTP; Fri, 18 Nov 2011 22:55:10 -0200 Message-ID: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> Date: Fri, 18 Nov 2011 22:55:10 -0200 From: "Nenhum_de_Nos" To: freebsd-stable@freebsd.org User-Agent: SquirrelMail/1.4.21 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: mount GPT from Windows 7 in FreeBSD 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 09:30:36 -0000 hail, I have two disks, and the one holding Windows appears just as ada1 or ad8, despite fdisk shows all partitions fine. I tried to kldload geom_gpt_something, but it says it was already loaded. I couldn't mount using mount_ntfs, so I would use fuse to have ability to write on it :) is there a way to solve this ? two disks. One just FreeBSD, the other just Windows. thanks, matheus -- We will call you cygnus, The God of balance you shall be A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? http://en.wikipedia.org/wiki/Posting_style From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 09:58:11 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B1491065672 for ; Sat, 19 Nov 2011 09:58:11 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 489D68FC12 for ; Sat, 19 Nov 2011 09:58:10 +0000 (UTC) Received: from omta11.westchester.pa.mail.comcast.net ([76.96.62.36]) by qmta07.westchester.pa.mail.comcast.net with comcast id yxxu1h00B0mv7h057xyA63; Sat, 19 Nov 2011 09:58:10 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta11.westchester.pa.mail.comcast.net with comcast id yxy91h00T1t3BNj3XxyAsF; Sat, 19 Nov 2011 09:58:10 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 65586102C1D; Sat, 19 Nov 2011 01:58:08 -0800 (PST) Date: Sat, 19 Nov 2011 01:58:08 -0800 From: Jeremy Chadwick To: Nenhum_de_Nos Message-ID: <20111119095808.GA87444@icarus.home.lan> References: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: mount GPT from Windows 7 in FreeBSD 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 09:58:11 -0000 On Fri, Nov 18, 2011 at 10:55:10PM -0200, Nenhum_de_Nos wrote: > hail, > > I have two disks, and the one holding Windows appears just as ada1 or ad8, despite fdisk shows all > partitions fine. > > I tried to kldload geom_gpt_something, but it says it was already loaded. I couldn't mount using > mount_ntfs, so I would use fuse to have ability to write on it :) > > is there a way to solve this ? > two disks. One just FreeBSD, the other just Windows. First: to my knowledge, fdisk does not support GPT, so I'm not sure what you mean when you say "it shows all partitions fine". You should probably use gpart(8) instead, e.g. "gpart show" and/or "gpart list". See the man page for usage. It will work with all types, and tell you what scheme is used (MBR, GPT, etc.). Next item: what do you mean it appears as "ada1 **or** ad8"? ada1 indicates you're using ahci.ko (not ataahci.ko) which supports AHCI via CAM(4), while the latter indicates you're using ata(4) (even if AHCI is in use; that would be ataahci.ko). Why/how would this change unless you are messing with cables or enabling/disabling drivers? Next item: I think you're referring to the geom_part_gpt.ko module, but you don't need to do that. GEOM and related kernel bits will load it automatically, which is why it told you it's already loaded. You can use "kldstat" to verify. Otherwise it's statically-included in your kernel. Next item: this sounds like the crux of your issue. As I understand it, NTFS support via kernel on FreeBSD is in an extremely bad state on numerous levels. You can find complaints about lack-of or badly-done UTF-8 filename/path support, lack of full write support, and the more important/major Non-MPSAFE filesystem declaration here: http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS It looks like attilio@freebsd.org has taken ownership of the NTFS driver in the kernel at this point. You may want to ask him if there are any patches you could try. However, as I understand it, the fuse-based NTFS is more reliable and has much higher compatibility. The trade-offs are added complexity getting it all to work, and slower speed. Final item: you sent this mail twice, once to stable@freebsd.org, once to freebsd-stable@freebsd.org. The mail ID headers appear to indicate they were separate/unique and the mail content themselves slightly differs (extra newlines, etc.). No need for this; they are the same list. 3909 11/18 22:55 Nenhum_de_Nos (0.8K) mount GPT from Windows 7 in FreeBSD 9 Message-ID: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> 3910 11/19 03:20 Nenhum_de_Nos (1.1K) mount GPT from Windows 7 in FreeBSD 9 Message-ID: <1c312a399e23d546f05f5e8396763323.squirrel@eternamente.info> -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 13:51:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEEB61065673 for ; Sat, 19 Nov 2011 13:51:26 +0000 (UTC) (envelope-from matheus@eternamente.info) Received: from phoenix.eternamente.info (phoenix.eternamente.info [109.169.62.232]) by mx1.freebsd.org (Postfix) with ESMTP id 838578FC14 for ; Sat, 19 Nov 2011 13:51:26 +0000 (UTC) Received: by phoenix.eternamente.info (Postfix, from userid 80) id 034BA1CC6C; Sat, 19 Nov 2011 10:51:19 -0300 (BRT) Received: from 187.115.177.208 (SquirrelMail authenticated user matheus) by eternamente.info with HTTP; Sat, 19 Nov 2011 11:51:19 -0200 Message-ID: <5f508e72e9cfcf7ebfe8ec53a68851e9.squirrel@eternamente.info> In-Reply-To: <20111119095808.GA87444@icarus.home.lan> References: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> <20111119095808.GA87444@icarus.home.lan> Date: Sat, 19 Nov 2011 11:51:19 -0200 From: "Nenhum_de_Nos" To: freebsd-stable@freebsd.org User-Agent: SquirrelMail/1.4.21 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: Re: mount GPT from Windows 7 in FreeBSD 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 13:51:26 -0000 On Sat, November 19, 2011 07:58, Jeremy Chadwick wrote: > On Fri, Nov 18, 2011 at 10:55:10PM -0200, Nenhum_de_Nos wrote: >> hail, >> >> I have two disks, and the one holding Windows appears just as ada1 or ad8, despite fdisk shows >> all >> partitions fine. >> >> I tried to kldload geom_gpt_something, but it says it was already loaded. I couldn't mount using >> mount_ntfs, so I would use fuse to have ability to write on it :) >> >> is there a way to solve this ? >> two disks. One just FreeBSD, the other just Windows. Jeremy, thanks and let's to the answers. > First: to my knowledge, fdisk does not support GPT, so I'm not sure what > you mean when you say "it shows all partitions fine". You should > probably use gpart(8) instead, e.g. "gpart show" and/or "gpart list". > See the man page for usage. It will work with all types, and tell you > what scheme is used (MBR, GPT, etc.). its all here :) macgyver# fdisk ada1 ******* Working on device /dev/ada1 ******* parameters extracted from in-core disklabel are: cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 7 (0x07),(NTFS, OS/2 HPFS, QNX-2 (16 bit) or Advanced UNIX) start 2048, size 204800 (100 Meg), flag 80 (active) beg: cyl 0/ head 107/ sector 16; end: cyl 48/ head 134/ sector 14 The data for partition 2 is: sysid 7 (0x07),(NTFS, OS/2 HPFS, QNX-2 (16 bit) or Advanced UNIX) start 206848, size 103165952 (50374 Meg), flag 0 beg: cyl 48/ head 134/ sector 15; end: cyl 1023/ head 223/ sector 19 The data for partition 3 is: sysid 7 (0x07),(NTFS, OS/2 HPFS, QNX-2 (16 bit) or Advanced UNIX) start 103372800, size 385019904 (187998 Meg), flag 0 beg: cyl 1023/ head 239/ sector 63; end: cyl 1023/ head 239/ sector 63 The data for partition 4 is: macgyver# gpart show ada1 => 34 488397101 ada1 GPT (232G) 34 488397101 - free - (232G) macgyver# gpart list ada1 Geom name: ada1 modified: false state: OK fwheads: 16 fwsectors: 63 last: 488397134 first: 34 entries: 128 scheme: GPT Consumers: 1. Name: ada1 Mediasize: 250059350016 (232G) Sectorsize: 512 Mode: r0w0e0 macgyver# dmesg | grep ada1 ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 ada1: ATA-7 SATA 1.x device ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad8 macgyver# > Next item: what do you mean it appears as "ada1 **or** ad8"? ada1 > indicates you're using ahci.ko (not ataahci.ko) which supports AHCI via > CAM(4), while the latter indicates you're using ata(4) (even if AHCI is > in use; that would be ataahci.ko). Why/how would this change unless you > are messing with cables or enabling/disabling drivers? As now I'm on FreeBSD in this machine, there is: macgyver# ls /dev/ada1* /dev/ada1 but the FreeBSD disk: macgyver# ls /dev/ada0* /dev/ada0 /dev/ada0s1 /dev/ada0s1a /dev/ada0s1b /dev/ada0s2 /dev/ada0s2a /dev/ada0s3 /dev/ada0s4 /dev/ada0s4a /dev/ada0s4b and I still have old ad4 and ad8: macgyver# ls /dev/ad4* /dev/ad8* /dev/ad4 /dev/ad4s1 /dev/ad4s1a /dev/ad4s1b /dev/ad4s2 /dev/ad4s2a /dev/ad4s3 /dev/ad4s4 /dev/ad4s4a /dev/ad4s4b /dev/ad8 and my point here is there even though fdisk shows some windows partitions, I can't even address them as I just see ada1/ad8. no ada1s1 or anything alike. > Next item: I think you're referring to the geom_part_gpt.ko module, > but you don't need to do that. GEOM and related kernel bits will load > it automatically, which is why it told you it's already loaded. You can > use "kldstat" to verify. Otherwise it's statically-included in your > kernel. I noticed that. I got to put it on loader.conf: for the record, no good and won't be able to boot the box (usb live solved). I thought I needed to load the module to see the gpt. > Next item: this sounds like the crux of your issue. As I understand it, > NTFS support via kernel on FreeBSD is in an extremely bad state on > numerous levels. You can find complaints about lack-of or badly-done > UTF-8 filename/path support, lack of full write support, and the more > important/major Non-MPSAFE filesystem declaration here: > http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS unfortunately I didn't got to this point as I never got to see the partitions :( but I plan to use ntfs from fuse. > It looks like attilio@freebsd.org has taken ownership of the NTFS driver > in the kernel at this point. You may want to ask him if there are any > patches you could try. > > However, as I understand it, the fuse-based NTFS is more reliable and > has much higher compatibility. The trade-offs are added complexity > getting it all to work, and slower speed. > > Final item: you sent this mail twice, once to stable@freebsd.org, once to > freebsd-stable@freebsd.org. The mail ID headers appear to indicate > they were separate/unique and the mail content themselves slightly > differs (extra newlines, etc.). No need for this; they are the same > list. > > 3909 11/18 22:55 Nenhum_de_Nos (0.8K) mount GPT from Windows 7 in FreeBSD 9 > Message-ID: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> > 3910 11/19 03:20 Nenhum_de_Nos (1.1K) mount GPT from Windows 7 in FreeBSD 9 > Message-ID: <1c312a399e23d546f05f5e8396763323.squirrel@eternamente.info> and this, sorry for the double post. I sent the message to freebsd-stable@ and waited a couple of hours. Nothing showed on the list. I thought was a problem with it, then sent again: stable@freebsd.org 03:20 1.4 k mount GPT from Windows 7 in FreeBSD 9 freebsd-stable@freebsd.org Fri, 22:55 1.1 k mount GPT from Windows 7 in FreeBSD 9 this is from squirrelmail. first message 22:55 from Friday in my time zone, the second was 3:20 from Saturday. then I tried to subscribe again to the list (I thought I was no more on it), and got the message I was there ... again, sorry but I just thought I was not there anymore. Was there any issues with the list yesterday ? usually it is not that slow (I think). thanks, matheus -- Pregando o ódio a Oi desde 2007 We will call you cygnus, The God of balance you shall be A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? http://en.wikipedia.org/wiki/Posting_style From owner-freebsd-stable@FreeBSD.ORG Sat Nov 19 16:44:56 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C03B106566B for ; Sat, 19 Nov 2011 16:44:56 +0000 (UTC) (envelope-from edhoprima@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id BF1288FC0C for ; Sat, 19 Nov 2011 16:44:55 +0000 (UTC) Received: by bkbzs8 with SMTP id zs8so6344575bkb.13 for ; Sat, 19 Nov 2011 08:44:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=tD77YH4mXRttkvqfBhnBhSOhQD7SrHAqq7No8RgHOhA=; b=Yc5oyg68nSYuF2PitV3rdMG2krGRjmrLxKvf8OyJNeHOAcAldblGjV+K7hnN5ezXkT ZcjC3E66hHee02gsLLbU6gZKuFBr5IQVqVDYUkYPWdAYVgei/JNWK2CT+TpdWtY5BOWK u1q+yXPyXdyWL0vik4EMKPHlPKkFtrNZupJ6w= Received: by 10.204.154.137 with SMTP id o9mr7678177bkw.80.1321719432148; Sat, 19 Nov 2011 08:17:12 -0800 (PST) MIME-Version: 1.0 Sender: edhoprima@gmail.com Received: by 10.204.129.89 with HTTP; Sat, 19 Nov 2011 08:16:51 -0800 (PST) In-Reply-To: <5f508e72e9cfcf7ebfe8ec53a68851e9.squirrel@eternamente.info> References: <678178215ed5d725a42eb2b4ae25102c.squirrel@eternamente.info> <20111119095808.GA87444@icarus.home.lan> <5f508e72e9cfcf7ebfe8ec53a68851e9.squirrel@eternamente.info> From: Edho Arief Date: Sat, 19 Nov 2011 23:16:51 +0700 X-Google-Sender-Auth: ESNly5ZsNrOB28MBiiu_XxRVTEE Message-ID: To: Nenhum_de_Nos Content-Type: text/plain; charset=UTF-8 Cc: freebsd-stable@freebsd.org Subject: Re: mount GPT from Windows 7 in FreeBSD 9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Nov 2011 16:44:56 -0000 On Sat, Nov 19, 2011 at 8:51 PM, Nenhum_de_Nos wrote: > > > unfortunately I didn't got to this point as I never got to see the partitions :( but I plan to use > ntfs from fuse. > I've also experienced this and from what I can tell is somehow there's GPT table in the disk but not actually used. FreeBSD recognized this and decided it's a GPT-labeled disk while in fact it's still MBR - hence why it's not seen in /dev (since there's GPT) but seen in fdisk (since it doesn't understand GPT). I usually notice this when converting disk from GPT to MBR using Windows' Disk Management tool. -- O< ascii ribbon campaign - stop html mail - www.asciiribbon.org