From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 01:05:49 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8087A106566B for ; Sun, 23 Oct 2011 01:05:49 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: from outgoing.tristatelogic.com (segfault.tristatelogic.com [69.62.255.118]) by mx1.freebsd.org (Postfix) with ESMTP id 5E2B98FC17 for ; Sun, 23 Oct 2011 01:05:49 +0000 (UTC) Received: from segfault-nmh-helo.tristatelogic.com (localhost [127.0.0.1]) by segfault.tristatelogic.com (Postfix) with ESMTP id 4D0F9BDC34 for ; Sat, 22 Oct 2011 17:47:44 -0700 (PDT) To: freebsd-net@freebsd.org Date: Sat, 22 Oct 2011 17:47:44 -0700 Message-ID: <29994.1319330864@tristatelogic.com> From: "Ronald F. Guilmette" Subject: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 01:05:49 -0000 I've been slowly bringing up a fresh new 8.2-RELEASE system on one of my static IPs, and I've set up some minimalist ipfw rules, just for the time being, to try to protect it from Evil Invaders. I arranged for these rules to log all unexpected inbound packets coming in via the one and only ethernet card. The card has been ifconfig'd as follows: ifconfig_rl0="inet 69.62.255.119 netmask 255.255.255.0" I'll admit to being ignorant about many of the finer details of networking generally, but to my way of thinking, the above configuration should cause the card to really only listen for inbound packets addressed to 69.62.255.119. Yes? No? Well, anyway, that's been my experience in the past. The odd thing is that I'm getting some inbound packets logged by my final ``catch all'' deny & log rule in my IPFW rules list, where the destination IP address on the packets being logged is *not* 69.62.255.119. This is absolutely puzzling to me, and I hope that somebody can explain it to me. I mean how can this occur? The destination IP addresses in question aren;t even in the same /24 as my machine, so I really don;t understand how or why my card is even receiving these packets. The inbound packets in question are not really a problem. I can easily figure out how to add additional ipfw rules to block them completely. But the very fact that my ethernet card is even hearing them, given its configured IP address, is rather disturbing to me, because it obviously means that there's something deep going on here that I just don't understand, but I would like to understand it. The packets in question seem to come in three flavors. About 1/3 of them look like this in the /var/log/security file: Oct 22 17:12:38 coredump kernel: ipfw: 1600 Deny UDP 0.0.0.0:68 255.255.255.255:67 in via rl0 Some others look like this: Oct 22 17:12:27 coredump kernel: ipfw: 1600 Deny UDP 67.159.149.215:50669 255.255.255.255:2223 in via rl0 Still others look like this: Oct 22 17:12:01 coredump kernel: ipfw: 1600 Deny UDP 67.159.139.178:520 67.159.139.191:520 in via rl0 The destination addresses for all of the logged packets represented above are quite clearly *not* the IP address of the machine I'm setting up. Not even close. Note that the machine I've been setting up is on a static IP address on an ordinary end-luser DSL line. Note also that all addresses within the 67.159.128.0/19 block belong to my own ISP, Surewest Broadband. So it would seem to be the case that some other folks or businesses who use my same ISP may perhaps be sending out some funny (and misdirected?) packets, but that's not an issue that concerns me. What does concern me is just that fact that my ethernet card seems to be listening to packets that aren't even addressed to it, and I really just don't understand why. Any enlightenment would be appreciated. Regards, rfg P.S. This is the first time I've ever touched FreeBSD 8.x. I've been using 7.x releases in the past however, and before that 6.x and 5.x releases and I've really never seen anything quite like this before. Do 8.x releases now cause ethernet cards to listen for stuff they should not even be listening for? Color me perplexed. From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 01:50:25 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6656E106564A; Sun, 23 Oct 2011 01:50:25 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 2C5338FC08; Sun, 23 Oct 2011 01:50:24 +0000 (UTC) Received: from lstewart1.loshell.room52.net (ppp59-167-184-191.static.internode.on.net [59.167.184.191]) by lauren.room52.net (Postfix) with ESMTPSA id D59657E820; Sun, 23 Oct 2011 12:35:15 +1100 (EST) Message-ID: <4EA36F53.9050907@freebsd.org> Date: Sun, 23 Oct 2011 12:35:15 +1100 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111016 Thunderbird/7.0.1 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20111022084931.GD1697@garage.freebsd.pl> In-Reply-To: <20111022084931.GD1697@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, Andre Oppermann Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 01:50:25 -0000 On 10/22/11 19:49, Pawel Jakub Dawidek wrote: > The panic message says: > > panic: tcp_input negative window: tp 0xfffffe007763e000 rcv_nxt 3718269252 rcv_adv 3718268291 > > I only have picture of the backtrace: > > http://people.freebsd.org/~pjd/misc/panic_negative_window.jpg > ewww that is not good. Can you give us any more information about the machine and what it's doing? Is it terminating TCP connections from the internet at large or only local LAN (i.e. is there likely to be packet loss happening)? Are you doing TSO or LRO? Do you have any non-default tuning in place? Cheers, Lawrence From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 01:58:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C63C6106564A for ; Sun, 23 Oct 2011 01:58:09 +0000 (UTC) (envelope-from qing.li@bluecoat.com) Received: from whisker.bluecoat.com (whisker.bluecoat.com [216.52.23.28]) by mx1.freebsd.org (Postfix) with ESMTP id A6C238FC0C for ; Sun, 23 Oct 2011 01:58:09 +0000 (UTC) Received: from PWSVL-EXCHTS-02.internal.cacheflow.com (sai-rp.bluecoat.com [10.2.2.126] (may be forged)) by whisker.bluecoat.com (8.14.2/8.14.2) with ESMTP id p9N1w8sX029902 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Sat, 22 Oct 2011 18:58:09 -0700 (PDT) Received: from PWSVL-EXCMBX-01.internal.cacheflow.com ([fe80::15bc:12e2:4676:340f]) by PWSVL-EXCHTS-02.internal.cacheflow.com ([fe80::4910:317f:407:6ecc%14]) with mapi id 14.01.0289.001; Sat, 22 Oct 2011 18:58:03 -0700 From: "Li, Qing" To: "Ronald F. Guilmette" , "freebsd-net@freebsd.org" Thread-Topic: IPFW shows me Strangeness in fresh 8.2-RELEASE system Thread-Index: AQHMkR/wGsdDSnU5ak+MogtbbeZf9pWJK7mS Date: Sun, 23 Oct 2011 01:58:03 +0000 Message-ID: References: <29994.1319330864@tristatelogic.com> In-Reply-To: <29994.1319330864@tristatelogic.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [216.52.23.68] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Subject: RE: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 01:58:09 -0000 First thing comes to mind is to check if "rl0" is running in promiscuous mo= de.=0A= =0A= Check ifconfig output, and do a "ifconfig rl0 -promisc" just for good measu= re and=0A= see what happens.=0A= =0A= --Qing=0A= =0A= ________________________________________=0A= From: owner-freebsd-net@freebsd.org [owner-freebsd-net@freebsd.org] on beha= lf of Ronald F. Guilmette [rfg@tristatelogic.com]=0A= Sent: Saturday, October 22, 2011 5:47 PM=0A= To: freebsd-net@freebsd.org=0A= Subject: IPFW shows me Strangeness in fresh 8.2-RELEASE system=0A= =0A= I've been slowly bringing up a fresh new 8.2-RELEASE system on one of my=0A= static IPs, and I've set up some minimalist ipfw rules, just for the time= =0A= being, to try to protect it from Evil Invaders. I arranged for these rules= =0A= to log all unexpected inbound packets coming in via the one and only ethern= et=0A= card.=0A= =0A= The card has been ifconfig'd as follows:=0A= =0A= ifconfig_rl0=3D"inet 69.62.255.119 netmask 255.255.255.0"=0A= =0A= I'll admit to being ignorant about many of the finer details of networking= =0A= generally, but to my way of thinking, the above configuration should cause= =0A= the card to really only listen for inbound packets addressed to 69.62.255.1= 19.=0A= Yes? No?=0A= =0A= Well, anyway, that's been my experience in the past.=0A= =0A= The odd thing is that I'm getting some inbound packets logged by my final= =0A= ``catch all'' deny & log rule in my IPFW rules list, where the destination= =0A= IP address on the packets being logged is *not* 69.62.255.119.=0A= =0A= This is absolutely puzzling to me, and I hope that somebody can explain it= =0A= to me. I mean how can this occur? The destination IP addresses in questio= n=0A= aren;t even in the same /24 as my machine, so I really don;t understand how= =0A= or why my card is even receiving these packets.=0A= =0A= The inbound packets in question are not really a problem. I can easily=0A= figure out how to add additional ipfw rules to block them completely.=0A= But the very fact that my ethernet card is even hearing them, given its=0A= configured IP address, is rather disturbing to me, because it obviously=0A= means that there's something deep going on here that I just don't understan= d,=0A= but I would like to understand it.=0A= =0A= The packets in question seem to come in three flavors. About 1/3 of them l= ook=0A= like this in the /var/log/security file:=0A= =0A= Oct 22 17:12:38 coredump kernel: ipfw: 1600 Deny UDP 0.0.0.0:68 255.255.255= .255:67 in via rl0=0A= =0A= Some others look like this:=0A= =0A= Oct 22 17:12:27 coredump kernel: ipfw: 1600 Deny UDP 67.159.149.215:50669 2= 55.255.255.255:2223 in via rl0=0A= =0A= Still others look like this:=0A= =0A= Oct 22 17:12:01 coredump kernel: ipfw: 1600 Deny UDP 67.159.139.178:520 67.= 159.139.191:520 in via rl0=0A= =0A= The destination addresses for all of the logged packets represented above a= re=0A= quite clearly *not* the IP address of the machine I'm setting up. Not even= =0A= close.=0A= =0A= Note that the machine I've been setting up is on a static IP address on an= =0A= ordinary end-luser DSL line. Note also that all addresses within the=0A= 67.159.128.0/19 block belong to my own ISP, Surewest Broadband. So it woul= d=0A= seem to be the case that some other folks or businesses who use my same ISP= =0A= may perhaps be sending out some funny (and misdirected?) packets, but that'= s=0A= not an issue that concerns me. What does concern me is just that fact that= =0A= my ethernet card seems to be listening to packets that aren't even addresse= d=0A= to it, and I really just don't understand why.=0A= =0A= Any enlightenment would be appreciated.=0A= =0A= =0A= Regards,=0A= rfg=0A= =0A= =0A= P.S. This is the first time I've ever touched FreeBSD 8.x. I've been usin= g=0A= 7.x releases in the past however, and before that 6.x and 5.x releases and= =0A= I've really never seen anything quite like this before. Do 8.x releases no= w=0A= cause ethernet cards to listen for stuff they should not even be listening= =0A= for?=0A= =0A= Color me perplexed.=0A= _______________________________________________=0A= freebsd-net@freebsd.org mailing list=0A= http://lists.freebsd.org/mailman/listinfo/freebsd-net=0A= To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"=0A= From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 04:22:40 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA7471065673 for ; Sun, 23 Oct 2011 04:22:40 +0000 (UTC) (envelope-from barney@pit.databus.com) Received: from out.smtp-auth.no-ip.com (out.smtp-auth.no-ip.com [8.23.224.60]) by mx1.freebsd.org (Postfix) with ESMTP id 94DC38FC0A for ; Sun, 23 Oct 2011 04:22:40 +0000 (UTC) X-No-IP: databus.com@noip-smtp X-No-IP: databus.com@noip-smtp X-Report-Spam-To: abuse@no-ip.com Received: from pit.databus.com (pool-96-232-115-103.nycmny.fios.verizon.net [96.232.115.103]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: databus.com@noip-smtp) by smtp-auth.no-ip.com (Postfix) with ESMTPSA id A57DF40033F; Sat, 22 Oct 2011 21:06:44 -0700 (PDT) Received: from pit.databus.com (localhost [127.0.0.1]) by pit.databus.com (8.14.5/8.14.5) with ESMTP id p9N46gDW091674; Sun, 23 Oct 2011 00:06:42 -0400 (EDT) (envelope-from barney@pit.databus.com) X-DKIM: Sendmail DKIM Filter v2.8.3 pit.databus.com p9N46gDW091674 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=databus.com; s=20091218; t=1319342803; bh=Zf5VYW7mOLrHR+qkatFwj86Uvimtxq6hylIjLhoZrYw=; l=4070; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:In-Reply-To; b=W4lIJmhTKs2DiEJv+vxZMM1pz/QDAJw2AjxkCfBf1kxFYi/5n8GQ973IUNJgsSCeI 6Q6+Q+QgPuMxKIvl4BkTS2wjogzpxXwnggLJZ9UI5dFkhD5RNb8i/ccRZ3Aa5mGVyR arx/cjRTtBHAKiG3HnSNdUeJPc/97MrdBXNY8aCI= Received: (from barney@localhost) by pit.databus.com (8.14.5/8.14.5/Submit) id p9N46eV2091673; Sun, 23 Oct 2011 00:06:40 -0400 (EDT) (envelope-from barney) Date: Sun, 23 Oct 2011 00:06:40 -0400 From: Barney Wolff To: "Ronald F. Guilmette" Message-ID: <20111023040640.GA91490@pit.databus.com> References: <29994.1319330864@tristatelogic.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <29994.1319330864@tristatelogic.com> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 04:22:40 -0000 I would bet that all of those packets are being sent to the broadcast ethernet address. Certainly the DHCP and RIP packets are likely to have been. Try running tcpdump with -e to see if that's right. There's a lot of junk on DSL; live with it. Unless the volume is a significant fraction of your bandwidth, it's harmless. On Sat, Oct 22, 2011 at 05:47:44PM -0700, Ronald F. Guilmette wrote: > > I've been slowly bringing up a fresh new 8.2-RELEASE system on one of my > static IPs, and I've set up some minimalist ipfw rules, just for the time > being, to try to protect it from Evil Invaders. I arranged for these rules > to log all unexpected inbound packets coming in via the one and only ethernet > card. > > The card has been ifconfig'd as follows: > > ifconfig_rl0="inet 69.62.255.119 netmask 255.255.255.0" > > I'll admit to being ignorant about many of the finer details of networking > generally, but to my way of thinking, the above configuration should cause > the card to really only listen for inbound packets addressed to 69.62.255.119. > Yes? No? > > Well, anyway, that's been my experience in the past. > > The odd thing is that I'm getting some inbound packets logged by my final > ``catch all'' deny & log rule in my IPFW rules list, where the destination > IP address on the packets being logged is *not* 69.62.255.119. > > This is absolutely puzzling to me, and I hope that somebody can explain it > to me. I mean how can this occur? The destination IP addresses in question > aren;t even in the same /24 as my machine, so I really don;t understand how > or why my card is even receiving these packets. > > The inbound packets in question are not really a problem. I can easily > figure out how to add additional ipfw rules to block them completely. > But the very fact that my ethernet card is even hearing them, given its > configured IP address, is rather disturbing to me, because it obviously > means that there's something deep going on here that I just don't understand, > but I would like to understand it. > > The packets in question seem to come in three flavors. About 1/3 of them look > like this in the /var/log/security file: > > Oct 22 17:12:38 coredump kernel: ipfw: 1600 Deny UDP 0.0.0.0:68 255.255.255.255:67 in via rl0 > > Some others look like this: > > Oct 22 17:12:27 coredump kernel: ipfw: 1600 Deny UDP 67.159.149.215:50669 255.255.255.255:2223 in via rl0 > > Still others look like this: > > Oct 22 17:12:01 coredump kernel: ipfw: 1600 Deny UDP 67.159.139.178:520 67.159.139.191:520 in via rl0 > > The destination addresses for all of the logged packets represented above are > quite clearly *not* the IP address of the machine I'm setting up. Not even > close. > > Note that the machine I've been setting up is on a static IP address on an > ordinary end-luser DSL line. Note also that all addresses within the > 67.159.128.0/19 block belong to my own ISP, Surewest Broadband. So it would > seem to be the case that some other folks or businesses who use my same ISP > may perhaps be sending out some funny (and misdirected?) packets, but that's > not an issue that concerns me. What does concern me is just that fact that > my ethernet card seems to be listening to packets that aren't even addressed > to it, and I really just don't understand why. > > Any enlightenment would be appreciated. > > > Regards, > rfg > > > P.S. This is the first time I've ever touched FreeBSD 8.x. I've been using > 7.x releases in the past however, and before that 6.x and 5.x releases and > I've really never seen anything quite like this before. Do 8.x releases now > cause ethernet cards to listen for stuff they should not even be listening > for? > > Color me perplexed. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Barney Wolff I never met a computer I didn't like. From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 05:10:57 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4350E1065673; Sun, 23 Oct 2011 05:10:57 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id D9DC38FC12; Sun, 23 Oct 2011 05:10:56 +0000 (UTC) Received: by iaky10 with SMTP id y10so8202582iak.13 for ; Sat, 22 Oct 2011 22:10:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version:content-type; bh=cw5I1gzp7n0pAQURDcFw8AR8RrTbf9jnmeFqv+CTVLU=; b=k76d6/j+ECAxt4OIfzH3w/5Gu4RUky04GIi2eFWcRYOOhsXbUmlDc8yBHS5bXncP81 B/hJp9YH0MJuNTGQ9jY4XDFKu5W+1eM0IOEDsZUBRrgN9uinSHNPgwCl+C01tD515Bxv HaCrkuqXG3xSZ2yF03hszfXaPFU2vpW5PSHuo= Received: by 10.42.153.74 with SMTP id l10mr33891595icw.52.1319346656139; Sat, 22 Oct 2011 22:10:56 -0700 (PDT) Received: from c-24-6-49-154.hsd1.ca.comcast.net (c-24-6-49-154.hsd1.ca.comcast.net. [24.6.49.154]) by mx.google.com with ESMTPS id ge16sm48341449ibb.2.2011.10.22.22.10.54 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 22 Oct 2011 22:10:55 -0700 (PDT) Date: Sat, 22 Oct 2011 22:10:52 -0700 (PDT) From: Garrett Cooper To: Lawrence Stewart In-Reply-To: <4EA36F53.9050907@freebsd.org> Message-ID: References: <20111022084931.GD1697@garage.freebsd.pl> <4EA36F53.9050907@freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Andre Oppermann , freebsd-current@freebsd.org, Pawel Jakub Dawidek Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 05:10:57 -0000 On Sun, 23 Oct 2011, Lawrence Stewart wrote: > On 10/22/11 19:49, Pawel Jakub Dawidek wrote: >> The panic message says: >> >> panic: tcp_input negative window: tp 0xfffffe007763e000 rcv_nxt >> 3718269252 rcv_adv 3718268291 >> >> I only have picture of the backtrace: >> >> http://people.freebsd.org/~pjd/misc/panic_negative_window.jpg >> > > ewww that is not good. Can you give us any more information about the machine > and what it's doing? Is it terminating TCP connections from the internet at > large or only local LAN (i.e. is there likely to be packet loss happening)? > Are you doing TSO or LRO? Do you have any non-default tuning in place? I can't speak for pjd@, but when this issue occured a couple months ago for me, the system I had run into the issue with was doing packet forwarding via ipfw from an internal NAT via ipfw to a corporate network. The system was using bce(4) on the internal and external interface. Thanks, -Garrett From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 06:11:27 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A8EA41065698; Sun, 23 Oct 2011 06:11:27 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 018678FC1F; Sun, 23 Oct 2011 06:11:26 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id 31661464; Sun, 23 Oct 2011 08:11:24 +0200 (CEST) Date: Sun, 23 Oct 2011 08:10:38 +0200 From: Pawel Jakub Dawidek To: Lawrence Stewart Message-ID: <20111023061038.GE1697@garage.freebsd.pl> References: <20111022084931.GD1697@garage.freebsd.pl> <4EA36F53.9050907@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="k3qmt+ucFURmlhDS" Content-Disposition: inline In-Reply-To: <4EA36F53.9050907@freebsd.org> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org, freebsd-current@freebsd.org, Andre Oppermann Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 06:11:27 -0000 --k3qmt+ucFURmlhDS Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 23, 2011 at 12:35:15PM +1100, Lawrence Stewart wrote: > On 10/22/11 19:49, Pawel Jakub Dawidek wrote: > > The panic message says: > > > > panic: tcp_input negative window: tp 0xfffffe007763e000 rcv_nxt 371826= 9252 rcv_adv 3718268291 > > > > I only have picture of the backtrace: > > > > http://people.freebsd.org/~pjd/misc/panic_negative_window.jpg > > >=20 > ewww that is not good. Can you give us any more information about the=20 > machine and what it's doing? Is it terminating TCP connections from the= =20 > internet at large or only local LAN (i.e. is there likely to be packet=20 > loss happening)? Are you doing TSO or LRO? Do you have any non-default=20 > tuning in place? It is my local file server. It is doing NFS and AFP over LAN and also downloads files from the internet. It is triggered after few hours. I changed the KASSERT() into printf() and added printing 'win' variable and this is what got logged during the night: 05:16:24 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110782726= 9 rcv_adv 1107826256 win=3D242 05:16:29 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110783345= 1 rcv_adv 1107832977 win=3D880 05:16:41 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110784956= 3 rcv_adv 1107848860 win=3D639 05:20:02 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110810823= 0 rcv_adv 1108107331 win=3D567 05:24:30 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110843330= 2 rcv_adv 1108432272 win=3D974 05:24:46 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110845038= 5 rcv_adv 1108450060 win=3D751 05:26:44 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110857481= 8 rcv_adv 1108573851 win=3D71 05:28:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110865410= 3 rcv_adv 1108653166 win=3D0 05:28:43 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110869239= 6 rcv_adv 1108691451 win=3D0 05:30:06 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110878125= 8 rcv_adv 1108780372 win=3D235 05:35:05 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110906757= 8 rcv_adv 1109067335 win=3D663 05:37:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110918040= 3 rcv_adv 1109179411 win=3D0 05:41:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 110942826= 5 rcv_adv 1109427375 win=3D170 And the systems seems to be fine. I'm happy to test patches, but one round would take 24h. My suggestion would be that if we won't be able to fix it before 9.0, we should turn this assertion off, as the system seems to be able to recover. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --k3qmt+ucFURmlhDS Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk6jr9oACgkQForvXbEpPzRU0QCdFfQwbyw7POvY827S/P9iFvcG Ru8An3jQWQ3oDpiSjZGeToiRh1c/5KGK =oRuU -----END PGP SIGNATURE----- --k3qmt+ucFURmlhDS-- From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 09:15:30 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6427B106566B for ; Sun, 23 Oct 2011 09:15:30 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id B26338FC12 for ; Sun, 23 Oct 2011 09:15:29 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p9N8ik5V047727 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 23 Oct 2011 11:44:46 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id p9N8ij5D083505; Sun, 23 Oct 2011 11:44:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id p9N8ijEU083504; Sun, 23 Oct 2011 11:44:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 23 Oct 2011 11:44:45 +0300 From: Kostik Belousov To: Pawel Jakub Dawidek Message-ID: <20111023084445.GB50300@deviant.kiev.zoral.com.ua> References: <20111022084931.GD1697@garage.freebsd.pl> <4EA36F53.9050907@freebsd.org> <20111023061038.GE1697@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NP+fiIeNjDlB/E6h" Content-Disposition: inline In-Reply-To: <20111023061038.GE1697@garage.freebsd.pl> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Lawrence Stewart , freebsd-current@freebsd.org, Andre Oppermann , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 09:15:30 -0000 --NP+fiIeNjDlB/E6h Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: > On Sun, Oct 23, 2011 at 12:35:15PM +1100, Lawrence Stewart wrote: > > On 10/22/11 19:49, Pawel Jakub Dawidek wrote: > > > The panic message says: > > > > > > panic: tcp_input negative window: tp 0xfffffe007763e000 rcv_nxt 3718= 269252 rcv_adv 3718268291 > > > > > > I only have picture of the backtrace: > > > > > > http://people.freebsd.org/~pjd/misc/panic_negative_window.jpg > > > > >=20 > > ewww that is not good. Can you give us any more information about the= =20 > > machine and what it's doing? Is it terminating TCP connections from the= =20 > > internet at large or only local LAN (i.e. is there likely to be packet= =20 > > loss happening)? Are you doing TSO or LRO? Do you have any non-default= =20 > > tuning in place? >=20 > It is my local file server. It is doing NFS and AFP over LAN and also > downloads files from the internet. It is triggered after few hours. > I changed the KASSERT() into printf() and added printing 'win' variable > and this is what got logged during the night: >=20 > 05:16:24 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1107827= 269 rcv_adv 1107826256 win=3D242 > 05:16:29 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1107833= 451 rcv_adv 1107832977 win=3D880 > 05:16:41 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1107849= 563 rcv_adv 1107848860 win=3D639 > 05:20:02 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108108= 230 rcv_adv 1108107331 win=3D567 > 05:24:30 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108433= 302 rcv_adv 1108432272 win=3D974 > 05:24:46 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108450= 385 rcv_adv 1108450060 win=3D751 > 05:26:44 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108574= 818 rcv_adv 1108573851 win=3D71 > 05:28:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108654= 103 rcv_adv 1108653166 win=3D0 > 05:28:43 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108692= 396 rcv_adv 1108691451 win=3D0 > 05:30:06 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1108781= 258 rcv_adv 1108780372 win=3D235 > 05:35:05 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1109067= 578 rcv_adv 1109067335 win=3D663 > 05:37:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1109180= 403 rcv_adv 1109179411 win=3D0 > 05:41:03 tcp_input negative window: tp 0xfffffe0026772b70 rcv_nxt 1109428= 265 rcv_adv 1109427375 win=3D170 >=20 > And the systems seems to be fine. >=20 > I'm happy to test patches, but one round would take 24h. >=20 > My suggestion would be that if we won't be able to fix it before 9.0, > we should turn this assertion off, as the system seems to be able to > recover. Shipped kernels have all assertions turned off. --NP+fiIeNjDlB/E6h Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk6j0/0ACgkQC3+MBN1Mb4h5iwCfQa26LAP0gzvVdmSIiR9rLNvj 5UsAnRPP8tZxdYYn9jOXOHo2pvnPM0bJ =/lze -----END PGP SIGNATURE----- --NP+fiIeNjDlB/E6h-- From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 09:26:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA3F7106564A for ; Sun, 23 Oct 2011 09:26:38 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: from outgoing.tristatelogic.com (segfault.tristatelogic.com [69.62.255.118]) by mx1.freebsd.org (Postfix) with ESMTP id C01D28FC12 for ; Sun, 23 Oct 2011 09:26:38 +0000 (UTC) Received: from segfault-nmh-helo.tristatelogic.com (localhost [127.0.0.1]) by segfault.tristatelogic.com (Postfix) with ESMTP id E783CBDC34 for ; Sun, 23 Oct 2011 02:26:36 -0700 (PDT) To: freebsd-net@freebsd.org Date: Sun, 23 Oct 2011 02:26:36 -0700 Message-ID: <33528.1319361996@tristatelogic.com> From: "Ronald F. Guilmette" Subject: RE: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 09:26:39 -0000 "Li, Qing" wrote: >First thing comes to mind is to check if "rl0" is running in promiscuous mode. No, the interface is NOT in promiscuous mode. So I still don't understand how it can be receiving these packets. From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 09:33:31 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29D951065670 for ; Sun, 23 Oct 2011 09:33:31 +0000 (UTC) (envelope-from yilinjing2006@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 082478FC14 for ; Sun, 23 Oct 2011 09:33:30 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RHuQs-0000bH-8g for freebsd-net@freebsd.org; Sun, 23 Oct 2011 02:33:30 -0700 Date: Sun, 23 Oct 2011 02:33:30 -0700 (PDT) From: jyl_2006 To: freebsd-net@freebsd.org Message-ID: <1319362410261-4929128.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: SCTP : problems in sending ASCONF chunks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 09:33:31 -0000 Hi, I use FreeBSD 9.0 Beta2 on two computers .=20 One have one wireless card, and the other one has two wireless card. When I use sctp with the feature of automatic address reconfiguration , I notice that no Asconf Chunk send. Here are INIT chunk message about association: Both in the INIT chunk and INIT ACK chunk , the Supported Extensions Parameter field indicates that ASCONF, ASCONF-ACK, FORWARD-TSN, PKTDROP, STREAM_RESET and AUTH are supported. So I think it will be ok ,if I use sctp_bindx() to bind a ip address, but n= o ASCONF chunk shows in Wireshark. After that ,I recompiled the kernel with the option of debug enable, in the file from dmesg =EF=BC=8CI notice that kernel get the new ip address ,but i= t does not send this ASCONF CHUNK. Here are messages: 192.168.1.50 is the new ip address that I want to bind to the association. USR Send complete qo:0 prw:1863859 unsent:18 tf:18 cooq:1 toqs:18 err:0 Ok laddr->ifa:0xc7744280 is possible, asconf_queue_mgmt: inserted asconf ADD_IP_ADDRESS: IPv4 address: 192.168.1.50:0 m-c-o put out 0 Ok, we have put out 0 chunks sctp_input() length:28 iphlen:20 sctp_input(): Packet of length 48 received on wlan0 with csum_flags 0x0. Ok, Common input processing called, m:0xc721dc00 iphlen:20 offset:32 length:48 stcb:0xc718e5dc stcb:0xc718e5dc state:8 sctp_process_control: iphlen=3D20, offset=3D32, length=3D48 stcb:0xc718e5dc sctp_process_control: processing a chunk type=3D3, len=3D16 -- View this message in context: http://freebsd.1045724.n5.nabble.com/SCTP-pro= blems-in-sending-ASCONF-chunks-tp4929128p4929128.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 09:44:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 365D9106566B for ; Sun, 23 Oct 2011 09:44:34 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id 9E2948FC08 for ; Sun, 23 Oct 2011 09:44:33 +0000 (UTC) Received: from [192.168.1.124] (p508FD9BF.dip.t-dialin.net [80.143.217.191]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id 993FC1C0B4611; Sun, 23 Oct 2011 11:44:31 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=utf-8 From: Michael Tuexen In-Reply-To: <1319362410261-4929128.post@n5.nabble.com> Date: Sun, 23 Oct 2011 11:44:30 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2E60D876-9B2C-4039-9366-40451D5914E2@lurchi.franken.de> References: <1319362410261-4929128.post@n5.nabble.com> To: jyl_2006 X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-net@freebsd.org Subject: Re: SCTP : problems in sending ASCONF chunks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 09:44:34 -0000 On Oct 23, 2011, at 11:33 AM, jyl_2006 wrote: > Hi, > I use FreeBSD 9.0 Beta2 on two computers .=20 > One have one wireless card, and the other one has two wireless card. >=20 > When I use sctp with the feature of automatic address reconfiguration = , I > notice that no Asconf Chunk send. >=20 > Here are INIT chunk message about association: > Both in the INIT chunk and INIT ACK chunk , the Supported Extensions > Parameter field indicates that ASCONF, ASCONF-ACK, FORWARD-TSN, = PKTDROP, > STREAM_RESET and AUTH are supported. >=20 > So I think it will be ok ,if I use sctp_bindx() to bind a ip address, = but no > ASCONF chunk shows in Wireshark. >=20 > After that ,I recompiled the kernel with the option of debug enable, = in the > file from dmesg =EF=BC=8CI notice that kernel get the new ip address = ,but it does > not send this ASCONF CHUNK. > Here are messages: >=20 > 192.168.1.50 is the new ip address that I want to bind to the = association. Can you provide the IP-addresses you use? Best regards Michael >=20 > USR Send complete qo:0 prw:1863859 unsent:18 tf:18 cooq:1 toqs:18 = err:0 > Ok laddr->ifa:0xc7744280 is possible, asconf_queue_mgmt: inserted = asconf > ADD_IP_ADDRESS: IPv4 address: 192.168.1.50:0 > m-c-o put out 0 > Ok, we have put out 0 chunks > sctp_input() length:28 iphlen:20 > sctp_input(): Packet of length 48 received on wlan0 with csum_flags = 0x0. > Ok, Common input processing called, m:0xc721dc00 iphlen:20 offset:32 > length:48 stcb:0xc718e5dc > stcb:0xc718e5dc state:8 > sctp_process_control: iphlen=3D20, offset=3D32, length=3D48 = stcb:0xc718e5dc > sctp_process_control: processing a chunk type=3D3, len=3D16 >=20 >=20 > -- > View this message in context: = http://freebsd.1045724.n5.nabble.com/SCTP-problems-in-sending-ASCONF-chunk= s-tp4929128p4929128.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 05:22:59 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0E5D106566C for ; Sun, 23 Oct 2011 05:22:59 +0000 (UTC) (envelope-from g.achari@yahoo.com) Received: from nm4-vm0.bullet.mail.ne1.yahoo.com (nm4-vm0.bullet.mail.ne1.yahoo.com [98.138.90.253]) by mx1.freebsd.org (Postfix) with SMTP id 8C8198FC0C for ; Sun, 23 Oct 2011 05:22:59 +0000 (UTC) Received: from [98.138.90.55] by nm4.bullet.mail.ne1.yahoo.com with NNFMP; 23 Oct 2011 05:10:15 -0000 Received: from [98.138.88.233] by tm8.bullet.mail.ne1.yahoo.com with NNFMP; 23 Oct 2011 05:10:15 -0000 Received: from [127.0.0.1] by omp1033.mail.ne1.yahoo.com with NNFMP; 23 Oct 2011 05:10:15 -0000 X-Yahoo-Newman-Id: 60049.97015.bm@omp1033.mail.ne1.yahoo.com Received: (qmail 90949 invoked from network); 23 Oct 2011 05:10:14 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=DKIM-Signature:Message-ID:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:MIME-Version:Date:Subject:From:Reply-To:To:Cc:Content-Type:Content-Transfer-Encoding; b=Sl+j18zkOpM0pbxcW2pwQeK7du66hJ22qQHh9tLWY8QeYBX2KbDnDwGkku1fnsbCvqHyur5vWZp08pKJ8ZgkKfnbpjLMsEzsHhU4UfUdTlXA4OVNJmJNftxxsAWsCQezScPAE+eiFngz/FpGk6gvvCpolHTCD5ahHrohGsBgEe4= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1319346614; bh=sj0BSMFBqW6YI3i5vl8ppIi11T4iUddOyS6pzaqiWa8=; h=Message-ID:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:MIME-Version:Date:Subject:From:Reply-To:To:Cc:Content-Type:Content-Transfer-Encoding; b=juhiFqB0HBvkCCFBvwSX0Pme8K4pwsYVyZt9V6IzJ+d10fI1WeQkJ7ttqAEzzO2wRQe1a3QzdfN+wziiCLA+nw5ZzNNo61Weg+hSaX1jeloVWCPLmoUI2ApjhHGPnfbvOw3Gv/vzYRSH4uLje+a6KcnIDMdWkxT2LXCaYlonK0A= Message-ID: <919871.89976.qm@smtp105-mob.biz.mail.ne1.yahoo.com> X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: DLkF7G8VM1nFCSZYXsDZGgVwMw6DWCKzvV9_z7Kk1Sv6aMn OegAScl4NW3XJ8v_XuADFtjiJsJ.EmGdB3Fdk1lCiGOiChz1hI7t6ht9pvgq 4u9k5xM7j5Kl2fEyTBHvozYy5JJBc_o_iu4yVG8OJ46glUWZZ3f.jpbF2dmH qDV6J4YkGZUewO4JkxzG88PidCYz.9ihkyRcuiXvrkqBIaGjw3P9WA3tuegO 58VfHS394rgP3KLde1wRLTf6rbe9aL9M06DToMP2FyxZwhkh9ZnOLexaqsuq RlV3QUGbglB6wEHLb14f4hZM2X1cQs7_0hTuxBP04mCOOyZR05eUxUX1A168 cgmVQfh5wdHuF6JHt5_XN2g-- X-Yahoo-SMTP: GYF4kgSswBDSmhd3we.7JOyd1spW Received: from nokia.com (g.achari@66.54.85.7 with xymcookie) by smtp105-mob.biz.mail.ne1.yahoo.com with SMTP; 22 Oct 2011 22:10:14 -0700 PDT MIME-Version: 1.0 Date: Sun, 23 Oct 2011 05:10:11 +0000 From: "g.achari@yahoo.com" To: "freebsd-net@freebsd.org" Content-Type: TEXT/PLAIN; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Sun, 23 Oct 2011 11:05:35 +0000 Cc: "419CACF2.5020402@norteglobal.com" <419CACF2.5020402@norteglobal.com> Subject: How can I create and use a Tap device&In-Reply X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "g.achari@yahoo.com" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 05:22:59 -0000 Sent from my Nokia phone From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 15:59:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA226106564A; Sun, 23 Oct 2011 15:59:16 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 8B0CF8FC08; Sun, 23 Oct 2011 15:59:16 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id DEB075D5; Sun, 23 Oct 2011 17:59:14 +0200 (CEST) Date: Sun, 23 Oct 2011 17:58:28 +0200 From: Pawel Jakub Dawidek To: Kostik Belousov Message-ID: <20111023155827.GH1697@garage.freebsd.pl> References: <20111022084931.GD1697@garage.freebsd.pl> <4EA36F53.9050907@freebsd.org> <20111023061038.GE1697@garage.freebsd.pl> <20111023084445.GB50300@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="K1n7F7fSdjvFAEnM" Content-Disposition: inline In-Reply-To: <20111023084445.GB50300@deviant.kiev.zoral.com.ua> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Lawrence Stewart , freebsd-current@freebsd.org, Andre Oppermann , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 15:59:16 -0000 --K1n7F7fSdjvFAEnM Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: > On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: > > My suggestion would be that if we won't be able to fix it before 9.0, > > we should turn this assertion off, as the system seems to be able to > > recover. >=20 > Shipped kernels have all assertions turned off. Yes, I'm aware of that, but many people compile their production kernels with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing it into a printf, so it will be visible. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --K1n7F7fSdjvFAEnM Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk6kOaMACgkQForvXbEpPzT1ngCgtl7iI+54RRH21P4rNgTdJm/V 2OQAn08pCE6pj5aAo4Sz6SiCLkv3rXlR =eaYX -----END PGP SIGNATURE----- --K1n7F7fSdjvFAEnM-- From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 22:01:31 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32C15106566B for ; Sun, 23 Oct 2011 22:01:31 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from relay1-bcrtfl2.verio.net (relay1-bcrtfl2.verio.net [131.103.218.142]) by mx1.freebsd.org (Postfix) with ESMTP id 024F18FC08 for ; Sun, 23 Oct 2011 22:01:30 +0000 (UTC) Received: from iad-wprd-xchw01.corp.verio.net (iad-wprd-xchw01.corp.verio.net [198.87.7.164]) by relay1-bcrtfl2.verio.net (Postfix) with ESMTP id 4FA98B0380E7 for ; Sun, 23 Oct 2011 17:31:53 -0400 (EDT) thread-index: AcyRyy5CyR5GDh/aQu61AIvwpPP9Ag== Received: from hometx-733b1p1.corp.verio.net ([10.144.2.53]) by iad-wprd-xchw01.corp.verio.net over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Sun, 23 Oct 2011 17:31:43 -0400 Received: by hometx-733b1p1.corp.verio.net (sSMTP sendmail emulation); Sun, 23 Oct 2011 16:31:34 -0500 Date: Sun, 23 Oct 2011 16:31:34 -0500 From: "David DeSimone" Content-Transfer-Encoding: 7bit To: Message-ID: <20111023213133.GC1536@verio.net> Content-Class: urn:content-classes:message Mail-Followup-To: freebsd-net@freebsd.org Importance: normal Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4862 References: <33528.1319361996@tristatelogic.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-reply-to: <33528.1319361996@tristatelogic.com> Precedence: bulk User-Agent: Mutt/1.5.20 (2009-12-10) X-OriginalArrivalTime: 23 Oct 2011 21:31:44.0741 (UTC) FILETIME=[297DF150:01CC91CB] Subject: Re: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 22:01:31 -0000 Ronald F. Guilmette wrote: > > "Li, Qing" wrote: > > >First thing comes to mind is to check if "rl0" is running in promiscuous mode. > > No, the interface is NOT in promiscuous mode. > > So I still don't understand how it can be receiving these packets. Ethernet cards filter their traffic based on MAC address, not based on IP address. Use tcpdump -e to examine the destination MAC of the packets you are receiving, in order to determine whether you should receive them. -- David DeSimone == Network Admin == fox@verio.net "I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it." -- Clarence Darrow This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you. From owner-freebsd-net@FreeBSD.ORG Sun Oct 23 22:34:40 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E986E106564A; Sun, 23 Oct 2011 22:34:40 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id C241C8FC0A; Sun, 23 Oct 2011 22:34:40 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9NMYeil039767; Sun, 23 Oct 2011 22:34:40 GMT (envelope-from eadler@freefall.freebsd.org) Received: (from eadler@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9NMYer5039763; Sun, 23 Oct 2011 22:34:40 GMT (envelope-from eadler) Date: Sun, 23 Oct 2011 22:34:40 GMT Message-Id: <201110232234.p9NMYer5039763@freefall.freebsd.org> To: eadler@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: eadler@FreeBSD.org Cc: Subject: Re: kern/161908: [netgraph] [patch] ng_vlan update for QinQ support X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2011 22:34:41 -0000 Old Synopsis: ng_vlan update for QinQ support New Synopsis: [netgraph] [patch] ng_vlan update for QinQ support Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: eadler Responsible-Changed-When: Sun Oct 23 22:34:17 UTC 2011 Responsible-Changed-Why: assign and add tags http://www.freebsd.org/cgi/query-pr.cgi?pr=161908 From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 02:20:45 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B77B2106564A for ; Mon, 24 Oct 2011 02:20:45 +0000 (UTC) (envelope-from yilinjing2006@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 93BB88FC13 for ; Mon, 24 Oct 2011 02:20:45 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIA9c-0008Qv-SW for freebsd-net@freebsd.org; Sun, 23 Oct 2011 19:20:44 -0700 Date: Sun, 23 Oct 2011 19:20:44 -0700 (PDT) From: jyl_2006 To: freebsd-net@freebsd.org Message-ID: <1319422844876-4931035.post@n5.nabble.com> In-Reply-To: <2E60D876-9B2C-4039-9366-40451D5914E2@lurchi.franken.de> References: <1319362410261-4929128.post@n5.nabble.com> <2E60D876-9B2C-4039-9366-40451D5914E2@lurchi.franken.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: SCTP : problems in sending ASCONF chunks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 02:20:45 -0000 Hi,T=C3=BCxen I will provide more detail: The Topology is: --------1(192.168.1.20) computer A ---------------------------1 computer B(192.168.1.80) --------2(192.168.1.50) means computer has two wireless cards , I name them with A_1 and A_2, computer B has one wireless cards, and its name is B_1. Now, A_1 init a association with B_1. Here, I provide message getting from wireshark , INIT=EF=BC=9A Internet Protocol, Src: 192.168.1.20 (192.168.1.20), Dst: 192.168.1.80 (192.168.1.80) Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) INIT=E3=80=80ACK: Internet Protocol, Src: 192.168.1.80 (192.168.1.80), Dst: 192.168.1.20 (192.168.1.20) Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) Debug message: SCTP_SACK process cum_ack:45452434 num_seg:0 a_rwnd:1864135 Check for chunk output prw:1864135 tqe:1 tf=3D0 Send called addr:0xc591c980 send length 2 Calling ipv4 output routine from low level src addr:c0a80114 Destination is c0a80150 RTP route is 0xc5caaaf8 through IP output returns 0 m-c-o put out 1 Ok, we have put out 1 chunks USR Send complete qo:0 prw:1863859 unsent:18 tf:18 cooq:1 toqs:18 err:0 Ok laddr->ifa:0xc5baab00 is possible, asconf_queue_mgmt: inserted asconf ADD_IP_ADDRESS: IPv4 address: 192.168.1.50:0 m-c-o put out 0 Ok, we have put out 0 chunks sctp_input() length:28 iphlen:20 sctp_input(): Packet of length 48 received on wlan0 with csum_flags 0x0. Ok, Common input processing called, m:0xc72dab00 iphlen:20 offset:32 length:48 stcb:0xcc2335dc stcb:0xcc2335dc state:8 sctp_process_control: iphlen=3D20, offset=3D32, length=3D48 stcb:0xcc2335dc sctp_process_control: processing a chunk type=3D3, len=3D16 Actually,I also test following Topology : --------1(192.168.1.20) =20 --------------1(192.168.1.80) computer A --------------------------- computer B --------2(192.168.2.20) =20 --------------2(192.168.2.80) means computer has two wireless cards , I name them with A_1 and A_2, computer B has two wireless cards, and its name is B_1 and B_2. The result from wireshark and debug message have same results. -- View this message in context: http://freebsd.1045724.n5.nabble.com/SCTP-pro= blems-in-sending-ASCONF-chunks-tp4929128p4931035.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 05:48:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6E5E106566C for ; Mon, 24 Oct 2011 05:48:20 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id 4490E8FC0C for ; Mon, 24 Oct 2011 05:48:20 +0000 (UTC) Received: from [192.168.1.124] (p508FBE32.dip.t-dialin.net [80.143.190.50]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id A4B5E1C0C0BEA; Mon, 24 Oct 2011 07:48:18 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=utf-8 From: Michael Tuexen In-Reply-To: <1319422844876-4931035.post@n5.nabble.com> Date: Mon, 24 Oct 2011 07:48:17 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <81904D4D-B792-4A08-8113-77F62E4F7C77@lurchi.franken.de> References: <1319362410261-4929128.post@n5.nabble.com> <2E60D876-9B2C-4039-9366-40451D5914E2@lurchi.franken.de> <1319422844876-4931035.post@n5.nabble.com> To: jyl_2006 X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-net@freebsd.org Subject: Re: SCTP : problems in sending ASCONF chunks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 05:48:21 -0000 Hi, OK, let me try to test this on one of my machines. I might take a couple of days... Best regards Michael On Oct 24, 2011, at 4:20 AM, jyl_2006 wrote: > Hi,T=C3=BCxen >=20 > I will provide more detail: > The Topology is: > --------1(192.168.1.20) > computer A ---------------------------1 computer B(192.168.1.80) > --------2(192.168.1.50) > means computer has two wireless cards , I name them with A_1 and A_2, > computer B has one wireless cards, and its name is B_1. >=20 > Now, A_1 init a association with B_1. Here, I provide message getting = from > wireshark , > INIT=EF=BC=9A > Internet Protocol, Src: 192.168.1.20 (192.168.1.20), Dst: 192.168.1.80 > (192.168.1.80) > Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, > FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) > INIT=E3=80=80ACK: > Internet Protocol, Src: 192.168.1.80 (192.168.1.80), Dst: 192.168.1.20 > (192.168.1.20) > Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, > FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) >=20 > Debug message: > SCTP_SACK process cum_ack:45452434 num_seg:0 a_rwnd:1864135 > Check for chunk output prw:1864135 tqe:1 tf=3D0 > Send called addr:0xc591c980 send length 2 > Calling ipv4 output routine from low level src addr:c0a80114 > Destination is c0a80150 > RTP route is 0xc5caaaf8 through > IP output returns 0 > m-c-o put out 1 > Ok, we have put out 1 chunks > USR Send complete qo:0 prw:1863859 unsent:18 tf:18 cooq:1 toqs:18 = err:0 > Ok laddr->ifa:0xc5baab00 is possible, asconf_queue_mgmt: inserted = asconf > ADD_IP_ADDRESS: IPv4 address: 192.168.1.50:0 > m-c-o put out 0 > Ok, we have put out 0 chunks > sctp_input() length:28 iphlen:20 > sctp_input(): Packet of length 48 received on wlan0 with csum_flags = 0x0. > Ok, Common input processing called, m:0xc72dab00 iphlen:20 offset:32 > length:48 stcb:0xcc2335dc > stcb:0xcc2335dc state:8 > sctp_process_control: iphlen=3D20, offset=3D32, length=3D48 = stcb:0xcc2335dc > sctp_process_control: processing a chunk type=3D3, len=3D16 >=20 > Actually,I also test following Topology : > --------1(192.168.1.20) =20 > --------------1(192.168.1.80) > computer A --------------------------- computer B > --------2(192.168.2.20) =20 > --------------2(192.168.2.80) > means computer has two wireless cards , I name them with A_1 and A_2, > computer B has two wireless cards, and its name is B_1 and B_2. >=20 > The result from wireshark and debug message have same results. >=20 >=20 > -- > View this message in context: = http://freebsd.1045724.n5.nabble.com/SCTP-problems-in-sending-ASCONF-chunk= s-tp4929128p4931035.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 09:59:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 709E3106566B for ; Mon, 24 Oct 2011 09:59:13 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 331E48FC16 for ; Mon, 24 Oct 2011 09:59:12 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIH2R-0000Su-51 for freebsd-net@freebsd.org; Mon, 24 Oct 2011 02:41:47 -0700 Date: Mon, 24 Oct 2011 02:41:47 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319449307149-4931883.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 09:59:13 -0000 There is my FreeBSD box: kernel --------------- # # GENERIC -- Generic kernel configuration file for FreeBSD/i386 # # For more information on this file, please read the config(5) manual page, # and/or the handbook section on Kernel Configuration Files: # # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html # # The handbook is also available locally in /usr/share/doc/handbook # if you've installed the doc distribution, otherwise always see the # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the # latest information. # # An exhaustive list of options and more detailed explanations of the # device lines is also present in the ../../conf/NOTES and NOTES files. # If you are in doubt as to the purpose or necessity of a line, check first # in NOTES. # # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z jkim $ cpu I686_CPU ident POINT07 options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # To make an SMP kernel, the next two lines are needed options SMP # Symmetric MultiProcessor Kernel device apic # I/O APIC # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA controllers device ahci # AHCI-compatible SATA controllers device ata # Legacy ATA/SATA controllers options ATA_CAM # Handle legacy controllers with CAM options ATA_STATIC_ID # Static device numbering device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA device siis # SiliconImage SiI3124/SiI3132/SiI3531 SATA # ATA/SCSI peripherals device scbus # SCSI bus (required for ATA/SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct ATA/SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver # syscons is the default console driver, resembling an SCO console device sc options SC_PIXEL_MODE # add support for the raster text mode device agp # support several AGP chipsets # Power management support (see NOTES for more options) #device apm # Add suspend/resume support for the i8254. device pmtimer # Serial (COM) ports device uart # Generic UART driver # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer #device plip # TCP/IP over parallel device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to sio, uart and/or ppc drivers): #device puc # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family device ixgbe # Intel PRO/10GbE Ethernet Card # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support #device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter options IPFIREWALL options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_FORWARD options DUMMYNET options IPI_PREEMPTION # netgraph(4). Enable the base netgraph code with the NETGRAPH option. # Individual node types can be enabled with the corresponding option # listed below; however, this is not strictly necessary as netgraph # will automatically load the corresponding KLD module if the node type # is not already compiled into the kernel. Each type below has a # corresponding man page, e.g., ng_async(8). options NETGRAPH # netgraph(4) system options NETGRAPH_CAR options NETGRAPH_IFACE options NETGRAPH_MPPC_ENCRYPTION options NETGRAPH_PPP options NETGRAPH_PPPOE options NETGRAPH_PPTPGRE options NETGRAPH_SOCKET options NETGRAPH_TCPMSS options NETGRAPH_ETHER options NETGRAPH_TEE options NETGRAPH_VJC options NETGRAPH_BPF options NETGRAPH_KSOCKET options NETGRAPH_IPFW dmesg.boot -------------------------- Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011 root@point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64 CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3bd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 2147483648 (2048 MB) avail memory = 2056187904 (1960 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 ix0: port 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 at device 0.0 on pci1 ix0: Using MSIX interrupts with 5 vectors ix0: Ethernet address: 00:25:90:3c:42:6a ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 ix1: port 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 at device 0.1 on pci1 ix1: Using MSIX interrupts with 5 vectors ix1: RX Descriptors exceed system mbuf max, using default instead! ix1: Ethernet address: 00:25:90:3c:42:6b ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 pci0: at device 26.0 (no driver attached) pci0: at device 26.1 (no driver attached) pci0: at device 26.2 (no driver attached) pci0: at device 26.7 (no driver attached) pcib2: irq 16 at device 28.0 on pci0 pci5: on pcib2 pcib3: irq 16 at device 28.4 on pci0 pci13: on pcib3 em0: port 0x3000-0x301f mem 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13 em0: Using an MSI interrupt em0: Ethernet address: 00:30:48:d2:12:ac pcib4: irq 17 at device 28.5 on pci0 pci15: on pcib4 em1: port 0x4000-0x401f mem 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15 em1: Using an MSI interrupt em1: Ethernet address: 00:30:48:d2:12:ad pci0: at device 29.0 (no driver attached) pci0: at device 29.1 (no driver attached) pci0: at device 29.2 (no driver attached) pci0: at device 29.7 (no driver attached) pcib5: at device 30.0 on pci0 pci17: on pcib5 vgapci0: port 0x5000-0x507f mem 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at device 31.2 on pci0 ata0: on atapci0 ata1: on atapci0 pci0: at device 31.3 (no driver attached) atapci1: port 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c20-0x1c2f irq 18 at device 31.5 on pci0 ata2: on atapci1 ata3: on atapci1 pci0: at device 31.6 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppbus0: on ppc0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 Timecounters tick every 1.000 msec ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding enabled, default to accept, logging disabled DUMMYNET 0 with IPv6 initialized (100409) load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched PRIO loaded ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad0 SMP: AP CPU #3 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 9351750 Hz quality 1000 Trying to mount root from ufs:/dev/ada0p2 [rw]... sysctl.conf -------------- net.inet.ip.fastforwarding=1 net.isr.direct=0 net.isr.direct_force=0 net.inet.ip.fw.one_pass=1 hw.intr_storm_threshold=9000 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 dev.ix.0.rx_processing_limit=4096 dev.ix.1.rx_processing_limit=4096 net.inet.ip.intr_queue_maxlen=256 dev.ix.0.flow_control=0 dev.ix.1.flow_control=0 vmstat -i ------------- # vmstat -i interrupt total rate irq1: atkbd0 1 0 irq6: fdc0 1 0 irq14: ata0 80855 9 irq15: ata1 1 0 cpu0:timer 9694291 1122 irq256: ix0:que 0 54169187 6273 irq257: ix0:que 1 19307922 2236 irq258: ix0:que 2 24265573 2810 irq259: ix0:que 3 25845865 2993 irq260: ix0:link 2 0 irq261: ix1:que 0 63210220 7321 irq262: ix1:que 1 265895 30 irq263: ix1:que 2 42114 4 irq264: ix1:que 3 98953 11 irq265: ix1:link 5 0 cpu3:timer 7165968 829 cpu1:timer 7386880 855 cpu2:timer 7307920 846 Total 218841653 25346 netstat ------------ # netstat -I ix0 -w1 input (ix0) output packets errs idrops bytes packets errs bytes colls 26592 0 0 30220498 20757 0 11490639 0 31232 0 0 34001150 24699 0 14554631 0 27785 0 0 29672129 23699 0 16154542 0 25525 0 0 27450183 22868 0 16180216 0 ^C # netstat -I ix1 -w1 input (ix1) output packets errs idrops bytes packets errs bytes colls 22398 0 0 17232635 23781 0 24718205 0 23671 0 0 17608589 25859 0 28970777 0 24096 0 0 17278557 27188 0 30804019 0 22043 0 0 16271250 24116 0 26238423 0 ^C top -SCHP ----------------- last pid: 74628; load averages: 1.26, 1.01, 0.89 up 0+02:25:14 13:22:11 110 processes: 9 running, 79 sleeping, 22 waiting CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% idle CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% idle CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% idle CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% idle Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% idle{idle: cpu2} 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% idle{idle: cpu1} 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% idle{idle: cpu3} 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% idle{idle: cpu0} 11 root -92 - 0K 416K CPU0 0 30:52 36.77% intr{irq261: ix1:que } 11 root -92 - 0K 416K CPU3 3 8:03 11.77% intr{irq259: ix0:que } 11 root -92 - 0K 416K CPU1 1 5:37 9.86% intr{irq257: ix0:que } 11 root -92 - 0K 416K WAIT 2 8:14 9.28% intr{irq258: ix0:que } 11 root -92 - 0K 416K RUN 0 7:07 5.18% intr{irq256: ix0:que } 0 root -92 0 0K 352K - 2 1:12 1.17% kernel{ix1 que} 12 root -16 - 0K 64K sleep 0 1:02 1.07% ng_queue{ng_queue1} 12 root -16 - 0K 64K sleep 0 1:02 0.88% ng_queue{ng_queue0} 12 root -16 - 0K 64K sleep 2 1:02 0.78% ng_queue{ng_queue3} 12 root -16 - 0K 64K sleep 0 1:02 0.78% ng_queue{ng_queue2} 0 root -92 0 0K 352K - 3 0:58 0.39% kernel{ix0 que} 11 root -60 - 0K 416K WAIT 3 5:21 0.00% intr{swi4: clock} 0 root -16 0 0K 352K sched 1 0:54 0.00% kernel{swapper} 14320 root 20 0 78528K 32900K select 1 0:52 0.00% mpd5{mpd5} 0 root -92 0 0K 352K - 0 0:28 0.00% kernel{ix0 que} 0 root -92 0 0K 352K - 2 0:25 0.00% kernel{dummynet} 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra 0 root -92 0 0K 352K - 2 0:17 0.00% kernel{ix0 que} 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow 0 root -92 0 0K 352K - 3 0:07 0.00% kernel{ix0 que} 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer 0 root -92 0 0K 352K - 1 0:02 0.00% kernel{ix1 linkq} 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd 11 root -92 - 0K 416K WAIT 1 0:01 0.00% intr{irq262: ix1:que } 11 root -68 - 0K 416K WAIT 2 0:01 0.00% intr{swi2: cambio} 13 root -8 - 0K 48K - 3 0:01 0.00% geom{g_up} 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00% collectd{collectd} 11 root -88 - 0K 416K WAIT 3 0:01 0.00% intr{irq14: ata0} 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd 13 root -8 - 0K 48K - 0 0:01 0.00% geom{g_down} 11 root -92 - 0K 416K WAIT 3 0:01 0.00% intr{irq264: ix1:que } 15 root -16 - 0K 16K sdflus 3 0:00 0.00% softdepflush 11 root -60 - 0K 416K WAIT 2 0:00 0.00% intr{swi4: clock} pciconf -lvc --------------- ix0@pci0:1:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) cap 03[e0] = VPD ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002590ffff3c426a ecap 000e[150] = unknown 1 ecap 0010[160] = unknown 1 ix1@pci0:1:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) cap 03[e0] = VPD ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 002590ffff3c426a ecap 000e[150] = unknown 1 ecap 0010[160] = unknown 1 The question is: ----------------- Why there are so much interrupts? Why the card is using only one intr on second port (ix1)? I've got another server with 82598EB 10 Gigabit AT CX4 Network Connection and the situation on it is dramatically different: last pid: 42234; load averages: 2.14, 1.65, 1.55 up 119+05:58:58 13:26:01 98 processes: 8 running, 62 sleeping, 28 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% idle CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% idle CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: cpu0} 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: cpu2} 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: cpu3} 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: cpu1} 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259: ix0:que } 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257: ix0:que } 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258: ix0:que } 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256: ix0:que } 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264: ix1:que } 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262: ix1:que } 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263: ix1:que } 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261: ix1:que } vmstat -i ------------- r# vmstat -i interrupt total rate irq19: atapci0 1667110 0 cpu0: timer 3386721810 328 irq256: ix0:que 0 3395843376 329 irq257: ix0:que 1 2642665824 256 irq258: ix0:que 2 2838302235 275 irq259: ix0:que 3 2176207954 211 irq260: ix0:link 18 0 irq261: ix1:que 0 282359321 27 irq262: ix1:que 1 3989170496 387 irq263: ix1:que 2 375956573 36 irq264: ix1:que 3 3966352151 384 irq265: ix1:link 1 0 irq266: em0:rx 0 10283114 0 irq269: em1:rx 0 10283114 0 cpu3: timer 3386697130 328 cpu1: timer 3386709028 328 cpu2: timer 3386700908 328 Total 33235920163 3225 netstat ------------ # netstat -I ix0 -w1 input (ix0) output packets errs idrops bytes packets errs bytes colls 264572 0 0 252383894 253290 0 178845781 0 266323 0 0 254825280 251277 0 173769218 0 274462 0 0 265247306 260819 0 181113304 0 271142 0 0 263263325 253941 0 171694438 0 ^C # netstat -I ix1 -w1 input (ix1) output packets errs idrops bytes packets errs bytes colls 259427 0 0 183038665 275711 0 262429749 0 248177 0 0 171479966 264441 0 252993752 0 255271 0 0 185006000 266886 0 247494148 0 264543 0 0 190970875 275087 0 259555066 0 ^C Tuning on both systems are almost the same As You can see, 82598EB card produce about 20 times less interrupts at about 10 times more pps. Please help me to fix the problem... -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4931883.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 11:07:08 2011 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1740E1065674 for ; Mon, 24 Oct 2011 11:07:08 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 04C388FC1F for ; Mon, 24 Oct 2011 11:07:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9OB77rQ025367 for ; Mon, 24 Oct 2011 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9OB77pw025365 for freebsd-net@FreeBSD.org; Mon, 24 Oct 2011 11:07:07 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Oct 2011 11:07:07 GMT Message-Id: <201110241107.p9OB77pw025365@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 11:07:08 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/161908 net [netgraph] [patch] ng_vlan update for QinQ support o kern/161899 net [route] ntpd(8): Repeating RTM_MISS packets causing hi o kern/161381 net [re] RTL8169SC - re0: PHY write failed o kern/161277 net [em] [patch] BMC cannot receive IPMI traffic after loa o kern/160873 net [igb] igb(4) from HEAD fails to build on 7-STABLE o kern/160750 net Intel PRO/1000 connection breaks under load until rebo o kern/160693 net [gif] [em] Multicast packet are not passed from GIF0 t o kern/160420 net [msk] phy write timeout on HP 5310m o kern/160293 net [ieee80211] ppanic] kernel panic during network setup o kern/160206 net [gif] gifX stops working after a while (IPv6 tunnel) o kern/159817 net [udp] write UDPv4: No buffer space available (code=55) o kern/159795 net [tcp] excessive duplicate ACKs and TCP session freezes o kern/159629 net [ipsec] [panic] kernel panic with IPsec in transport m o kern/159621 net [tcp] [panic] panic: soabort: so_count o kern/159603 net [netinet] [patch] in_ifscrubprefix() - network route c o kern/159601 net [netinet] [patch] in_scrubprefix() - loopback route re o kern/159294 net [em] em watchdog timeouts o kern/159203 net [wpi] Intel 3945ABG Wireless LAN not support IBSS o kern/158930 net [bpf] BPF element leak in ifp->bpf_if->bif_dlist o kern/158726 net [ip6] [patch] ICMPv6 Router Announcement flooding limi o kern/158694 net [ix] [lagg] ix0 is not working within lagg(4) o kern/158665 net [ip6] [panic] kernel pagefault in in6_setscope() o kern/158635 net [em] TSO breaks BPF packet captures with em driver f kern/157802 net [dummynet] [panic] kernel panic in dummynet o kern/157785 net amd64 + jail + ipfw + natd = very slow outbound traffi o kern/157429 net [re] Realtek RTL8169 doesn't work with re(4) o kern/157418 net [em] em driver lockup during boot on Supermicro X9SCM- o kern/157410 net [ip6] IPv6 Router Advertisements Cause Excessive CPU U o kern/157287 net [re] [panic] INVARIANTS panic (Memory modified after f o kern/157209 net [ip6] [patch] locking error in rip6_input() (sys/netin o kern/157200 net [network.subr] [patch] stf(4) can not communicate betw o kern/157182 net [lagg] lagg interface not working together with epair o kern/156877 net [dummynet] [panic] dummynet move_pkt() null ptr derefe o kern/156667 net [em] em0 fails to init on CURRENT after March 17 o kern/156408 net [vlan] Routing failure when using VLANs vs. Physical e o kern/156328 net [icmp]: host can ping other subnet but no have IP from o kern/156317 net [ip6] Wrong order of IPv6 NS DAD/MLD Report o kern/156283 net [ip6] [patch] nd6_ns_input - rtalloc_mpath does not re o kern/156279 net [if_bridge][divert][ipfw] unable to correctly re-injec o kern/156226 net [lagg]: failover does not announce the failover to swi o kern/156030 net [ip6] [panic] Crash in nd6_dad_start() due to null ptr o kern/155772 net ifconfig(8): ioctl (SIOCAIFADDR): File exists on direc o kern/155680 net [multicast] problems with multicast s kern/155642 net [request] Add driver for Realtek RTL8191SE/RTL8192SE W o kern/155604 net [flowtable] Flowtable excessively caches dest MAC addr o kern/155597 net [panic] Kernel panics with "sbdrop" message o kern/155585 net [tcp] [panic] tcp_output tcp_mtudisc loop until kernel o kern/155420 net [vlan] adding vlan break existent vlan o bin/155365 net [patch] routed(8): if.c in routed fails to compile if o kern/155177 net [route] [panic] Panic when inject routes in kernel o kern/155030 net [igb] igb(4) DEVICE_POLLING does not work with carp(4) o kern/155010 net [msk] ntfs-3g via iscsi using msk driver cause kernel o kern/155004 net [bce] [panic] kernel panic in bce0 driver o kern/154943 net [gif] ifconfig gifX create on existing gifX clears IP s kern/154851 net [request]: Port brcm80211 driver from Linux to FreeBSD o kern/154850 net [netgraph] [patch] ng_ether fails to name nodes when t o kern/154679 net [em] Fatal trap 12: "em1 taskq" only at startup (8.1-R o kern/154600 net [tcp] [panic] Random kernel panics on tcp_output o kern/154557 net [tcp] Freeze tcp-session of the clients, if in the gat o kern/154443 net [if_bridge] Kernel module bridgestp.ko missing after u o kern/154286 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/154255 net [nfs] NFS not responding o kern/154214 net [stf] [panic] Panic when creating stf interface o kern/154185 net race condition in mb_dupcl o kern/154169 net [multicast] [ip6] Node Information Query multicast add o kern/154134 net [ip6] stuck kernel state in LISTEN on ipv6 daemon whic o kern/154091 net [netgraph] [panic] netgraph, unaligned mbuf? o conf/154062 net [vlan] [patch] change to way of auto-generatation of v o kern/153937 net [ral] ralink panics the system (amd64 freeBSDD 8.X) wh o kern/153936 net [ixgbe] [patch] MPRC workaround incorrectly applied to o kern/153816 net [ixgbe] ixgbe doesn't work properly with the Intel 10g o kern/153772 net [ixgbe] [patch] sysctls reference wrong XON/XOFF varia o kern/153497 net [netgraph] netgraph panic due to race conditions o kern/153454 net [patch] [wlan] [urtw] Support ad-hoc and hostap modes o kern/153308 net [em] em interface use 100% cpu o kern/153244 net [em] em(4) fails to send UDP to port 0xffff o kern/152893 net [netgraph] [panic] 8.2-PRERELEASE panic in netgraph o kern/152853 net [em] tftpd (and likely other udp traffic) fails over e o kern/152828 net [em] poor performance on 8.1, 8.2-PRE o kern/152569 net [net]: Multiple ppp connections and routing table prob o kern/152235 net [arp] Permanent local ARP entries are not properly upd o kern/152141 net [vlan] [patch] encapsulate vlan in ng_ether before out o kern/152036 net [libc] getifaddrs(3) returns truncated sockaddrs for n o kern/151690 net [ep] network connectivity won't work until dhclient is o kern/151681 net [nfs] NFS mount via IPv6 leads to hang on client with o kern/151593 net [igb] [panic] Kernel panic when bringing up igb networ o kern/150920 net [ixgbe][igb] Panic when packets are dropped with heade o kern/150557 net [igb] igb0: Watchdog timeout -- resetting o kern/150251 net [patch] [ixgbe] Late cable insertion broken o kern/150249 net [ixgbe] Media type detection broken o bin/150224 net ppp(8) does not reassign static IP after kill -KILL co f kern/149969 net [wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149937 net [ipfilter] [patch] kernel panic in ipfilter IP fragmen o kern/149643 net [rum] device not sending proper beacon frames in ap mo o kern/149609 net [panic] reboot after adding second default route o kern/149117 net [inet] [patch] in_pcbbind: redundant test o kern/149086 net [multicast] Generic multicast join failure in 8.1 o kern/148018 net [flowtable] flowtable crashes on ia64 o kern/147912 net [boot] FreeBSD 8 Beta won't boot on Thinkpad i1300 11 o kern/147894 net [ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147155 net [ip6] setfb not work with ipv6 o kern/146845 net [libc] close(2) returns error 54 (connection reset by f kern/146792 net [flowtable] flowcleaner 100% cpu's core load o kern/146719 net [pf] [panic] PF or dumynet kernel panic o kern/146534 net [icmp6] wrong source address in echo reply o kern/146427 net [mwl] Additional virtual access points don't work on m o kern/146426 net [mwl] 802.11n rates not possible on mwl o kern/146425 net [mwl] mwl dropping all packets during and after high u f kern/146394 net [vlan] IP source address for outgoing connections o bin/146377 net [ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net [vlan] wrong destination MAC address o kern/146165 net [wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net [ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net [panic] mpd + CoA = kernel panic o bin/145934 net [patch] add count option to netstat(1) o kern/145825 net [panic] panic: soabort: so_count o kern/145728 net [lagg] Stops working lagg between two servers. p kern/145600 net TCP/ECN behaves different to CE/CWR than ns2 reference f kern/144917 net [flowtable] [panic] flowtable crashes system [regressi o kern/144882 net MacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net [if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net [rc.d] async dhclient breaks stuff for too many people o kern/144616 net [nat] [panic] ip_nat panic FreeBSD 7.2 o kern/144572 net [carp] CARP preemption mode traffic partially goes to f kern/144315 net [ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/144231 net bind/connect/sendto too strict about sockaddr length o kern/143846 net [gif] bringing gif3 tunnel down causes gif0 tunnel to s kern/143673 net [stf] [request] there should be a way to support multi s kern/143666 net [ip6] [request] PMTU black hole detection not implemen o kern/143622 net [pfil] [patch] unlock pfil lock while calling firewall o kern/143593 net [ipsec] When using IPSec, tcpdump doesn't show outgoin o kern/143591 net [ral] RT2561C-based DLink card (DWL-510) fails to work o kern/143208 net [ipsec] [gif] IPSec over gif interface not working o kern/143034 net [panic] system reboots itself in tcp code [regression] o kern/142877 net [hang] network-related repeatable 8.0-STABLE hard hang o kern/142774 net Problem with outgoing connections on interface with mu o kern/142772 net [libc] lla_lookup: new lle malloc failed o kern/142018 net [iwi] [patch] Possibly wrong interpretation of beacon- o kern/141861 net [wi] data garbled with WEP and wi(4) with Prism 2.5 f kern/141741 net Etherlink III NIC won't work after upgrade to FBSD 8, o kern/140742 net rum(4) Two asus-WL167G adapters cannot talk to each ot o kern/140682 net [netgraph] [panic] random panic in netgraph o kern/140634 net [vlan] destroying if_lagg interface with if_vlan membe o kern/140619 net [ifnet] [patch] refine obsolete if_var.h comments desc o kern/140346 net [wlan] High bandwidth use causes loss of wlan connecti o kern/140142 net [ip6] [panic] FreeBSD 7.2-amd64 panic w/IPv6 o kern/140066 net [bwi] install report for 8.0 RC 2 (multiple problems) o kern/139565 net [ipfilter] ipfilter ioctl SIOCDELST broken o kern/139387 net [ipsec] Wrong lenth of PF_KEY messages in promiscuous o bin/139346 net [patch] arp(8) add option to remove static entries lis o kern/139268 net [if_bridge] [patch] allow if_bridge to forward just VL p kern/139204 net [arp] DHCP server replies rejected, ARP entry lost bef o kern/139117 net [lagg] + wlan boot timing (EBUSY) o kern/139058 net [ipfilter] mbuf cluster leak on FreeBSD 7.2 o kern/138850 net [dummynet] dummynet doesn't work correctly on a bridge o kern/138782 net [panic] sbflush_internal: cc 0 || mb 0xffffff004127b00 o kern/138688 net [rum] possibly broken on 8 Beta 4 amd64: able to wpa a o kern/138678 net [lo] FreeBSD does not assign linklocal address to loop o kern/138620 net [lagg] [patch] lagg port bpf-writes blocked o kern/138407 net [gre] gre(4) interface does not come up after reboot o kern/138332 net [tun] [lor] ifconfig tun0 destroy causes LOR if_adata/ o kern/138266 net [panic] kernel panic when udp benchmark test used as r o kern/138177 net [ipfilter] FreeBSD crashing repeatedly in ip_nat.c:257 f kern/138029 net [bpf] [panic] periodically kernel panic and reboot o kern/137881 net [netgraph] [panic] ng_pppoe fatal trap 12 p bin/137841 net [patch] wpa_supplicant(8) cannot verify SHA256 signed p kern/137776 net [rum] panic in rum(4) driver on 8.0-BETA2 o bin/137641 net ifconfig(8): various problems with "vlan_device.vlan_i o kern/137392 net [ip] [panic] crash in ip_nat.c line 2577 o kern/137372 net [ral] FreeBSD doesn't support wireless interface from o kern/137089 net [lagg] lagg falsely triggers IPv6 duplicate address de o bin/136994 net [patch] ifconfig(8) print carp mac address o kern/136911 net [netgraph] [panic] system panic on kldload ng_bpf.ko t o bin/136661 net [patch] ndp(8) ignores -f option o kern/136618 net [pf][stf] panic on cloning interface without unit numb o kern/135502 net [periodic] Warning message raised by rtfree function i o kern/134583 net [hang] Machine with jail freezes after random amount o o kern/134531 net [route] [panic] kernel crash related to routes/zebra o kern/134157 net [dummynet] dummynet loads cpu for 100% and make a syst o kern/133969 net [dummynet] [panic] Fatal trap 12: page fault while in o kern/133968 net [dummynet] [panic] dummynet kernel panic o kern/133736 net [udp] ip_id not protected ... o kern/133595 net [panic] Kernel Panic at pcpu.h:195 o kern/133572 net [ppp] [hang] incoming PPTP connection hangs the system o kern/133490 net [bpf] [panic] 'kmem_map too small' panic on Dell r900 o kern/133235 net [netinet] [patch] Process SIOCDLIFADDR command incorre f kern/133213 net arp and sshd errors on 7.1-PRERELEASE o kern/133060 net [ipsec] [pfsync] [panic] Kernel panic with ipsec + pfs o kern/132889 net [ndis] [panic] NDIS kernel crash on load BCM4321 AGN d o conf/132851 net [patch] rc.conf(5): allow to setfib(1) for service run o kern/132734 net [ifmib] [panic] panic in net/if_mib.c o kern/132705 net [libwrap] [patch] libwrap - infinite loop if hosts.all o kern/132672 net [ndis] [panic] ndis with rt2860.sys causes kernel pani o kern/132554 net [ipl] There is no ippool start script/ipfilter magic t o kern/132354 net [nat] Getting some packages to ipnat(8) causes crash o kern/132277 net [crypto] [ipsec] poor performance using cryptodevice f o kern/131781 net [ndis] ndis keeps dropping the link o kern/131776 net [wi] driver fails to init o kern/131753 net [altq] [panic] kernel panic in hfsc_dequeue o kern/131601 net [ipfilter] [panic] 7-STABLE panic in nat_finalise (tcp o bin/131567 net [socket] [patch] Update for regression/sockets/unix_cm o bin/131365 net route(8): route add changes interpretation of network f kern/130820 net [ndis] wpa_supplicant(8) returns 'no space on device' o kern/130628 net [nfs] NFS / rpc.lockd deadlock on 7.1-R o conf/130555 net [rc.d] [patch] No good way to set ipfilter variables a o kern/130525 net [ndis] [panic] 64 bit ar5008 ndisgen-erated driver cau o kern/130311 net [wlan_xauth] [panic] hostapd restart causing kernel pa o kern/130109 net [ipfw] Can not set fib for packets originated from loc f kern/130059 net [panic] Leaking 50k mbufs/hour f kern/129719 net [nfs] [panic] Panic during shutdown, tcp_ctloutput: in o kern/129517 net [ipsec] [panic] double fault / stack overflow f kern/129508 net [carp] [panic] Kernel panic with EtherIP (may be relat o kern/129219 net [ppp] Kernel panic when using kernel mode ppp o kern/129197 net [panic] 7.0 IP stack related panic o bin/128954 net ifconfig(8) deletes valid routes o bin/128602 net [an] wpa_supplicant(8) crashes with an(4) o kern/128448 net [nfs] 6.4-RC1 Boot Fails if NFS Hostname cannot be res o bin/128295 net [patch] ifconfig(8) does not print TOE4 or TOE6 capabi o bin/128001 net wpa_supplicant(8), wlan(4), and wi(4) issues o kern/127826 net [iwi] iwi0 driver has reduced performance and connecti o kern/127815 net [gif] [patch] if_gif does not set vlan attributes from o kern/127724 net [rtalloc] rtfree: 0xc5a8f870 has 1 refs f bin/127719 net [arp] arp: Segmentation fault (core dumped) f kern/127528 net [icmp]: icmp socket receives icmp replies not owned by p kern/127360 net [socket] TOE socket options missing from sosetopt() o bin/127192 net routed(8) removes the secondary alias IP of interface f kern/127145 net [wi]: prism (wi) driver crash at bigger traffic o kern/126895 net [patch] [ral] Add antenna selection (marked as TBD) o kern/126874 net [vlan]: Zebra problem if ifconfig vlanX destroy o kern/126695 net rtfree messages and network disruption upon use of if_ o kern/126339 net [ipw] ipw driver drops the connection o kern/126075 net [inet] [patch] internet control accesses beyond end of o bin/125922 net [patch] Deadlock in arp(8) o kern/125920 net [arp] Kernel Routing Table loses Ethernet Link status o kern/125845 net [netinet] [patch] tcp_lro_rx() should make use of hard o kern/125258 net [socket] socket's SO_REUSEADDR option does not work o kern/125239 net [gre] kernel crash when using gre o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124160 net [libc] connect(2) function loops indefinitely o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/123892 net [tap] [patch] No buffer space available o kern/123890 net [ppp] [panic] crash & reboot on work with PPP low-spee o kern/123858 net [stf] [patch] stf not usable behind a NAT o kern/123796 net [ipf] FreeBSD 6.1+VPN+ipnat+ipf: port mapping does not o kern/123758 net [panic] panic while restarting net/freenet6 o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 f kern/123045 net [ng_mppc] ng_mppc_decompress - disabling node o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices f kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge o kern/122685 net It is not visible passing packets in tcpdump(1) o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup ieee o bin/121895 net [patch] rtsol(8)/rtsold(8) doesn't handle managed netw s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121443 net [gif] [lor] icmp6_input/nd6_lookup o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o bin/121359 net [patch] [security] ppp(8): fix local stack overflow in o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/120966 net [rum] kernel panic with if_rum and WPA encryption o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120266 net [udp] [panic] gnugk causes kernel panic when closing U o bin/120060 net routed(8) deletes link-level routes in the presence of o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119432 net [arp] route add -host -iface causes arp e o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/118727 net [netgraph] [patch] [request] add new ng_pf module o kern/117423 net [vlan] Duplicate IP on different interfaces o bin/117339 net [patch] route(8): loading routing management commands o kern/117271 net [tap] OpenVPN TAP uses 99% CPU on releng_6 when if_tap o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/115019 net [netgraph] ng_ether upper hook packet flow stops on ad o kern/115002 net [wi] if_wi timeout. failed allocation (busy bit). ifco o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o kern/113432 net [ucom] WARNING: attempt to net_add_domain(netgraph) af o kern/112722 net [ipsec] [udp] IP v4 udp fragmented packet reject o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/111537 net [inet6] [patch] ip6_input() treats mbuf cluster wrong o kern/111457 net [ral] ral(4) freeze o kern/110284 net [if_ethersubr] Invalid Assumption in SIOCSIFADDR in et o kern/110249 net [kernel] [regression] [patch] setsockopt() error regre o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] o kern/107944 net [wi] [patch] Forget to unlock mutex-locks o conf/107035 net [patch] bridge(8): bridge interface given in rc.conf n o kern/106444 net [netgraph] [panic] Kernel Panic on Binding to an ip to o kern/106438 net [ipf] ipfilter: keep state does not seem to allow repl o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/105945 net Address can disappear from network interface s kern/105943 net Network stack may modify read-only mbuf chain copies o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] o kern/104851 net [inet6] [patch] On link routes not configured when usi o kern/104751 net [netgraph] kernel panic, when getting info about my tr o kern/103191 net Unpredictable reboot o kern/103135 net [ipsec] ipsec with ipfw divert (not NAT) encodes a pac o kern/102540 net [netgraph] [patch] supporting vlan(4) by ng_fec(4) o conf/102502 net [netgraph] [patch] ifconfig name does't rename netgrap o kern/102035 net [plip] plip networking disables parallel port printing o kern/101948 net [ipf] [panic] Kernel Panic Trap No 12 Page Fault - cau o kern/100709 net [libc] getaddrinfo(3) should return TTL info o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/98978 net [ipf] [patch] ipfilter drops OOW packets under 6.1-Rel o kern/98597 net [inet6] Bug in FreeBSD 6.1 IPv6 link-local DAD procedu o bin/98218 net wpa_supplicant(8) blacklist not working o kern/97306 net [netgraph] NG_L2TP locks after connection with failed o conf/97014 net [gif] gifconfig_gif? in rc.conf does not recognize IPv f kern/96268 net [socket] TCP socket performance drops by 3000% if pack o kern/95519 net [ral] ral0 could not map mbuf o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/95267 net packet drops periodically appear f kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/93019 net [ppp] ppp and tunX problems: no traffic after restarti o kern/92880 net [libc] [patch] almost rewritten inet_network(3) functi s kern/92279 net [dc] Core faults everytime I reboot, possible NIC issu o kern/91859 net [ndis] if_ndis does not work with Asus WL-138 s kern/91777 net [ipf] [patch] wrong behaviour with skip rule inside an o kern/91364 net [ral] [wep] WF-511 RT2500 Card PCI and WEP o kern/91311 net [aue] aue interface hanging s kern/90086 net [hang] 5.4p8 on supermicro P8SCT hangs during boot if o kern/87521 net [ipf] [panic] using ipfilter "auth" keyword leads to k o kern/87421 net [netgraph] [panic]: ng_ether + ng_eiface + if_bridge s kern/86920 net [ndis] ifconfig: SIOCS80211: Invalid argument [regress o kern/86871 net [tcp] [patch] allocation logic for PCBs in TIME_WAIT s o kern/86427 net [lor] Deadlock with FASTIPSEC and nat o kern/86103 net [ipf] Illegal NAT Traversal in IPFilter o kern/85780 net 'panic: bogus refcnt 0' in routing/ipv6 o bin/85445 net ifconfig(8): deprecated keyword to ifconfig inoperativ p kern/85320 net [gre] [patch] possible depletion of kernel stack in ip o bin/82975 net route change does not parse classfull network as given o kern/82881 net [netgraph] [panic] ng_fec(4) causes kernel panic after o kern/82468 net Using 64MB tcp send/recv buffers, trafficflow stops, i o bin/82185 net [patch] ndp(8) can delete the incorrect entry o kern/81095 net IPsec connection stops working if associated network i o kern/78968 net FreeBSD freezes on mbufs exhaustion (network interface o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o kern/77341 net [ip6] problems with IPV6 implementation o kern/77273 net [ipf] ipfilter breaks ipv6 statefull filtering on 5.3 s kern/77195 net [ipf] [patch] ipfilter ioctl SIOCGNATL does not match o kern/75873 net Usability problem with non-RFC-compliant IP spoof prot s kern/75407 net [an] an(4): no carrier after short time a kern/71474 net [route] route lookup does not skip interfaces marked d o kern/71469 net default route to internet magically disappears with mu o kern/70904 net [ipf] ipfilter ipnat problem with h323 proxy support o kern/68889 net [panic] m_copym, length > size of mbuf chain o kern/66225 net [netgraph] [patch] extend ng_eiface(4) control message o kern/65616 net IPSEC can't detunnel GRE packets after real ESP encryp s kern/60293 net [patch] FreeBSD arp poison patch a kern/56233 net IPsec tunnel (ESP) over IPv6: MTU computation is wrong s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr s kern/39937 net ipstealth issue a kern/38554 net [patch] changing interface ipaddress doesn't seem to w o kern/34665 net [ipf] [hang] ipfilter rcmd proxy "hangs". o kern/31940 net ip queue length too short for >500kpps o kern/31647 net [libc] socket calls can return undocumented EINVAL o kern/30186 net [libc] getaddrinfo(3) does not handle incorrect servna o kern/27474 net [ipf] [ppp] Interactive use of user PPP and ipfilter c f kern/24959 net [patch] proper TCP_NOPUSH/TCP_CORK compatibility o conf/23063 net [arp] [patch] for static ARP tables in rc.network o kern/21998 net [socket] [patch] ident only for outgoing connections o kern/5877 net [socket] sb_cc counts control data as well as data dat 379 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 12:27:54 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37AC2106564A; Mon, 24 Oct 2011 12:27:54 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0C6F98FC0C; Mon, 24 Oct 2011 12:27:54 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id B7ED446B0C; Mon, 24 Oct 2011 08:27:53 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3AFDD8A037; Mon, 24 Oct 2011 08:27:53 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Date: Mon, 24 Oct 2011 08:14:22 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110617; KDE/4.5.5; amd64; ; ) References: <20111022084931.GD1697@garage.freebsd.pl> <20111023084445.GB50300@deviant.kiev.zoral.com.ua> <20111023155827.GH1697@garage.freebsd.pl> In-Reply-To: <20111023155827.GH1697@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201110240814.22368.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Mon, 24 Oct 2011 08:27:53 -0400 (EDT) Cc: Kostik Belousov , Lawrence Stewart , Andre Oppermann , Pawel Jakub Dawidek , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 12:27:54 -0000 On Sunday, October 23, 2011 11:58:28 am Pawel Jakub Dawidek wrote: > On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: > > On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: > > > My suggestion would be that if we won't be able to fix it before 9.0, > > > we should turn this assertion off, as the system seems to be able to > > > recover. > > > > Shipped kernels have all assertions turned off. > > Yes, I'm aware of that, but many people compile their production kernels > with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. > corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing > it into a printf, so it will be visible. No, the kernel is corrupting things in other places when this is true, so if you are running with INVARIANTS, we want to know about it. Specifically, in several places in TCP we assume that rcv_adv >= rcv_nxt, and depend on being able to do 'rcv_adv - rcv_nxt'. In this case, it looks like the difference is consistently less than one frame. I suspect the other end of the connection is sending just beyond the end of the advertised window (it probably assumes it is better to send a full frame if it has that much pending data even though part of it is beyond the window edge vs sending a truncated packet that just fills the window) and that that frame is accepted ok in the header prediction case and it's ACK is delayed, but the next packet to arrive then trips over this assumption. Since 'win' is guaranteed to be non-negative and we explicitly cast 'rcv_adv - rcv_nxt' to (int) in the following line that the assert is checking for: tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); I think we already handle this case ok and perhaps the assertion can just be removed? Not sure if others feel that it warrants a comment to note that this is the case being handled. Also, I'm not sure if this case can "leak" into the timewait code? If so, we will need to fix this case: /* * Recover last window size sent. */ KASSERT(SEQ_GEQ(tp->rcv_adv, tp->rcv_nxt), ("tcp_twstart negative window: tp %p rcv_nxt %u rcv_adv %u", tp, tp->rcv_nxt, tp->rcv_adv)); tw->last_win = (tp->rcv_adv - tp->rcv_nxt) >> tp->rcv_scale; So that it sets last_win to 0 instead of some really huge value. -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 14:10:36 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25354106566C for ; Mon, 24 Oct 2011 14:10:36 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id D9BC18FC14 for ; Mon, 24 Oct 2011 14:10:34 +0000 (UTC) Received: by yxt33 with SMTP id 33so2563993yxt.13 for ; Mon, 24 Oct 2011 07:10:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:date:subject:to :message-id:mime-version:x-mailer; bh=B/4w5b9F8lyqBXVzBwIa6MIxOHtvETxMWZU0RdswK8k=; b=OaJUymdyvSCchuIsujkC0coEeNRUegNP27zauDHsEq23wCHWISos0SpssqDZD3umaD Cz/fBytA9/sOJHryx+XQi77+zKYsK4tRZiNEUt15PLTplFnZGtyxplcan1WqMPfWTMFx xIrMaXZ5W7bYkT97pSKsABZzRpks7bxnwhZnI= Received: by 10.223.61.211 with SMTP id u19mr43495532fah.29.1319463837759; Mon, 24 Oct 2011 06:43:57 -0700 (PDT) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id 5sm13182033fad.9.2011.10.24.06.43.56 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 24 Oct 2011 06:43:56 -0700 (PDT) From: Nikolay Denev Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Mon, 24 Oct 2011 16:43:57 +0300 To: freebsd-net@freebsd.org Message-Id: Mime-Version: 1.0 (Apple Message framework v1251.1) X-Mailer: Apple Mail (2.1251.1) Subject: Possible sge(4)/atphy(4) regression on RELENG_9? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 14:10:36 -0000 Hello, I've recently upgraded a box running RELENG_8 to RELENG_9 and = immediately I noticed much slower network connection. Running iperf shows about 20-30Mbits which was almost full GigE = (~900Mbits) speed before. I'm noticing interface errors : [16:37]ndenev@nas:~% netstat -I sge0 Name Mtu Network Address Ipkts Ierrs Idrop = Opkts Oerrs Coll sge0 1500 00:0a:e4:86:62:fa 76114295 42197 0 = 103559806 10324 0 sge0 1500 10.0.0.0 nas 76109575 - - = 119109557 - - Both the switch and the card show 1000 full-duplex. I've tried playing with rxcsum,txcsum,vlanhwtag,tso but disabling even = all of them do not change anything. I've tried different switch port and changed the cable. Here is devinfo for my hardware : sge0 pnpinfo vendor=3D0x1039 device=3D0x0191 subvendor=3D0x103c = subdevice=3D0x2a70 class=3D0x020000 atphy0 pnpinfo oui=3D0xc82e model=3D0x1 rev=3D0x6 at phyno=3D0 Of course all of this can mean hardware problem, I just want to ask if = somebody is seeing something similar, since there are quite a lot minibus related changes as far as I can see. I'll boot RELENG_8 again tomorrow and do a quick test again to verify = that this is not a hardware issue. Regards, Nikolay From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 16:56:05 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61E1C106564A for ; Mon, 24 Oct 2011 16:56:05 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id F11958FC08 for ; Mon, 24 Oct 2011 16:56:04 +0000 (UTC) Received: by wyi40 with SMTP id 40so8256716wyi.13 for ; Mon, 24 Oct 2011 09:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=T9gFcCNI9yqBUR2DphY8hBOHcx7P6bBNlT5SlGyGU2Q=; b=nO30W2PHpO3TudnIBFGnq3SlunFyFSMS2TE7Gsq7u1aB7JQtEYHAQsTR7HvPnO6RtG rBkupV96I05K7M7iMjmkw0W8YTKd+RcF5FmseZ/RQfImA4D9nlnlmbd5pyxCJCqiFu5r nJeuk4J8NTuZ5sD304Jbx9wFmQVFpHziZ+jjc= MIME-Version: 1.0 Received: by 10.216.132.194 with SMTP id o44mr3301849wei.101.1319475363827; Mon, 24 Oct 2011 09:56:03 -0700 (PDT) Received: by 10.180.8.34 with HTTP; Mon, 24 Oct 2011 09:56:03 -0700 (PDT) In-Reply-To: <20111023213133.GC1536@verio.net> References: <33528.1319361996@tristatelogic.com> <20111023213133.GC1536@verio.net> Date: Mon, 24 Oct 2011 12:56:03 -0400 Message-ID: From: Ryan Stone To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: IPFW shows me Strangeness in fresh 8.2-RELEASE system X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 16:56:05 -0000 On Sun, Oct 23, 2011 at 5:31 PM, David DeSimone wrote: > Ethernet cards filter their traffic based on MAC address, not based on > IP address. > > Use tcpdump -e to examine the destination MAC of the packets you are > receiving, in order to determine whether you should receive them. tcpdump -pe, lest he confuse the issue by having tcpdump put the interface in promiscuous mode. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 16:59:45 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5AC6106566C for ; Mon, 24 Oct 2011 16:59:45 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id B42ED8FC08 for ; Mon, 24 Oct 2011 16:59:44 +0000 (UTC) Received: by wwi18 with SMTP id 18so9026467wwi.31 for ; Mon, 24 Oct 2011 09:59:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qKpgJWszW4uhCnB3MvgF7oVpV/BlP7YTtKDUw9RQhf8=; b=Pvui4UUDtOkodT4OGqdWHq0DvI/3gbR+1jXrcHohm9HQzqPOVa1FL21s5FI/4HSKi0 Au5I1oPVMUYauaoGFBfLIUxrTLWkTuLxdUNLawwfzxmAugphuwkraMuir3lIuXKU8091 y79Wg3p7/oSSLHM6h6+SI7cLRQGMXSf11iDj4= MIME-Version: 1.0 Received: by 10.227.42.75 with SMTP id r11mr5579556wbe.1.1319475583264; Mon, 24 Oct 2011 09:59:43 -0700 (PDT) Received: by 10.180.79.103 with HTTP; Mon, 24 Oct 2011 09:59:43 -0700 (PDT) In-Reply-To: <1319449307149-4931883.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com> Date: Mon, 24 Oct 2011 09:59:43 -0700 Message-ID: From: Jack Vogel To: Sergey Saley Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 16:59:45 -0000 You need to increase your mbuf pool, note below in your messages where it has insufficient to configure for ix1, I don't know that this will change the interrupt difference but it should be addressed anyway. I also noticed that you have the adapter in a PCIE 1.0 slot (only 2.5 Gb/s), that will limit you, you should move to a 2.0 slot if possible. Regards, Jack On Mon, Oct 24, 2011 at 2:41 AM, Sergey Saley wrote: > There is my FreeBSD box: > > kernel > --------------- > # > # GENERIC -- Generic kernel configuration file for FreeBSD/i386 > # > # For more information on this file, please read the config(5) manual page, > # and/or the handbook section on Kernel Configuration Files: > # > # > > http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html > # > # The handbook is also available locally in /usr/share/doc/handbook > # if you've installed the doc distribution, otherwise always see the > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the > # latest information. > # > # An exhaustive list of options and more detailed explanations of the > # device lines is also present in the ../../conf/NOTES and NOTES files. > # If you are in doubt as to the purpose or necessity of a line, check first > # in NOTES. > # > # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z jkim $ > > cpu I686_CPU > ident POINT07 > > > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_DIRHASH # Improve performance on big > directories > options MD_ROOT # MD is a potential root device > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires > PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options PRINTF_BUFR_SIZE=128 # Prevent printf output being > interspersed. > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for > hwpmc(4) > #options KDTRACE_HOOKS # Kernel DTrace hooks > options INCLUDE_CONFIG_FILE # Include this file in kernel > > > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor Kernel > device apic # I/O APIC > > # CPU frequency control > device cpufreq > > # Bus support. > device acpi > device pci > > # Floppy drives > device fdc > > # ATA controllers > device ahci # AHCI-compatible SATA controllers > device ata # Legacy ATA/SATA controllers > options ATA_CAM # Handle legacy controllers with CAM > options ATA_STATIC_ID # Static device numbering > device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC > SATA > device siis # SiliconImage SiI3124/SiI3132/SiI3531 SATA > > # ATA/SCSI peripherals > device scbus # SCSI bus (required for ATA/SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct ATA/SCSI > access) > device ses # SCSI Environmental Services (and SAF-TE) > > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse > > device kbdmux # keyboard multiplexer > > device vga # VGA video card driver > > > # syscons is the default console driver, resembling an SCO console > device sc > options SC_PIXEL_MODE # add support for the raster text mode > > device agp # support several AGP chipsets > > # Power management support (see NOTES for more options) > #device apm > # Add suspend/resume support for the i8254. > device pmtimer > > # Serial (COM) ports > device uart # Generic UART driver > > # Parallel port > device ppc > device ppbus # Parallel port bus (required) > device lpt # Printer > #device plip # TCP/IP over parallel > device ppi # Parallel port interface device > #device vpo # Requires scbus and da > > # If you've got a "dumb" serial or parallel PCI card that is > # supported by the puc(4) glue driver, uncomment the following > # line to enable it (connects to sio, uart and/or ppc drivers): > #device puc > > # PCI Ethernet NICs. > device em # Intel PRO/1000 Gigabit Ethernet Family > device ixgbe # Intel PRO/10GbE Ethernet Card > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > #device tun # Packet tunnel. > device pty # BSD-style compatibility pseudo ttys > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying (translation) > device firmware # firmware assist module > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_FORWARD > options DUMMYNET > options IPI_PREEMPTION > > # netgraph(4). Enable the base netgraph code with the NETGRAPH option. > # Individual node types can be enabled with the corresponding option > # listed below; however, this is not strictly necessary as netgraph > # will automatically load the corresponding KLD module if the node type > # is not already compiled into the kernel. Each type below has a > # corresponding man page, e.g., ng_async(8). > > options NETGRAPH # netgraph(4) system > > options NETGRAPH_CAR > options NETGRAPH_IFACE > options NETGRAPH_MPPC_ENCRYPTION > options NETGRAPH_PPP > options NETGRAPH_PPPOE > options NETGRAPH_PPTPGRE > options NETGRAPH_SOCKET > options NETGRAPH_TCPMSS > options NETGRAPH_ETHER > options NETGRAPH_TEE > options NETGRAPH_VJC > options NETGRAPH_BPF > options NETGRAPH_KSOCKET > options NETGRAPH_IPFW > > dmesg.boot > -------------------------- > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011 > root@point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64 > CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz K8-class > CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > > > Features=0xbfebfbff > Features2=0xe3bd > AMD Features=0x20100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 2147483648 (2048 MB) > avail memory = 2056187904 (1960 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > ix0: port > 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 at > device 0.0 on pci1 > ix0: Using MSIX interrupts with 5 vectors > ix0: Ethernet address: 00:25:90:3c:42:6a > ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 > ix1: port > 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 at > device 0.1 on pci1 > ix1: Using MSIX interrupts with 5 vectors > ix1: RX Descriptors exceed system mbuf max, using default instead! > ix1: Ethernet address: 00:25:90:3c:42:6b > ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 > pci0: at device 26.0 (no driver attached) > pci0: at device 26.1 (no driver attached) > pci0: at device 26.2 (no driver attached) > pci0: at device 26.7 (no driver attached) > pcib2: irq 16 at device 28.0 on pci0 > pci5: on pcib2 > pcib3: irq 16 at device 28.4 on pci0 > pci13: on pcib3 > em0: port 0x3000-0x301f mem > 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13 > em0: Using an MSI interrupt > em0: Ethernet address: 00:30:48:d2:12:ac > pcib4: irq 17 at device 28.5 on pci0 > pci15: on pcib4 > em1: port 0x4000-0x401f mem > 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15 > em1: Using an MSI interrupt > em1: Ethernet address: 00:30:48:d2:12:ad > pci0: at device 29.0 (no driver attached) > pci0: at device 29.1 (no driver attached) > pci0: at device 29.2 (no driver attached) > pci0: at device 29.7 (no driver attached) > pcib5: at device 30.0 on pci0 > pci17: on pcib5 > vgapci0: port 0x5000-0x507f mem > 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at device > 31.2 on pci0 > ata0: on atapci0 > ata1: on atapci0 > pci0: at device 31.3 (no driver attached) > atapci1: port > > 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c20-0x1c2f > irq 18 at device 31.5 on pci0 > ata2: on atapci1 > ata3: on atapci1 > pci0: at device 31.6 (no driver attached) > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: on ppc0 > lpt0: on ppbus0 > lpt0: Interrupt-driven port > ppi0: on ppbus0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > est2: on cpu2 > p4tcc2: on cpu2 > est3: on cpu3 > p4tcc3: on cpu3 > Timecounters tick every 1.000 msec > ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding > enabled, default to accept, logging disabled > DUMMYNET 0 with IPv6 initialized (100409) > load_dn_sched dn_sched QFQ loaded > load_dn_sched dn_sched RR loaded > load_dn_sched dn_sched WF2Q+ loaded > load_dn_sched dn_sched FIFO loaded > load_dn_sched dn_sched PRIO loaded > ada0 at ata0 bus 0 scbus0 target 0 lun 0 > ada0: ATA-8 SATA 2.x device > ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) > ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad0 > SMP: AP CPU #3 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #2 Launched! > Timecounter "TSC-low" frequency 9351750 Hz quality 1000 > Trying to mount root from ufs:/dev/ada0p2 [rw]... > > sysctl.conf > -------------- > net.inet.ip.fastforwarding=1 > net.isr.direct=0 > net.isr.direct_force=0 > net.inet.ip.fw.one_pass=1 > hw.intr_storm_threshold=9000 > kern.ipc.nmbclusters=262144 > kern.ipc.nmbjumbop=262144 > dev.ix.0.rx_processing_limit=4096 > dev.ix.1.rx_processing_limit=4096 > net.inet.ip.intr_queue_maxlen=256 > dev.ix.0.flow_control=0 > dev.ix.1.flow_control=0 > > vmstat -i > ------------- > # vmstat -i > interrupt total rate > irq1: atkbd0 1 0 > irq6: fdc0 1 0 > irq14: ata0 80855 9 > irq15: ata1 1 0 > cpu0:timer 9694291 1122 > irq256: ix0:que 0 54169187 6273 > irq257: ix0:que 1 19307922 2236 > irq258: ix0:que 2 24265573 2810 > irq259: ix0:que 3 25845865 2993 > irq260: ix0:link 2 0 > irq261: ix1:que 0 63210220 7321 > irq262: ix1:que 1 265895 30 > irq263: ix1:que 2 42114 4 > irq264: ix1:que 3 98953 11 > irq265: ix1:link 5 0 > cpu3:timer 7165968 829 > cpu1:timer 7386880 855 > cpu2:timer 7307920 846 > Total 218841653 25346 > > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 26592 0 0 30220498 20757 0 11490639 0 > 31232 0 0 34001150 24699 0 14554631 0 > 27785 0 0 29672129 23699 0 16154542 0 > 25525 0 0 27450183 22868 0 16180216 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 22398 0 0 17232635 23781 0 24718205 0 > 23671 0 0 17608589 25859 0 28970777 0 > 24096 0 0 17278557 27188 0 30804019 0 > 22043 0 0 16271250 24116 0 26238423 0 > ^C > > > top -SCHP > ----------------- > last pid: 74628; load averages: 1.26, 1.01, 0.89 > up 0+02:25:14 13:22:11 > 110 processes: 9 running, 79 sleeping, 22 waiting > CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% idle > CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% idle > CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% idle > CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% idle > Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% idle{idle: > cpu2} > 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% idle{idle: > cpu1} > 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% idle{idle: > cpu3} > 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% idle{idle: > cpu0} > 11 root -92 - 0K 416K CPU0 0 30:52 36.77% intr{irq261: > ix1:que } > 11 root -92 - 0K 416K CPU3 3 8:03 11.77% intr{irq259: > ix0:que } > 11 root -92 - 0K 416K CPU1 1 5:37 9.86% intr{irq257: > ix0:que } > 11 root -92 - 0K 416K WAIT 2 8:14 9.28% intr{irq258: > ix0:que } > 11 root -92 - 0K 416K RUN 0 7:07 5.18% intr{irq256: > ix0:que } > 0 root -92 0 0K 352K - 2 1:12 1.17% kernel{ix1 > que} > 12 root -16 - 0K 64K sleep 0 1:02 1.07% > ng_queue{ng_queue1} > 12 root -16 - 0K 64K sleep 0 1:02 0.88% > ng_queue{ng_queue0} > 12 root -16 - 0K 64K sleep 2 1:02 0.78% > ng_queue{ng_queue3} > 12 root -16 - 0K 64K sleep 0 1:02 0.78% > ng_queue{ng_queue2} > 0 root -92 0 0K 352K - 3 0:58 0.39% kernel{ix0 > que} > 11 root -60 - 0K 416K WAIT 3 5:21 0.00% intr{swi4: > clock} > 0 root -16 0 0K 352K sched 1 0:54 0.00% > kernel{swapper} > 14320 root 20 0 78528K 32900K select 1 0:52 0.00% mpd5{mpd5} > 0 root -92 0 0K 352K - 0 0:28 0.00% kernel{ix0 > que} > 0 root -92 0 0K 352K - 2 0:25 0.00% > kernel{dummynet} > 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra > 0 root -92 0 0K 352K - 2 0:17 0.00% kernel{ix0 > que} > 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow > 0 root -92 0 0K 352K - 3 0:07 0.00% kernel{ix0 > que} > 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd > 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd > 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd > 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer > 0 root -92 0 0K 352K - 1 0:02 0.00% kernel{ix1 > linkq} > 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd > 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd > 11 root -92 - 0K 416K WAIT 1 0:01 0.00% intr{irq262: > ix1:que } > 11 root -68 - 0K 416K WAIT 2 0:01 0.00% intr{swi2: > cambio} > 13 root -8 - 0K 48K - 3 0:01 0.00% geom{g_up} > 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00% > collectd{collectd} > 11 root -88 - 0K 416K WAIT 3 0:01 0.00% intr{irq14: > ata0} > 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd > 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd > 13 root -8 - 0K 48K - 0 0:01 0.00% geom{g_down} > 11 root -92 - 0K 416K WAIT 3 0:01 0.00% intr{irq264: > ix1:que } > 15 root -16 - 0K 16K sdflus 3 0:00 0.00% softdepflush > 11 root -60 - 0K 416K WAIT 2 0:00 0.00% intr{swi4: > clock} > > pciconf -lvc > --------------- > ix0@pci0:1:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class = network > subclass = ethernet > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] = VPD > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 002590ffff3c426a > ecap 000e[150] = unknown 1 > ecap 0010[160] = unknown 1 > ix1@pci0:1:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class = network > subclass = ethernet > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] = VPD > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 002590ffff3c426a > ecap 000e[150] = unknown 1 > ecap 0010[160] = unknown 1 > > The question is: > ----------------- > Why there are so much interrupts? > Why the card is using only one intr on second port (ix1)? > > I've got another server with 82598EB 10 Gigabit AT CX4 Network Connection > and the situation on it is dramatically different: > > last pid: 42234; load averages: 2.14, 1.65, 1.55 > up 119+05:58:58 13:26:01 > 98 processes: 8 running, 62 sleeping, 28 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% idle > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% idle > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle > Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: cpu0} > 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: cpu2} > 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: cpu3} > 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: cpu1} > 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259: > ix0:que } > 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257: > ix0:que } > 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258: > ix0:que } > 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256: > ix0:que } > 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264: > ix1:que } > 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262: > ix1:que } > 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263: > ix1:que } > 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261: > ix1:que } > > vmstat -i > ------------- > r# vmstat -i > interrupt total rate > irq19: atapci0 1667110 0 > cpu0: timer 3386721810 328 > irq256: ix0:que 0 3395843376 329 > irq257: ix0:que 1 2642665824 256 > irq258: ix0:que 2 2838302235 275 > irq259: ix0:que 3 2176207954 211 > irq260: ix0:link 18 0 > irq261: ix1:que 0 282359321 27 > irq262: ix1:que 1 3989170496 387 > irq263: ix1:que 2 375956573 36 > irq264: ix1:que 3 3966352151 384 > irq265: ix1:link 1 0 > irq266: em0:rx 0 10283114 0 > irq269: em1:rx 0 10283114 0 > cpu3: timer 3386697130 328 > cpu1: timer 3386709028 328 > cpu2: timer 3386700908 328 > Total 33235920163 3225 > > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 264572 0 0 252383894 253290 0 178845781 0 > 266323 0 0 254825280 251277 0 173769218 0 > 274462 0 0 265247306 260819 0 181113304 0 > 271142 0 0 263263325 253941 0 171694438 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 259427 0 0 183038665 275711 0 262429749 0 > 248177 0 0 171479966 264441 0 252993752 0 > 255271 0 0 185006000 266886 0 247494148 0 > 264543 0 0 190970875 275087 0 259555066 0 > ^C > > Tuning on both systems are almost the same > As You can see, 82598EB card produce about 20 times less interrupts at > about > 10 times more pps. > > Please help me to fix the problem... > > -- > View this message in context: > http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4931883.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 17:46:25 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 385B9106564A for ; Mon, 24 Oct 2011 17:46:25 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 148808FC0A for ; Mon, 24 Oct 2011 17:46:24 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIObQ-0005Sm-8t for freebsd-net@freebsd.org; Mon, 24 Oct 2011 10:46:24 -0700 Date: Mon, 24 Oct 2011 10:46:24 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319478384269-4933498.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 17:46:25 -0000 Jack Vogel wrote: > > You need to increase your mbuf pool, note below in your messages where it > has > insufficient to configure for ix1, I don't know that this will change the > interrupt > difference but it should be addressed anyway. > Thank You for answer! Nobody's fault but mine :-( Being fixing... Jack Vogel wrote: > > I also noticed that you have the adapter in a PCIE 1.0 slot (only 2.5 > Gb/s), > that > will limit you, you should move to a 2.0 slot if possible. > PCI Express Bus: Speed 2.5Gb/s Width x8 Does it not enough? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4933498.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 17:54:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE320106564A for ; Mon, 24 Oct 2011 17:54:33 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 726928FC14 for ; Mon, 24 Oct 2011 17:54:33 +0000 (UTC) Received: by wwi18 with SMTP id 18so9104914wwi.31 for ; Mon, 24 Oct 2011 10:54:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=N8BIMSvOXzuI+W36Tga11atqS7exe+imIH8mbayZHGY=; b=vBAtbWUtupppx1FQoC2jF94iiPhGk/1KpDeKBlLAe45lWMWufjZ7NJLrMk8B9KqFRP IhUXGzJhEMavHYOuKtBcHBVfMcQnJmEKXcber0SosJRmKqnRLJ2jpnnN4yoNiURWqAEB i0xIef3aJJkFolL+ehq2IgqqizzIAMHHdjnB4= Received: by 10.227.27.165 with SMTP id i37mr9313075wbc.106.1319478872068; Mon, 24 Oct 2011 10:54:32 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id fy13sm40319391wbb.18.2011.10.24.10.54.28 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 24 Oct 2011 10:54:31 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 24 Oct 2011 10:52:52 -0700 From: YongHyeon PYUN Date: Mon, 24 Oct 2011 10:52:52 -0700 To: Nikolay Denev Message-ID: <20111024175252.GB4663@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: Possible sge(4)/atphy(4) regression on RELENG_9? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 17:54:33 -0000 On Mon, Oct 24, 2011 at 04:43:57PM +0300, Nikolay Denev wrote: > Hello, > > I've recently upgraded a box running RELENG_8 to RELENG_9 and immediately I noticed much slower network connection. > Running iperf shows about 20-30Mbits which was almost full GigE (~900Mbits) speed before. > > I'm noticing interface errors : > > [16:37]ndenev@nas:~% netstat -I sge0 > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll > sge0 1500 00:0a:e4:86:62:fa 76114295 42197 0 103559806 10324 0 > sge0 1500 10.0.0.0 nas 76109575 - - 119109557 - - > > Both the switch and the card show 1000 full-duplex. > I've tried playing with rxcsum,txcsum,vlanhwtag,tso but disabling even all of them do not change anything. > I've tried different switch port and changed the cable. > > Here is devinfo for my hardware : > > sge0 pnpinfo vendor=0x1039 device=0x0191 subvendor=0x103c subdevice=0x2a70 class=0x020000 > atphy0 pnpinfo oui=0xc82e model=0x1 rev=0x6 at phyno=0 > > Of course all of this can mean hardware problem, I just want to ask if somebody is seeing something similar, since > there are quite a lot minibus related changes as far as I can see. > > I'll boot RELENG_8 again tomorrow and do a quick test again to verify that this is not a hardware issue. > I don't have sge(4) controller so it would be better to let us know which revision introduced the regression. Just looking over the code change didn't reveal the possible cause. BTW, I thought sge(4) shall use rgephy(4). Can you also verify whether sge(4) in stable/8 also use atphy(4)? From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 18:01:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D02B1106566B for ; Mon, 24 Oct 2011 18:01:19 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 5F1818FC17 for ; Mon, 24 Oct 2011 18:01:18 +0000 (UTC) Received: by wwn22 with SMTP id 22so4017124wwn.1 for ; Mon, 24 Oct 2011 11:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bPjuP1W6sPq48LI7g6325vcQs/+ewyd7HP7NkM6jBGg=; b=W38TmYt2KFvaPcP87M5P+THWd7AO6q65OrdYhZyjRACB/MykYO3btOQWk0tO0iPxWV 2uZnzKUlGIfqfGWud+vAzd70ysz9sweXTqOmFkdHzL1fwyyetqGKzrVn34p/lCboifff EsozN+GN6+SqQkzVqGoBAZd58cw33gh2aG+v0= MIME-Version: 1.0 Received: by 10.216.9.204 with SMTP id 54mr3531704wet.1.1319479278071; Mon, 24 Oct 2011 11:01:18 -0700 (PDT) Received: by 10.180.79.103 with HTTP; Mon, 24 Oct 2011 11:01:17 -0700 (PDT) In-Reply-To: <1319478384269-4933498.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> Date: Mon, 24 Oct 2011 11:01:17 -0700 Message-ID: From: Jack Vogel To: Sergey Saley Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 18:01:20 -0000 On Mon, Oct 24, 2011 at 10:46 AM, Sergey Saley wrote: > > Jack Vogel wrote: > > > > You need to increase your mbuf pool, note below in your messages where it > > has > > insufficient to configure for ix1, I don't know that this will change the > > interrupt > > difference but it should be addressed anyway. > > > > Thank You for answer! > Nobody's fault but mine :-( > Being fixing... > > > Jack Vogel wrote: > > > > I also noticed that you have the adapter in a PCIE 1.0 slot (only 2.5 > > Gb/s), > > that > > will limit you, you should move to a 2.0 slot if possible. > > > > PCI Express Bus: Speed 2.5Gb/s Width x8 > Does it not enough? > > > > > Depends on what is 'enough' :) With bidirectional traffic on two ports you will be bus limited, on a 2.0 x8 slot you will not. So, given a choice, I'd go with the newer architecture... Jack From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 18:19:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61CDE106564A for ; Mon, 24 Oct 2011 18:19:16 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 20AFB8FC0A for ; Mon, 24 Oct 2011 18:19:15 +0000 (UTC) Received: by ggnq2 with SMTP id q2so6179364ggn.13 for ; Mon, 24 Oct 2011 11:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=J0IMBlKdG8cgYPDysy3ob1MZjMGeWZAomqzDbFyZu3Q=; b=nIEPxnb+wrQVymMeqFtzq2AQOFs+bTZYUZVP091HOky2+BD0vwuFFpjpMAc4bL/6b5 V29oVnOPZzIsb1TlEpCYX7gePM1uxopvb2BuDJMK/Ku/5JgiftzH4nD+E61Md0YQtcOV aNS2v1YODUuUhPStXoSGRHhT3EmdN/z9oG+jk= Received: by 10.223.76.11 with SMTP id a11mr45046342fak.1.1319480354943; Mon, 24 Oct 2011 11:19:14 -0700 (PDT) Received: from imba-brutale.totalterror.net ([93.152.152.135]) by mx.google.com with ESMTPS id a21sm943875fao.18.2011.10.24.11.18.56 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 24 Oct 2011 11:19:05 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <20111024175252.GB4663@michelle.cdnetworks.com> Date: Mon, 24 Oct 2011 21:18:57 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <0DF73F37-3E46-4F7D-AA6B-B7EB2F2276AB@gmail.com> References: <20111024175252.GB4663@michelle.cdnetworks.com> To: pyunyh@gmail.com X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-net@freebsd.org Subject: Re: Possible sge(4)/atphy(4) regression on RELENG_9? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 18:19:16 -0000 On Oct 24, 2011, at 8:52 PM, YongHyeon PYUN wrote: > On Mon, Oct 24, 2011 at 04:43:57PM +0300, Nikolay Denev wrote: >> Hello, >>=20 >> I've recently upgraded a box running RELENG_8 to RELENG_9 and = immediately I noticed much slower network connection. >> Running iperf shows about 20-30Mbits which was almost full GigE = (~900Mbits) speed before. >>=20 >> I'm noticing interface errors : >>=20 >> [16:37]ndenev@nas:~% netstat -I sge0 >> Name Mtu Network Address Ipkts Ierrs Idrop = Opkts Oerrs Coll >> sge0 1500 00:0a:e4:86:62:fa 76114295 42197 0 = 103559806 10324 0 >> sge0 1500 10.0.0.0 nas 76109575 - - = 119109557 - - >>=20 >> Both the switch and the card show 1000 full-duplex. >> I've tried playing with rxcsum,txcsum,vlanhwtag,tso but disabling = even all of them do not change anything. >> I've tried different switch port and changed the cable. >>=20 >> Here is devinfo for my hardware : >>=20 >> sge0 pnpinfo vendor=3D0x1039 device=3D0x0191 subvendor=3D0x103c = subdevice=3D0x2a70 class=3D0x020000 >> atphy0 pnpinfo oui=3D0xc82e model=3D0x1 rev=3D0x6 at phyno=3D0 >>=20 >> Of course all of this can mean hardware problem, I just want to ask = if somebody is seeing something similar, since >> there are quite a lot minibus related changes as far as I can see. >>=20 >> I'll boot RELENG_8 again tomorrow and do a quick test again to verify = that this is not a hardware issue. >>=20 >=20 > I don't have sge(4) controller so it would be better to let us know > which revision introduced the regression. Just looking over the > code change didn't reveal the possible cause. > BTW, I thought sge(4) shall use rgephy(4). Can you also verify > whether sge(4) in stable/8 also use atphy(4)? I've just checked my logs and I can confirm that it was atphy(4) even in = stable/8. Sep 26 15:55:19 nas kernel: atphy0: PHY 0 = on miibus0 Sep 26 15:55:19 nas kernel: atphy0: none, 10baseT, 10baseT-FDX, = 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto I'll post more info when I try again stable/8 on this hardware. Thanks! From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 19:08:45 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D50B8106564A for ; Mon, 24 Oct 2011 19:08:45 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id AFAAD8FC13 for ; Mon, 24 Oct 2011 19:08:45 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIPt6-0005Ly-Rz for freebsd-net@freebsd.org; Mon, 24 Oct 2011 12:08:44 -0700 Date: Mon, 24 Oct 2011 12:08:44 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319483324861-4933765.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 19:08:45 -0000 Jack Vogel wrote: > > On Mon, Oct 24, 2011 at 10:46 AM, Sergey Saley <sergeysaley@>wrote: > >> >> Jack Vogel wrote: >> > >> > You need to increase your mbuf pool, note below in your messages where >> it >> > has >> > insufficient to configure for ix1, I don't know that this will change >> the >> > interrupt >> > difference but it should be addressed anyway. >> > >> >> > Nothing has changed after increasing mbufs count increasing: irq256: ix0:que 0 3013004 2431 irq257: ix0:que 1 970295 783 irq258: ix0:que 2 574782 463 irq259: ix0:que 3 520764 420 irq260: ix0:link 1 0 irq261: ix1:que 0 3185946 2571 irq262: ix1:que 1 20425 16 irq263: ix1:que 2 10098 8 irq264: ix1:que 3 6999 5 irq265: ix1:link 3 0 Still only one irq being really used for ix1... -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4933765.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 19:38:03 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F67D106566B for ; Mon, 24 Oct 2011 19:38:03 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id E9F188FC08 for ; Mon, 24 Oct 2011 19:38:02 +0000 (UTC) Received: by wwi18 with SMTP id 18so9247127wwi.31 for ; Mon, 24 Oct 2011 12:38:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=He6ZN7/HmJImwAGYH5wuU61ma/wOHaB0tS8KNqfN/jQ=; b=SJV7R7NoWUTWjOUPckAgeBhlI3xztJ2Wjrw98TwEQySACEXCKQpYRtDFVctH1MgZ9k R/TC2nIq6mL19yVmfVLjr3qF/twv3HirG5Q6eROLuq2eI6OCn/IItL95JaMYlDxto2ut TLwdnfYqT7/GiQTVHkbE6jlPFY4Nr1MN/7J28= MIME-Version: 1.0 Received: by 10.227.19.193 with SMTP id c1mr9325137wbb.24.1319485081801; Mon, 24 Oct 2011 12:38:01 -0700 (PDT) Received: by 10.180.8.34 with HTTP; Mon, 24 Oct 2011 12:38:01 -0700 (PDT) In-Reply-To: <1319483324861-4933765.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> Date: Mon, 24 Oct 2011 15:38:01 -0400 Message-ID: From: Ryan Stone To: Sergey Saley Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 19:38:03 -0000 On Mon, Oct 24, 2011 at 3:08 PM, Sergey Saley wrote= : > Nothing has changed after increasing mbufs count increasing: > > irq256: ix0:que 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03013004 =A0 =A0 =A0 2431 > irq257: ix0:que 1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 970295 =A0 =A0 =A0 =A07= 83 > irq258: ix0:que 2 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 574782 =A0 =A0 =A0 =A04= 63 > irq259: ix0:que 3 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 520764 =A0 =A0 =A0 =A04= 20 > irq260: ix0:link =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 1 =A0 =A0 = =A0 =A0 =A00 > irq261: ix1:que 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03185946 =A0 =A0 =A0 2571 > irq262: ix1:que 1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A020425 =A0 =A0 =A0 = =A0 16 > irq263: ix1:que 2 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A010098 =A0 =A0 =A0 = =A0 =A08 > irq264: ix1:que 3 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 6999 =A0 =A0 =A0 = =A0 =A05 > irq265: ix1:link =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 3 =A0 =A0 = =A0 =A0 =A00 What kind of traffic are you sending? How many flows? Is it straight Ethernet/IP/TCP(/UDP), or is there any kind of tunneling in use? From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 19:51:25 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 864AB106566C for ; Mon, 24 Oct 2011 19:51:25 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 5F1C18FC0C for ; Mon, 24 Oct 2011 19:51:25 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIQYO-0000W4-R0 for freebsd-net@freebsd.org; Mon, 24 Oct 2011 12:51:24 -0700 Date: Mon, 24 Oct 2011 12:51:24 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319485884830-4933934.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 19:51:25 -0000 Ryan Stone-2 wrote: >=20 > On Mon, Oct 24, 2011 at 3:08 PM, Sergey Saley <sergeysaley@> wrote: >> Nothing has changed after increasing mbufs count increasing: >> >> irq256: ix0:que 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A03013004 =C2=A0 =C2=A0 =C2=A0 2431 >> irq257: ix0:que 1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 970295 =C2=A0 =C2=A0 =C2=A0 =C2=A0783 >> irq258: ix0:que 2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 574782 =C2=A0 =C2=A0 =C2=A0 =C2=A0463 >> irq259: ix0:que 3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 520764 =C2=A0 =C2=A0 =C2=A0 =C2=A0420 >> irq260: ix0:link =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 >> irq261: ix1:que 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A03185946 =C2=A0 =C2=A0 =C2=A0 2571 >> irq262: ix1:que 1 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A020425 =C2=A0 =C2=A0 =C2=A0 =C2=A0 16 >> irq263: ix1:que 2 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A010098 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A08 >> irq264: ix1:que 3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 6999 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A05 >> irq265: ix1:link =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 3 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 >=20 > What kind of traffic are you sending? How many flows? Is it straight > Ethernet/IP/TCP(/UDP), or is there any kind of tunneling in use? >=20 >=20 MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much= -interrupts-on-ixgbe-tp4931883p4933934.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 20:10:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BE98106567A for ; Mon, 24 Oct 2011 20:10:10 +0000 (UTC) (envelope-from aboyer@averesystems.com) Received: from zimbra.averesystems.com (75-149-8-245-Pennsylvania.hfc.comcastbusiness.net [75.149.8.245]) by mx1.freebsd.org (Postfix) with ESMTP id C9BB78FC0C for ; Mon, 24 Oct 2011 20:10:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.averesystems.com (Postfix) with ESMTP id BA03C8BC005; Mon, 24 Oct 2011 15:54:02 -0400 (EDT) X-Virus-Scanned: amavisd-new at averesystems.com Received: from zimbra.averesystems.com ([127.0.0.1]) by localhost (zimbra.averesystems.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2PGAbn3q3Gvk; Mon, 24 Oct 2011 15:53:50 -0400 (EDT) Received: from riven.arriad.com (fw.arriad.com [10.0.0.16]) by zimbra.averesystems.com (Postfix) with ESMTPSA id 0F0D28BC02F; Mon, 24 Oct 2011 15:53:47 -0400 (EDT) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Andrew Boyer In-Reply-To: <1319449307149-4931883.post@n5.nabble.com> Date: Mon, 24 Oct 2011 15:52:43 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1319449307149-4931883.post@n5.nabble.com> To: Sergey Saley X-Mailer: Apple Mail (2.1084) Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 20:10:10 -0000 You could try this patch. It disables the interrupt while it's being = handled. (The driver already re-enables it at the end of the handler = and the task.) -Andrew Index: sys/dev/ixgbe/ixgbe.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sys/dev/ixgbe/ixgbe.c (revision 226698) +++ sys/dev/ixgbe/ixgbe.c (working copy) @@ -1362,6 +1362,7 @@ bool more_tx, more_rx; u32 newitr =3D 0; =20 + ixgbe_disable_queue(adapter, que->msix); ++que->irqs; =20 more_rx =3D ixgbe_rxeof(que, adapter->rx_process_limit); On Oct 24, 2011, at 5:41 AM, Sergey Saley wrote: > There is my FreeBSD box: >=20 > kernel > --------------- > # > # GENERIC -- Generic kernel configuration file for FreeBSD/i386 > # > # For more information on this file, please read the config(5) manual = page, > # and/or the handbook section on Kernel Configuration Files: > # > # =20 > = http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-con= fig.html > # > # The handbook is also available locally in /usr/share/doc/handbook > # if you've installed the doc distribution, otherwise always see the > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the > # latest information. > # > # An exhaustive list of options and more detailed explanations of the > # device lines is also present in the ../../conf/NOTES and NOTES = files. > # If you are in doubt as to the purpose or necessity of a line, check = first > # in NOTES. > # > # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z = jkim $ >=20 > cpu I686_CPU > ident POINT07 >=20 >=20 > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread = preemption > options INET # InterNETworking > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates = support > options UFS_DIRHASH # Improve performance on big = directories > options MD_ROOT # MD is a potential root device > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires = PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions > options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) > #options KDTRACE_HOOKS # Kernel DTrace hooks > options INCLUDE_CONFIG_FILE # Include this file in kernel >=20 >=20 > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor = Kernel > device apic # I/O APIC >=20 > # CPU frequency control > device cpufreq >=20 > # Bus support. > device acpi > device pci >=20 > # Floppy drives > device fdc >=20 > # ATA controllers > device ahci # AHCI-compatible SATA = controllers > device ata # Legacy ATA/SATA controllers > options ATA_CAM # Handle legacy controllers with CAM > options ATA_STATIC_ID # Static device numbering > device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA > device siis # SiliconImage = SiI3124/SiI3132/SiI3531 SATA >=20 > # ATA/SCSI peripherals > device scbus # SCSI bus (required for = ATA/SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct = ATA/SCSI access) > device ses # SCSI Environmental Services = (and SAF-TE) >=20 >=20 > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse >=20 > device kbdmux # keyboard multiplexer >=20 > device vga # VGA video card driver >=20 >=20 > # syscons is the default console driver, resembling an SCO console > device sc > options SC_PIXEL_MODE # add support for the raster text mode >=20 > device agp # support several AGP chipsets >=20 > # Power management support (see NOTES for more options) > #device apm > # Add suspend/resume support for the i8254. > device pmtimer >=20 > # Serial (COM) ports > device uart # Generic UART driver >=20 > # Parallel port > device ppc > device ppbus # Parallel port bus (required) > device lpt # Printer > #device plip # TCP/IP over parallel > device ppi # Parallel port interface device > #device vpo # Requires scbus and da >=20 > # If you've got a "dumb" serial or parallel PCI card that is > # supported by the puc(4) glue driver, uncomment the following > # line to enable it (connects to sio, uart and/or ppc drivers): > #device puc >=20 > # PCI Ethernet NICs. > device em # Intel PRO/1000 Gigabit = Ethernet Family > device ixgbe # Intel PRO/10GbE Ethernet Card >=20 > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > #device tun # Packet tunnel. > device pty # BSD-style compatibility pseudo = ttys > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying = (translation) > device firmware # firmware assist module >=20 > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter >=20 >=20 > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_FORWARD > options DUMMYNET > options IPI_PREEMPTION >=20 > # netgraph(4). Enable the base netgraph code with the NETGRAPH option. > # Individual node types can be enabled with the corresponding option > # listed below; however, this is not strictly necessary as netgraph > # will automatically load the corresponding KLD module if the node = type > # is not already compiled into the kernel. Each type below has a > # corresponding man page, e.g., ng_async(8). >=20 > options NETGRAPH # netgraph(4) system >=20 > options NETGRAPH_CAR > options NETGRAPH_IFACE > options NETGRAPH_MPPC_ENCRYPTION > options NETGRAPH_PPP > options NETGRAPH_PPPOE > options NETGRAPH_PPTPGRE > options NETGRAPH_SOCKET > options NETGRAPH_TCPMSS > options NETGRAPH_ETHER > options NETGRAPH_TEE > options NETGRAPH_VJC > options NETGRAPH_BPF > options NETGRAPH_KSOCKET > options NETGRAPH_IPFW >=20 > dmesg.boot > -------------------------- > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011 > root@point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64 > CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz = K8-class > CPU) > Origin =3D "GenuineIntel" Id =3D 0x6fb Family =3D 6 Model =3D f = Stepping =3D 11 >=20 > = Features=3D0xbfebfbff > = Features2=3D0xe3bd > AMD Features=3D0x20100800 > AMD Features2=3D0x1 > TSC: P-state invariant, performance statistics > real memory =3D 2147483648 (2048 MB) > avail memory =3D 2056187904 (1960 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > ix0: = port > 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 = at > device 0.0 on pci1 > ix0: Using MSIX interrupts with 5 vectors > ix0: Ethernet address: 00:25:90:3c:42:6a > ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 > ix1: = port > 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 = at > device 0.1 on pci1 > ix1: Using MSIX interrupts with 5 vectors > ix1: RX Descriptors exceed system mbuf max, using default instead! > ix1: Ethernet address: 00:25:90:3c:42:6b > ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 > pci0: at device 26.0 (no driver attached) > pci0: at device 26.1 (no driver attached) > pci0: at device 26.2 (no driver attached) > pci0: at device 26.7 (no driver attached) > pcib2: irq 16 at device 28.0 on pci0 > pci5: on pcib2 > pcib3: irq 16 at device 28.4 on pci0 > pci13: on pcib3 > em0: port 0x3000-0x301f = mem > 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13 > em0: Using an MSI interrupt > em0: Ethernet address: 00:30:48:d2:12:ac > pcib4: irq 17 at device 28.5 on pci0 > pci15: on pcib4 > em1: port 0x4000-0x401f = mem > 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15 > em1: Using an MSI interrupt > em1: Ethernet address: 00:30:48:d2:12:ad > pci0: at device 29.0 (no driver attached) > pci0: at device 29.1 (no driver attached) > pci0: at device 29.2 (no driver attached) > pci0: at device 29.7 (no driver attached) > pcib5: at device 30.0 on pci0 > pci17: on pcib5 > vgapci0: port 0x5000-0x507f mem > 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17 > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci0: port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at = device > 31.2 on pci0 > ata0: on atapci0 > ata1: on atapci0 > pci0: at device 31.3 (no driver attached) > atapci1: port > = 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c= 20-0x1c2f > irq 18 at device 31.5 on pci0 > ata2: on atapci1 > ata3: on atapci1 > pci0: at device 31.6 (no driver attached) > acpi_button0: on acpi0 > atrtc0: port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on = acpi0 > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 > ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on = acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: on ppc0 > lpt0: on ppbus0 > lpt0: Interrupt-driven port > ppi0: on ppbus0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=3D0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > est2: on cpu2 > p4tcc2: on cpu2 > est3: on cpu3 > p4tcc3: on cpu3 > Timecounters tick every 1.000 msec > ipfw2 initialized, divert loadable, nat loadable, rule-based = forwarding > enabled, default to accept, logging disabled > DUMMYNET 0 with IPv6 initialized (100409) > load_dn_sched dn_sched QFQ loaded > load_dn_sched dn_sched RR loaded > load_dn_sched dn_sched WF2Q+ loaded > load_dn_sched dn_sched FIFO loaded > load_dn_sched dn_sched PRIO loaded > ada0 at ata0 bus 0 scbus0 target 0 lun 0 > ada0: ATA-8 SATA 2.x device > ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) > ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad0 > SMP: AP CPU #3 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #2 Launched! > Timecounter "TSC-low" frequency 9351750 Hz quality 1000 > Trying to mount root from ufs:/dev/ada0p2 [rw]... >=20 > sysctl.conf > -------------- > net.inet.ip.fastforwarding=3D1 > net.isr.direct=3D0 > net.isr.direct_force=3D0 > net.inet.ip.fw.one_pass=3D1 > hw.intr_storm_threshold=3D9000 > kern.ipc.nmbclusters=3D262144 > kern.ipc.nmbjumbop=3D262144 > dev.ix.0.rx_processing_limit=3D4096 > dev.ix.1.rx_processing_limit=3D4096 > net.inet.ip.intr_queue_maxlen=3D256 > dev.ix.0.flow_control=3D0 > dev.ix.1.flow_control=3D0 >=20 > vmstat -i > ------------- > # vmstat -i > interrupt total rate > irq1: atkbd0 1 0 > irq6: fdc0 1 0 > irq14: ata0 80855 9 > irq15: ata1 1 0 > cpu0:timer 9694291 1122 > irq256: ix0:que 0 54169187 6273 > irq257: ix0:que 1 19307922 2236 > irq258: ix0:que 2 24265573 2810 > irq259: ix0:que 3 25845865 2993 > irq260: ix0:link 2 0 > irq261: ix1:que 0 63210220 7321 > irq262: ix1:que 1 265895 30 > irq263: ix1:que 2 42114 4 > irq264: ix1:que 3 98953 11 > irq265: ix1:link 5 0 > cpu3:timer 7165968 829 > cpu1:timer 7386880 855 > cpu2:timer 7307920 846 > Total 218841653 25346 >=20 > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 26592 0 0 30220498 20757 0 11490639 0 > 31232 0 0 34001150 24699 0 14554631 0 > 27785 0 0 29672129 23699 0 16154542 0 > 25525 0 0 27450183 22868 0 16180216 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 22398 0 0 17232635 23781 0 24718205 0 > 23671 0 0 17608589 25859 0 28970777 0 > 24096 0 0 17278557 27188 0 30804019 0 > 22043 0 0 16271250 24116 0 26238423 0 > ^C >=20 >=20 > top -SCHP > ----------------- > last pid: 74628; load averages: 1.26, 1.01, 0.89 = = =20 > up 0+02:25:14 13:22:11 > 110 processes: 9 running, 79 sleeping, 22 waiting > CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% = idle > CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% = idle > CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% = idle > CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% = idle > Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M = Free > Swap: 4096M Total, 4096M Free >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% = idle{idle: > cpu2} > 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% = idle{idle: > cpu1} > 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% = idle{idle: > cpu3} > 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% = idle{idle: > cpu0} > 11 root -92 - 0K 416K CPU0 0 30:52 36.77% = intr{irq261: > ix1:que } > 11 root -92 - 0K 416K CPU3 3 8:03 11.77% = intr{irq259: > ix0:que } > 11 root -92 - 0K 416K CPU1 1 5:37 9.86% = intr{irq257: > ix0:que } > 11 root -92 - 0K 416K WAIT 2 8:14 9.28% = intr{irq258: > ix0:que } > 11 root -92 - 0K 416K RUN 0 7:07 5.18% = intr{irq256: > ix0:que } > 0 root -92 0 0K 352K - 2 1:12 1.17% = kernel{ix1 > que} > 12 root -16 - 0K 64K sleep 0 1:02 1.07% > ng_queue{ng_queue1} > 12 root -16 - 0K 64K sleep 0 1:02 0.88% > ng_queue{ng_queue0} > 12 root -16 - 0K 64K sleep 2 1:02 0.78% > ng_queue{ng_queue3} > 12 root -16 - 0K 64K sleep 0 1:02 0.78% > ng_queue{ng_queue2} > 0 root -92 0 0K 352K - 3 0:58 0.39% = kernel{ix0 > que} > 11 root -60 - 0K 416K WAIT 3 5:21 0.00% = intr{swi4: > clock} > 0 root -16 0 0K 352K sched 1 0:54 0.00% > kernel{swapper} > 14320 root 20 0 78528K 32900K select 1 0:52 0.00% = mpd5{mpd5} > 0 root -92 0 0K 352K - 0 0:28 0.00% = kernel{ix0 > que} > 0 root -92 0 0K 352K - 2 0:25 0.00% > kernel{dummynet} > 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra > 0 root -92 0 0K 352K - 2 0:17 0.00% = kernel{ix0 > que} > 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow > 0 root -92 0 0K 352K - 3 0:07 0.00% = kernel{ix0 > que} > 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd > 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd > 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd > 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer > 0 root -92 0 0K 352K - 1 0:02 0.00% = kernel{ix1 > linkq} > 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd > 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd > 11 root -92 - 0K 416K WAIT 1 0:01 0.00% = intr{irq262: > ix1:que } > 11 root -68 - 0K 416K WAIT 2 0:01 0.00% = intr{swi2: > cambio} > 13 root -8 - 0K 48K - 3 0:01 0.00% = geom{g_up} > 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00% > collectd{collectd} > 11 root -88 - 0K 416K WAIT 3 0:01 0.00% = intr{irq14: > ata0} > 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd > 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd > 13 root -8 - 0K 48K - 0 0:01 0.00% = geom{g_down} > 11 root -92 - 0K 416K WAIT 3 0:01 0.00% = intr{irq264: > ix1:que } > 15 root -16 - 0K 16K sdflus 3 0:00 0.00% = softdepflush > 11 root -60 - 0K 416K WAIT 2 0:00 0.00% = intr{swi4: > clock} >=20 > pciconf -lvc > --------------- > ix0@pci0:1:0:0: class=3D0x020000 card=3D0x061115d9 chip=3D0x10fb8086 = rev=3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class =3D network > subclass =3D ethernet > cap 01[40] =3D powerspec 3 supports D0 D3 current D0 > cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks > cap 11[70] =3D MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] =3D VPD > ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] =3D Serial 1 002590ffff3c426a > ecap 000e[150] =3D unknown 1 > ecap 0010[160] =3D unknown 1 > ix1@pci0:1:0:1: class=3D0x020000 card=3D0x061115d9 chip=3D0x10fb8086 = rev=3D0x01 > hdr=3D0x00 > vendor =3D 'Intel Corporation' > device =3D '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class =3D network > subclass =3D ethernet > cap 01[40] =3D powerspec 3 supports D0 D3 current D0 > cap 05[50] =3D MSI supports 1 message, 64 bit, vector masks > cap 11[70] =3D MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] =3D PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] =3D VPD > ecap 0001[100] =3D AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] =3D Serial 1 002590ffff3c426a > ecap 000e[150] =3D unknown 1 > ecap 0010[160] =3D unknown 1 >=20 > The question is: > ----------------- > Why there are so much interrupts? > Why the card is using only one intr on second port (ix1)? >=20 > I've got another server with 82598EB 10 Gigabit AT CX4 Network = Connection > and the situation on it is dramatically different: >=20 > last pid: 42234; load averages: 2.14, 1.65, 1.55 = = =20 > up 119+05:58:58 13:26:01 > 98 processes: 8 running, 62 sleeping, 28 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% = idle > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% = idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% = idle > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% = idle > Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M = Free > Swap: 4096M Total, 4096M Free >=20 > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: = cpu0} > 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: = cpu2} > 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: = cpu3} > 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: = cpu1} > 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259: > ix0:que } > 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257: > ix0:que } > 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258: > ix0:que } > 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256: > ix0:que } > 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264: > ix1:que } > 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262: > ix1:que } > 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263: > ix1:que } > 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261: > ix1:que } >=20 > vmstat -i > ------------- > r# vmstat -i > interrupt total rate > irq19: atapci0 1667110 0 > cpu0: timer 3386721810 328 > irq256: ix0:que 0 3395843376 329 > irq257: ix0:que 1 2642665824 256 > irq258: ix0:que 2 2838302235 275 > irq259: ix0:que 3 2176207954 211 > irq260: ix0:link 18 0 > irq261: ix1:que 0 282359321 27 > irq262: ix1:que 1 3989170496 387 > irq263: ix1:que 2 375956573 36 > irq264: ix1:que 3 3966352151 384 > irq265: ix1:link 1 0 > irq266: em0:rx 0 10283114 0 > irq269: em1:rx 0 10283114 0 > cpu3: timer 3386697130 328 > cpu1: timer 3386709028 328 > cpu2: timer 3386700908 328 > Total 33235920163 3225 >=20 > netstat=20 > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 264572 0 0 252383894 253290 0 178845781 0 > 266323 0 0 254825280 251277 0 173769218 0 > 274462 0 0 265247306 260819 0 181113304 0 > 271142 0 0 263263325 253941 0 171694438 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 259427 0 0 183038665 275711 0 262429749 0 > 248177 0 0 171479966 264441 0 252993752 0 > 255271 0 0 185006000 266886 0 247494148 0 > 264543 0 0 190970875 275087 0 259555066 0 > ^C >=20 > Tuning on both systems are almost the same > As You can see, 82598EB card produce about 20 times less interrupts at = about > 10 times more pps. >=20 > Please help me to fix the problem... >=20 > -- > View this message in context: = http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp493188= 3p4931883.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -------------------------------------------------- Andrew Boyer aboyer@averesystems.com From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 20:32:50 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBF55106564A for ; Mon, 24 Oct 2011 20:32:50 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0BECC8FC0A for ; Mon, 24 Oct 2011 20:32:49 +0000 (UTC) Received: by wyi40 with SMTP id 40so8547936wyi.13 for ; Mon, 24 Oct 2011 13:32:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Y0xiSwapESy7A8NzC++uCUj2URFgRUOZsTPfGk51u7A=; b=XOQgFwQO1iv2OKK/fBKEoH+aiqALNExzd9+2cbbZ/maGJPB/9Pi3fvNT27nKM7pMi9 46Qshq7BMfkoUOyLmF7mmjs7g2BDMtsRWYydfYayKhYGY0jThlWFU9EMpYeusy4PWl/W IUP8San5041iTJKdzE0HKSkFdY4fk5RojZV4Q= MIME-Version: 1.0 Received: by 10.216.229.211 with SMTP id h61mr3581967weq.24.1319488368703; Mon, 24 Oct 2011 13:32:48 -0700 (PDT) Received: by 10.180.8.34 with HTTP; Mon, 24 Oct 2011 13:32:48 -0700 (PDT) In-Reply-To: <1319485884830-4933934.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> Date: Mon, 24 Oct 2011 16:32:48 -0400 Message-ID: From: Ryan Stone To: Sergey Saley Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 20:32:50 -0000 On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley wrote: > MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). > Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. > It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. PPPoE is your problem. The Intel cards can't load-balance PPPoE traffic, so everything goes to one queue. It may be possible to write a netgraph module to load-balance the traffic across your CPUs. From owner-freebsd-net@FreeBSD.ORG Mon Oct 24 21:05:58 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C36521065670; Mon, 24 Oct 2011 21:05:58 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id ED5938FC12; Mon, 24 Oct 2011 21:05:57 +0000 (UTC) Received: by wwi18 with SMTP id 18so9354402wwi.31 for ; Mon, 24 Oct 2011 14:05:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=kbvipacLw8lRihcOLyK7ZUf+k6sikRSc7Gz5+p/WGKo=; b=dP+nXwQnGIHTOz5Y1nv6YcGVX/NXzd8j3rXls025M9c28SUWkD+jHvITzCvp7T5bmf TeJTrvUx3Rg2l0U7eQ8f5Xa2zUEixuvhiNZqvSdTBAbEJNLUvUn6e4wRMmsnfZhgnHKE XAqLyRt8ILzmDzL2TsHsa9IxNJW40GrH9vz/0= Received: by 10.216.54.20 with SMTP id h20mr2956348wec.97.1319490356673; Mon, 24 Oct 2011 14:05:56 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id gd18sm41238063wbb.5.2011.10.24.14.05.52 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 24 Oct 2011 14:05:55 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Mon, 24 Oct 2011 14:04:16 -0700 From: YongHyeon PYUN Date: Mon, 24 Oct 2011 14:04:16 -0700 To: freebsd-current@FreeBSD.org Message-ID: <20111024210416.GE4663@michelle.cdnetworks.com> References: <20111017002213.GB3338@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111017002213.GB3338@michelle.cdnetworks.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: Call for testers : ALi/ULi M5261/M5263 ethernet controller X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2011 21:05:58 -0000 On Sun, Oct 16, 2011 at 05:22:13PM -0700, YongHyeon PYUN wrote: > Hi, > > If you have ALi/ULi M5261/M5263 ethernet controller please try the > patch at the following URL and let me know how it works. > http://people.freebsd.org/~yongari/dc/dc.uli562x.diff > > The patch was generated against latest HEAD and it should be > cleanly applied to latest stable/8 and stable/7. > I committed revised version to HEAD(r226699, r226701). > Thanks. From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 01:59:13 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5823E1065673 for ; Tue, 25 Oct 2011 01:59:13 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id DBCFC8FC08 for ; Tue, 25 Oct 2011 01:59:12 +0000 (UTC) Received: by wwn22 with SMTP id 22so4455414wwn.1 for ; Mon, 24 Oct 2011 18:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=UA2PpxVmSP/8XxgaXEd+ES29kyH93Y7SUkOJSC5qfmY=; b=pRdkihE0WcAqaUOROI8LSA+kKtSgMDlxhgzGvLq4ubLx8dKX/QGe+DYRZXMZ2Zxlno tS/Y02cdIghQYfOqj47oA6LqhM5g5Ku0ysc2ljwEa5jkclAKBaWsSJHKRF/w+JKIHU43 +0Yoj4LpdcNYZL01sBMAwc+L9E0rcCi1OVMjY= MIME-Version: 1.0 Received: by 10.216.221.11 with SMTP id q11mr3882646wep.86.1319507951174; Mon, 24 Oct 2011 18:59:11 -0700 (PDT) Received: by 10.180.105.162 with HTTP; Mon, 24 Oct 2011 18:59:11 -0700 (PDT) In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> Date: Mon, 24 Oct 2011 21:59:11 -0400 Message-ID: From: Arnaud Lacombe To: Ryan Stone Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, Sergey Saley Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 01:59:13 -0000 Hi, On Mon, Oct 24, 2011 at 4:32 PM, Ryan Stone wrote: > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley wro= te: >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. > > PPPoE is your problem. =A0The Intel cards can't load-balance PPPoE > traffic, so everything goes to one queue. =A0It may be possible to write > a netgraph module to load-balance the traffic across your CPUs. > NetGraph already runs a thread per CPU, as you can see in top's output. There is also still have plenty of CPU usable, so I'd assume some hard-limit are hit, maybe any of `net.graph.maxdata' or `net.graph.maxalloc'. Sergey, what is the output of: # vmstat -z | grep NetGraph Thanks, - Arnaud From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 06:08:17 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24EA6106564A for ; Tue, 25 Oct 2011 06:08:17 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id EECD88FC0A for ; Tue, 25 Oct 2011 06:08:16 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIaBM-0006Jx-6P for freebsd-net@freebsd.org; Mon, 24 Oct 2011 23:08:16 -0700 Date: Mon, 24 Oct 2011 23:08:16 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319522896192-4935167.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 06:08:17 -0000 Arnaud Lacombe-6 wrote: >=20 > Hi, >=20 > On Mon, Oct 24, 2011 at 4:32 PM, Ryan Stone <rysto32@> wrote: >> On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> wrote= : >>> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >>> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >>> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. >> >> PPPoE is your problem. =C2=A0The Intel cards can't load-balance PPPoE >> traffic, so everything goes to one queue. =C2=A0It may be possible to wr= ite >> a netgraph module to load-balance the traffic across your CPUs. >> > NetGraph already runs a thread per CPU, as you can see in top's > output. There is also still have plenty of CPU usable, so I'd assume > some hard-limit are hit, maybe any of `net.graph.maxdata' or > `net.graph.maxalloc'. >=20 > Sergey, what is the output of: >=20 > # vmstat -z | grep NetGraph >=20 >=20 # vmstat -z | grep NetGraph ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP NetGraph items: 72, 4118, 2, 404,40806244, 0, 0 NetGraph data items: 72, 522, 0, 406,658514523, 0, 0 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much= -interrupts-on-ixgbe-tp4931883p4935167.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 07:22:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B321106566C for ; Tue, 25 Oct 2011 07:22:09 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 425308FC0C for ; Tue, 25 Oct 2011 07:22:09 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIbKq-00039y-FM for freebsd-net@freebsd.org; Tue, 25 Oct 2011 00:22:08 -0700 Date: Tue, 25 Oct 2011 00:22:08 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319527328469-4935272.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 07:22:09 -0000 Ryan Stone-2 wrote: > > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> wrote: >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. > > PPPoE is your problem. The Intel cards can't load-balance PPPoE > traffic, so everything goes to one queue. It may be possible to write > a netgraph module to load-balance the traffic across your CPUs. > OK, thank You for explanation. And what about the large number of interrupts? As for me, it's too much... irq256: ix0:que 0 240536944 6132 irq257: ix0:que 1 89090444 2271 irq258: ix0:que 2 93222085 2376 irq259: ix0:que 3 89435179 2280 irq260: ix0:link 1 0 irq261: ix1:que 0 269468769 6870 irq262: ix1:que 1 110974 2 irq263: ix1:que 2 434214 11 irq264: ix1:que 3 112281 2 irq265: ix1:link 1 0 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4935272.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 08:06:35 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 523071065676 for ; Tue, 25 Oct 2011 08:06:35 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id B4AF98FC0C for ; Tue, 25 Oct 2011 08:06:34 +0000 (UTC) Received: by wwi18 with SMTP id 18so298530wwi.31 for ; Tue, 25 Oct 2011 01:06:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KuO/lImrecu3JxIxWLeYmStuY2b8HmDBwR4pgqERdrQ=; b=a0S6nUqMSjJCEqQK2uHhG9o9NJiVLSBLXqejcAjp0VuPIqf7CLRtaOGk4jSLlC9nK9 t4NsS+XwVNVLZKTEm+5Z7LE3J4A0RKFFLKHa4vnMGjQKnG0ReahpgrjZPrAkUzeyLs7h Sul5eJWsjLvV27zwRPh/YrA6h+JvY72eCPmtY= MIME-Version: 1.0 Received: by 10.227.60.131 with SMTP id p3mr2310335wbh.4.1319529993507; Tue, 25 Oct 2011 01:06:33 -0700 (PDT) Received: by 10.180.79.103 with HTTP; Tue, 25 Oct 2011 01:06:33 -0700 (PDT) In-Reply-To: <1319527328469-4935272.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> <1319527328469-4935272.post@n5.nabble.com> Date: Tue, 25 Oct 2011 01:06:33 -0700 Message-ID: From: Jack Vogel To: Sergey Saley Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 08:06:35 -0000 On Tue, Oct 25, 2011 at 12:22 AM, Sergey Saley wrote: > > Ryan Stone-2 wrote: > > > > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> > wrote: > >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). > >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. > >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. > > > > PPPoE is your problem. The Intel cards can't load-balance PPPoE > > traffic, so everything goes to one queue. It may be possible to write > > a netgraph module to load-balance the traffic across your CPUs. > > > > OK, thank You for explanation. > And what about the large number of interrupts? > As for me, it's too much... > irq256: ix0:que 0 240536944 6132 > irq257: ix0:que 1 89090444 2271 > irq258: ix0:que 2 93222085 2376 > irq259: ix0:que 3 89435179 2280 > irq260: ix0:link 1 0 > irq261: ix1:que 0 269468769 6870 > irq262: ix1:que 1 110974 2 > irq263: ix1:que 2 434214 11 > irq264: ix1:que 3 112281 2 > irq265: ix1:link 1 0 > > How do you decide its 'too much' ? It may be that with your traffic you end up not being able to use offloads, just thinking. Its not like the hardware just "makes it up", it interrupts on the last descriptor of a packet which has the RS bit set. With TSO you will get larger chunks of data and thus less interrupts but your traffic probably doesn't qualify for it. Jack From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 08:09:41 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3970C106566B for ; Tue, 25 Oct 2011 08:09:41 +0000 (UTC) (envelope-from sol289@gmail.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0B8E68FC12 for ; Tue, 25 Oct 2011 08:09:40 +0000 (UTC) Received: by iaky10 with SMTP id y10so423980iak.13 for ; Tue, 25 Oct 2011 01:09:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; bh=73U9gxbp9YcjLB0f8lbypLDehn5f6UIVBjs1KKkyg/g=; b=oJ58r1ISXhsGqMYERJmxlmiHEwFBXq5JyB4Z3e09bl6LQ9ZljAKnzzRJNjv1oHlhtz mbvi83O1c3wVoZvwYcgQXD+6SwuawTaSLlj+/rjPsfSAoiBNLwMf9Mfz3fKpEloQSE1O fssNWWilVKhzQZ7GXNAluGmAvwsVTQJMHpgaA= Received: by 10.42.135.69 with SMTP id o5mr43509459ict.34.1319530179044; Tue, 25 Oct 2011 01:09:39 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.47.131 with HTTP; Tue, 25 Oct 2011 01:09:19 -0700 (PDT) From: alexander lunyov Date: Tue, 25 Oct 2011 11:09:19 +0300 Message-ID: To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 Subject: carp over openvpn? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 08:09:41 -0000 Hello. I'm trying to make work carp over openvpn in bridge mode. I have 3 servers, VPN-IN, VPN-OUT1 and VPN-OUT2, they connected to different ethernet networks and cannot see each other on data link level. All servers run 8.2-RELEASE. VPN-IN is a openvpn server in bridge mode, VPN-OUT1 and VPN-OUT2 are openvpn clients. I configured on each server address from 10.80.90.0/24 network as alias, so address space is looking like this: VPN-IN@bridge0: 10.80.90.63 - bridged to tap0 VPN-OUT1@em0: 10.80.90.4 - bridged to tap0 VPN-OUT2@em0: 10.80.90.6 - bridged to tap0 Servers have real IPs, which i masked as x.x.x.x, y.y.y.y and z.z.z.z. When VPN-OUT1 and VPN-OUT2 connects to VPN-IN i can ping all 10.80.90. addresses from anywhere, so the vpn is working. When i create CARP interfaces on both VPN-OUT-s, carp0 on both is in MASTER state and VPN-IN cannot ping carp address 10.80.90.10 (VPN-OUTs ping own 10.80.90.10 address ok). On VPN-IN@bridge0 i see advertisements from both VPN-OUTs: # tcpdump -i bridge0 net 10.80.90.0/24 18:34:48.505618 IP 10.80.90.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 18:34:48.801474 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36 18:34:49.546667 IP 10.80.90.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 18:34:50.198569 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36 On VPN-OUT1@bridge0 i see advertisements from VPN-OUT2: # tcpdump -i bridge0 net 10.80.90.0/24 00:35:39.811034 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 00:35:40.852178 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 On VPN-OUT2@bridge0 i see advertisements from VPN-OUT1: # tcpdump -i bridge0 net 10.80.90.0/24 00:35:39.811034 IP 10.80.90.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 00:35:40.852178 IP 10.80.90.4 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 10, authtype none, intvl 1s, length 36 When i try to ping carp address 10.80.90.10 from VPN-IN, I see arp requests but nobody answers, though ARP reaches VPN-OUTs: VPN-OUT2# tcpdump -i bridge0 net 10.80.90.0/24 07:49:30.014907 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36 07:49:30.700133 ARP, Request who-has 10.80.90.10 tell 10.80.90.63, length 28 07:49:31.412868 IP 10.80.90.6 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36 07:49:31.700014 ARP, Request who-has 10.80.90.10 tell 10.80.90.63, length 28 So, why carp interfaces on VPN-OUTs doesn't see each other advertisements and ARP from VPN-IN? VPN-OUT1# netstat -s -p carp carp: 6515137 packets received (IPv4) 42246 packets sent (IPv4) ifconfigs: VPN-IN# ifconfig bge0: flags=8843 metric 0 mtu 1500 options=c019b ether 00:19:99:16:32:fd inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.255 media: Ethernet autoselect (100baseTX ) status: active lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 tap0: flags=8943 metric 0 mtu 1500 options=80000 ether 00:bd:cd:f5:1a:00 Opened by PID 86461 bridge0: flags=8843 metric 0 mtu 1500 ether 76:38:a6:0e:16:36 inet 10.80.90.63 netmask 0xffffff00 broadcast 10.80.90.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143 ifmaxaddr 0 port 3 priority 128 path cost 2000000 VPN-OUT1# ifconfig em0: flags=8943 metric 0 mtu 1500 options=2098 ether 00:25:90:06:a7:ee inet y.y.y.y netmask 0xffffff00 broadcast y.y.y.255 inet 10.80.90.4 netmask 0xffffff00 broadcast 10.80.90.255 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 tap0: flags=8943 metric 0 mtu 1500 options=80000 ether 00:bd:98:a7:80:00 Opened by PID 79699 bridge0: flags=8843 metric 0 mtu 1500 ether a6:be:59:84:94:7f id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 20000 member: tap0 flags=143 ifmaxaddr 0 port 4 priority 128 path cost 2000000 carp0: flags=49 metric 0 mtu 1500 inet 10.80.90.10 netmask 0xffffff00 carp: MASTER vhid 1 advbase 1 advskew 10 VPN-OUT2# ifconfig em0: flags=8943 metric 0 mtu 1500 options=2098 ether 00:25:90:00:59:1a inet z.z.z.z netmask 0xffffff00 broadcast z.z.z.255 inet 10.80.90.6 netmask 0xffffff00 broadcast 10.80.90.255 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=3 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 nd6 options=3 tap0: flags=8943 metric 0 mtu 1500 options=80000 ether 00:bd:2e:29:90:00 Opened by PID 75704 bridge0: flags=8843 metric 0 mtu 1500 ether ba:37:68:2b:7d:32 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: em0 flags=143 ifmaxaddr 0 port 1 priority 128 path cost 20000 member: tap0 flags=143 ifmaxaddr 0 port 4 priority 128 path cost 2000000 carp0: flags=49 metric 0 mtu 1500 inet 10.80.90.10 netmask 0xffffff00 carp: MASTER vhid 1 advbase 1 advskew 100 p.s.: i also tried freevrrpd, and i see the same behavior - i see advertisements from both VPN-OUTs, but they don't see each other. p.p.s.: if i'm writing to wrong list, please, point me to the right one where i can ask this question. -- your sweet isn't ready yet From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 08:21:18 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F02B10656D3 for ; Tue, 25 Oct 2011 08:21:18 +0000 (UTC) (envelope-from sergeysaley@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 044848FC16 for ; Tue, 25 Oct 2011 08:21:17 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIcG5-000090-Cp for freebsd-net@freebsd.org; Tue, 25 Oct 2011 01:21:17 -0700 Date: Tue, 25 Oct 2011 01:21:17 -0700 (PDT) From: Sergey Saley To: freebsd-net@freebsd.org Message-ID: <1319530877390-4935427.post@n5.nabble.com> In-Reply-To: References: <1319449307149-4931883.post@n5.nabble.com> <1319478384269-4933498.post@n5.nabble.com> <1319483324861-4933765.post@n5.nabble.com> <1319485884830-4933934.post@n5.nabble.com> <1319527328469-4935272.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Too much interrupts on ixgbe X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 08:21:18 -0000 Jack Vogel wrote: > > On Tue, Oct 25, 2011 at 12:22 AM, Sergey Saley <sergeysaley@>wrote: > >> >> Ryan Stone-2 wrote: >> > >> > On Mon, Oct 24, 2011 at 3:51 PM, Sergey Saley <sergeysaley@> >> wrote: >> >> MPD5, netgraph, pppoe.Types of traffic - any (customer traffic). >> >> Bying this card I counted on a 3-4G traffic at 3-4K pppoe sessions. >> >> It turned to 600-700Mbit/s, about 50K pps at 700-800 pppoe sessions. >> > >> > PPPoE is your problem. The Intel cards can't load-balance PPPoE >> > traffic, so everything goes to one queue. It may be possible to write >> > a netgraph module to load-balance the traffic across your CPUs. >> > >> >> OK, thank You for explanation. >> And what about the large number of interrupts? >> As for me, it's too much... >> irq256: ix0:que 0 240536944 6132 >> irq257: ix0:que 1 89090444 2271 >> irq258: ix0:que 2 93222085 2376 >> irq259: ix0:que 3 89435179 2280 >> irq260: ix0:link 1 0 >> irq261: ix1:que 0 269468769 6870 >> irq262: ix1:que 1 110974 2 >> irq263: ix1:que 2 434214 11 >> irq264: ix1:que 3 112281 2 >> irq265: ix1:link 1 0 >> >> > How do you decide its 'too much' ? It may be that with your traffic you > end > up > not being able to use offloads, just thinking. Its not like the hardware > just "makes > it up", it interrupts on the last descriptor of a packet which has the RS > bit set. > With TSO you will get larger chunks of data and thus less interrupts but > your > traffic probably doesn't qualify for it. > It's easy. I have several servers with a similar task and load. About 30K pps, about 500-600M traffic, about 600-700 pppoe connections. One difference - em Here is a typical vmstat -i point06# vmstat -i interrupt total rate irq17: atapci0 6173367 0 cpu0: timer 3904389748 465 irq256: em0 3754877950 447 irq257: em1 2962728160 352 cpu2: timer 3904389720 465 cpu1: timer 3904389720 465 cpu3: timer 3904389721 465 Total 22341338386 2661 point05# vmstat -i interrupt total rate irq14: ata0 35 0 irq19: atapci1 8323568 0 cpu0: timer 3905440143 465 irq256: em0 3870403571 461 irq257: em1 1541695487 183 cpu1: timer 3905439895 465 cpu3: timer 3905439895 465 cpu2: timer 3905439895 465 Total 21042182489 2506 point04# vmstat -i interrupt total rate irq19: atapci0 6047874 0 cpu0: timer 3901683760 464 irq256: em0 823774953 98 irq257: em1 1340659093 159 cpu1: timer 3901683730 464 cpu2: timer 3901683730 464 cpu3: timer 3901683730 464 Total 17777216870 2117 -- View this message in context: http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4935427.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 07:01:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD90E1065674 for ; Tue, 25 Oct 2011 07:01:46 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: from fallback5.mail.ru (fallback5.mail.ru [94.100.176.59]) by mx1.freebsd.org (Postfix) with ESMTP id 86E068FC14 for ; Tue, 25 Oct 2011 07:01:46 +0000 (UTC) Received: from f197.mail.ru (f197.mail.ru [217.69.129.206]) by fallback5.mail.ru (mPOP.Fallback_MX) with ESMTP id 3819D71D30F3 for ; Tue, 25 Oct 2011 10:44:52 +0400 (MSK) Received: from mail by f197.mail.ru with local id 1RIaki-0002pq-00 for freebsd-net@freebsd.org; Tue, 25 Oct 2011 10:44:48 +0400 Received: from [77.45.196.1] by e.mail.ru with HTTP; Tue, 25 Oct 2011 10:44:48 +0400 From: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= To: freebsd-net@freebsd.org Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [77.45.196.1] Date: Tue, 25 Oct 2011 10:44:48 +0400 X-Priority: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Message-Id: X-Mras: Ok X-Mras: Ok X-Mailman-Approved-At: Tue, 25 Oct 2011 11:12:54 +0000 Subject: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 07:01:46 -0000 SGkgQUxMICEgIElmIEkgY29ubmVjdCBnaWdhYml0IHN3aXRjaCB0byBteSBjYXJkIC0gc3lzdGVt IGhhbmcgdW50aWwgSSB1bnBsdWcgcGF0Y2hjb3JkIGZyb20gZGV2aWNlLgpXaXRoIDEwME1iaXQg c3dpdGNoIGNhcmQgd29yayBnb29kLgpTeXN0ZW06ICBDdXJyZW50IC0gcjIyNjE2MwoK From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 12:40:11 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 50FD6106567B for ; Tue, 25 Oct 2011 12:40:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 20B988FC1F for ; Tue, 25 Oct 2011 12:40:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9PCeBYX094661 for ; Tue, 25 Oct 2011 12:40:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9PCeBX1094660; Tue, 25 Oct 2011 12:40:11 GMT (envelope-from gnats) Date: Tue, 25 Oct 2011 12:40:11 GMT Message-Id: <201110251240.p9PCeBX1094660@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: rozhuk.im@gmail.com Cc: Subject: Re: kern/161908: [netgraph] [patch] ng_vlan update for QinQ support X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rozhuk.im@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 12:40:11 -0000 The following reply was made to PR kern/161908; it has been noted by GNATS. From: rozhuk.im@gmail.com To: , Cc: Subject: Re: kern/161908: [netgraph] [patch] ng_vlan update for QinQ support Date: Tue, 25 Oct 2011 21:38:40 +0900 This is a multi-part message in MIME format. ------=_NextPart_000_033B_01CC935E.7927C480 Content-Type: text/plain; charset="windows-1251" Content-Transfer-Encoding: quoted-printable Fixed ng_parce_types in struct ng_cmdlist ng_vlan_cmdlist: "delvidflt": ng_parse_int16_type -> ng_parse_uint16_type "getencap": ng_parse_int32_type -> ng_parse_uint32_type "setencap": ng_parse_int32_type -> ng_parse_uint32_type "getencapproto": ng_parse_int32_type -> ng_parse_uint16_type "setencapproto": ng_parse_int32_type -> ng_parse_uint16_type -- Rozhuk Ivan =A0=20 ------=_NextPart_000_033B_01CC935E.7927C480 Content-Type: application/octet-stream; name="ng_vlan.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="ng_vlan.patch" --- /usr/src/sys/netgraph/ng_vlan.c.orig 2009-08-03 17:13:06.000000000 = +0900=0A= +++ /usr/src/sys/netgraph/ng_vlan.c 2011-10-23 03:41:00.000000000 +0900=0A= @@ -1,5 +1,6 @@=0A= /*-=0A= * Copyright (c) 2003 IPNET Internet Communication Company=0A= + * Copyright (c) 2011 Rozhuk Ivan =0A= * All rights reserved.=0A= *=0A= * Redistribution and use in source and binary forms, with or without=0A= @@ -46,6 +47,22 @@=0A= #include =0A= #include =0A= =0A= +=0A= +=0A= +struct ng_vlan_private {=0A= + hook_p downstream_hook;=0A= + hook_p nomatch_hook;=0A= + int encap_enable;=0A= + u_int16_t encap_proto;=0A= + hook_p vlan_hook[(EVL_VLID_MASK + 1)];=0A= +};=0A= +typedef struct ng_vlan_private *priv_p;=0A= +=0A= +#define VLAN_TAG_MASK 0xFFFF=0A= +#define HOOK_VLAN_TAG_SET_MASK ((uintptr_t)((~0) & ~(VLAN_TAG_MASK)))=0A= +#define IS_HOOK_VLAN_SET(hook_data) ((((uintptr_t)hook_data) & = HOOK_VLAN_TAG_SET_MASK) =3D=3D HOOK_VLAN_TAG_SET_MASK)=0A= +=0A= +=0A= static ng_constructor_t ng_vlan_constructor;=0A= static ng_rcvmsg_t ng_vlan_rcvmsg;=0A= static ng_shutdown_t ng_vlan_shutdown;=0A= @@ -105,11 +122,46 @@=0A= },=0A= {=0A= NGM_VLAN_COOKIE,=0A= + NGM_VLAN_DEL_VID_FLT,=0A= + "delvidflt",=0A= + &ng_parse_uint16_type,=0A= + NULL=0A= + },=0A= + {=0A= + NGM_VLAN_COOKIE,=0A= NGM_VLAN_GET_TABLE,=0A= "gettable",=0A= NULL,=0A= &ng_vlan_table_type=0A= },=0A= + {=0A= + NGM_VLAN_COOKIE,=0A= + NGM_VLAN_GET_ENCAP,=0A= + "getencap",=0A= + NULL,=0A= + &ng_parse_uint32_type=0A= + },=0A= + {=0A= + NGM_VLAN_COOKIE,=0A= + NGM_VLAN_SET_ENCAP,=0A= + "setencap",=0A= + &ng_parse_uint32_type,=0A= + NULL=0A= + },=0A= + {=0A= + NGM_VLAN_COOKIE,=0A= + NGM_VLAN_GET_ENCAP_PROTO,=0A= + "getencapproto",=0A= + NULL,=0A= + &ng_parse_uint16_type=0A= + },=0A= + {=0A= + NGM_VLAN_COOKIE,=0A= + NGM_VLAN_SET_ENCAP_PROTO,=0A= + "setencapproto",=0A= + &ng_parse_uint16_type,=0A= + NULL=0A= + },=0A= { 0 }=0A= };=0A= =0A= @@ -126,47 +178,43 @@=0A= };=0A= NETGRAPH_INIT(vlan, &ng_vlan_typestruct);=0A= =0A= -struct filter {=0A= - LIST_ENTRY(filter) next;=0A= - u_int16_t vlan;=0A= - hook_p hook;=0A= -};=0A= =0A= -#define HASHSIZE 16=0A= -#define HASH(id) ((((id) >> 8) ^ ((id) >> 4) ^ (id)) & 0x0f)=0A= -LIST_HEAD(filterhead, filter);=0A= =0A= -typedef struct {=0A= - hook_p downstream_hook;=0A= - hook_p nomatch_hook;=0A= - struct filterhead hashtable[HASHSIZE];=0A= - u_int32_t nent;=0A= -} *priv_p;=0A= +//**********************************************************************= **=0A= +// HELPER STUFF=0A= +//**********************************************************************= **=0A= =0A= -static struct filter *=0A= -ng_vlan_findentry(priv_p priv, u_int16_t vlan)=0A= +static __inline int=0A= +m_chk(struct mbuf **mp, int len)=0A= {=0A= - struct filterhead *chain =3D &priv->hashtable[HASH(vlan)];=0A= - struct filter *f;=0A= + if ((*mp)->m_pkthdr.len < len) {=0A= + m_freem((*mp));=0A= + (*mp) =3D NULL;=0A= + return (EINVAL);=0A= + }=0A= + if ((*mp)->m_len < len && ((*mp) =3D m_pullup((*mp), len)) =3D=3D NULL)=0A= + return (ENOBUFS);=0A= =0A= - LIST_FOREACH(f, chain, next)=0A= - if (f->vlan =3D=3D vlan)=0A= - return (f);=0A= - return (NULL);=0A= +return (0);=0A= }=0A= =0A= +//**********************************************************************= **=0A= +// NETGRAPH NODE STUFF=0A= +//**********************************************************************= **=0A= +=0A= static int=0A= ng_vlan_constructor(node_p node)=0A= {=0A= priv_p priv;=0A= - int i;=0A= =0A= priv =3D malloc(sizeof(*priv), M_NETGRAPH, M_NOWAIT | M_ZERO);=0A= if (priv =3D=3D NULL)=0A= return (ENOMEM);=0A= - for (i =3D 0; i < HASHSIZE; i++)=0A= - LIST_INIT(&priv->hashtable[i]);=0A= + priv->encap_enable =3D 1;=0A= + priv->encap_proto =3D htons(ETHERTYPE_VLAN);=0A= + =0A= NG_NODE_SET_PRIVATE(node, priv);=0A= +=0A= return (0);=0A= }=0A= =0A= @@ -193,13 +241,14 @@=0A= ng_vlan_rcvmsg(node_p node, item_p item, hook_p lasthook)=0A= {=0A= const priv_p priv =3D NG_NODE_PRIVATE(node);=0A= - int error =3D 0;=0A= struct ng_mesg *msg, *resp =3D NULL;=0A= struct ng_vlan_filter *vf;=0A= - struct filter *f;=0A= hook_p hook;=0A= struct ng_vlan_table *t;=0A= - int i;=0A= + uintptr_t hook_data;=0A= + int i, vlan_count;=0A= + u_int16_t vid;=0A= + int error =3D 0;=0A= =0A= NGI_GET_MSG(item, msg);=0A= /* Deal with message according to cookie and command. */=0A= @@ -214,12 +263,12 @@=0A= }=0A= vf =3D (struct ng_vlan_filter *)msg->data;=0A= /* Sanity check the VLAN ID value. */=0A= - if (vf->vlan & ~EVL_VLID_MASK) {=0A= + if (vf->vid & ~EVL_VLID_MASK || vf->pcp & ~7 || vf->cfi & ~1) {=0A= error =3D EINVAL;=0A= break;=0A= }=0A= /* Check that a referenced hook exists. */=0A= - hook =3D ng_findhook(node, vf->hook);=0A= + hook =3D ng_findhook(node, vf->hook_name);=0A= if (hook =3D=3D NULL) {=0A= error =3D ENOENT;=0A= break;=0A= @@ -231,30 +280,18 @@=0A= break;=0A= }=0A= /* And is not already in service. */=0A= - if (NG_HOOK_PRIVATE(hook) !=3D NULL) {=0A= + if (IS_HOOK_VLAN_SET(NG_HOOK_PRIVATE(hook))) {=0A= error =3D EEXIST;=0A= break;=0A= }=0A= /* Check we don't already trap this VLAN. */=0A= - if (ng_vlan_findentry(priv, vf->vlan)) {=0A= + if (priv->vlan_hook[vf->vid] !=3D NULL) {=0A= error =3D EEXIST;=0A= break;=0A= }=0A= - /* Create filter. */=0A= - f =3D malloc(sizeof(*f),=0A= - M_NETGRAPH, M_NOWAIT | M_ZERO);=0A= - if (f =3D=3D NULL) {=0A= - error =3D ENOMEM;=0A= - break;=0A= - }=0A= - /* Link filter and hook together. */=0A= - f->hook =3D hook;=0A= - f->vlan =3D vf->vlan;=0A= - NG_HOOK_SET_PRIVATE(hook, f);=0A= - /* Register filter in a hash table. */=0A= - LIST_INSERT_HEAD(=0A= - &priv->hashtable[HASH(f->vlan)], f, next);=0A= - priv->nent++;=0A= + /* Link vlan and hook together. */=0A= + NG_HOOK_SET_PRIVATE(hook, (void *)(HOOK_VLAN_TAG_SET_MASK | = EVL_MAKETAG(vf->vid, vf->pcp, vf->cfi)));=0A= + priv->vlan_hook[vf->vid] =3D hook;=0A= break;=0A= case NGM_VLAN_DEL_FILTER:=0A= /* Check that message is long enough. */=0A= @@ -264,35 +301,125 @@=0A= }=0A= /* Check that hook exists and is active. */=0A= hook =3D ng_findhook(node, (char *)msg->data);=0A= - if (hook =3D=3D NULL ||=0A= - (f =3D NG_HOOK_PRIVATE(hook)) =3D=3D NULL) {=0A= + if (hook =3D=3D NULL) {=0A= + error =3D ENOENT;=0A= + break;=0A= + }=0A= + hook_data =3D (uintptr_t)NG_HOOK_PRIVATE(hook);=0A= + if (IS_HOOK_VLAN_SET(hook_data) =3D=3D 0) {=0A= + error =3D ENOENT;=0A= + break;=0A= + }=0A= +#ifdef NETGRAPH_DEBUG=0A= + if (priv->vlan_hook[EVL_VLANOFTAG(hook_data)] !=3D hook)=0A= + printf("%s: NGM_VLAN_DEL_FILTER: Invalid VID for Hook =3D %s\n", = __func__, (char *)msg->data);=0A= +#endif=0A= + /* Purge a rule that refers to this hook. */=0A= + priv->vlan_hook[EVL_VLANOFTAG(hook_data)] =3D NULL;=0A= + NG_HOOK_SET_PRIVATE(hook, NULL);=0A= + break;=0A= + case NGM_VLAN_DEL_VID_FLT:=0A= + /* Check that message is long enough. */=0A= + if (msg->header.arglen !=3D sizeof(u_int16_t)) {=0A= + error =3D EINVAL;=0A= + break;=0A= + }=0A= + vid =3D (*((u_int16_t *)msg->data));=0A= + /* Sanity check the VLAN ID value. */=0A= + if (vid & ~EVL_VLID_MASK) {=0A= + error =3D EINVAL;=0A= + break;=0A= + }=0A= + /* Check that hook exists and is active. */=0A= + hook =3D priv->vlan_hook[vid];=0A= + if (hook =3D=3D NULL) {=0A= + error =3D ENOENT;=0A= + break;=0A= + }=0A= + hook_data =3D (uintptr_t)NG_HOOK_PRIVATE(hook);=0A= + if (IS_HOOK_VLAN_SET(hook_data) =3D=3D 0) {=0A= error =3D ENOENT;=0A= break;=0A= }=0A= +#ifdef NETGRAPH_DEBUG=0A= + if (EVL_VLANOFTAG(hook_data) !=3D vid)=0A= + printf("%s: NGM_VLAN_DEL_VID_FLT: Invalid VID Hook =3D %us, must = be: %us\n", __func__, (u_int16_t)EVL_VLANOFTAG(hook_data), vid);=0A= +#endif=0A= /* Purge a rule that refers to this hook. */=0A= + priv->vlan_hook[vid] =3D NULL;=0A= NG_HOOK_SET_PRIVATE(hook, NULL);=0A= - LIST_REMOVE(f, next);=0A= - priv->nent--;=0A= - free(f, M_NETGRAPH);=0A= break;=0A= case NGM_VLAN_GET_TABLE:=0A= + /* calculate vlans */=0A= + vlan_count =3D 0;=0A= + for (i =3D 0; i < (EVL_VLID_MASK + 1); i ++) {=0A= + if (priv->vlan_hook[i] !=3D NULL=0A= + && NG_HOOK_IS_VALID(priv->vlan_hook[i]))=0A= + vlan_count ++;=0A= + }=0A= +=0A= + /* allocate memory for responce */=0A= NG_MKRESPONSE(resp, msg, sizeof(*t) +=0A= - priv->nent * sizeof(*t->filter), M_NOWAIT);=0A= + vlan_count * sizeof(*t->filter), M_NOWAIT);=0A= if (resp =3D=3D NULL) {=0A= error =3D ENOMEM;=0A= break;=0A= }=0A= +=0A= + /* pack data to responce */=0A= t =3D (struct ng_vlan_table *)resp->data;=0A= - t->n =3D priv->nent;=0A= + t->n =3D 0;=0A= vf =3D &t->filter[0];=0A= - for (i =3D 0; i < HASHSIZE; i++) {=0A= - LIST_FOREACH(f, &priv->hashtable[i], next) {=0A= - vf->vlan =3D f->vlan;=0A= - strncpy(vf->hook, NG_HOOK_NAME(f->hook),=0A= + for (i =3D 0; i < (EVL_VLID_MASK + 1); i ++) {=0A= + hook =3D priv->vlan_hook[i];=0A= + if (hook =3D=3D NULL=0A= + || NG_HOOK_NOT_VALID(hook))=0A= + continue;=0A= + hook_data =3D (uintptr_t)NG_HOOK_PRIVATE(hook);=0A= + if (IS_HOOK_VLAN_SET(hook_data) =3D=3D 0)=0A= + continue;=0A= +#ifdef NETGRAPH_DEBUG=0A= + if (EVL_VLANOFTAG(hook_data) !=3D i)=0A= + printf("%s: NGM_VLAN_GET_TABLE: hook %s VID =3D %us, must be: = %i\n", __func__, NG_HOOK_NAME(hook), = (u_int16_t)EVL_VLANOFTAG(hook_data), i);=0A= +#endif=0A= + vf->vid =3D i;=0A= + vf->pcp =3D EVL_PRIOFTAG(hook_data);=0A= + vf->cfi =3D EVL_CFIOFTAG(hook_data);=0A= + strncpy(vf->hook_name, NG_HOOK_NAME(hook),=0A= NG_HOOKSIZ);=0A= - vf++;=0A= - }=0A= + vf ++;=0A= + t->n ++;=0A= + }=0A= + break;=0A= + case NGM_VLAN_GET_ENCAP:=0A= + NG_MKRESPONSE(resp, msg, sizeof(u_int32_t), M_NOWAIT);=0A= + if (resp =3D=3D NULL) {=0A= + error =3D ENOMEM;=0A= + break;=0A= + }=0A= + (*((u_int32_t *)resp->data)) =3D priv->encap_enable;=0A= + break;=0A= + case NGM_VLAN_SET_ENCAP:=0A= + if (msg->header.arglen !=3D sizeof(u_int32_t)) {=0A= + error =3D EINVAL;=0A= + break;=0A= + }=0A= + priv->encap_enable =3D ((*((u_int32_t *)msg->data)) !=3D 0);=0A= + break;=0A= + case NGM_VLAN_GET_ENCAP_PROTO:=0A= + NG_MKRESPONSE(resp, msg, sizeof(u_int16_t), M_NOWAIT);=0A= + if (resp =3D=3D NULL) {=0A= + error =3D ENOMEM;=0A= + break;=0A= + }=0A= + (*((u_int16_t *)resp->data)) =3D ntohs(priv->encap_proto);=0A= + break;=0A= + case NGM_VLAN_SET_ENCAP_PROTO:=0A= + if (msg->header.arglen !=3D sizeof(u_int16_t)) {=0A= + error =3D EINVAL;=0A= + break;=0A= }=0A= + priv->encap_proto =3D htons((*((u_int16_t *)msg->data)));=0A= break;=0A= default: /* Unknown command. */=0A= error =3D EINVAL;=0A= @@ -302,8 +429,6 @@=0A= case NGM_FLOW_COOKIE:=0A= {=0A= struct ng_mesg *copy;=0A= - struct filterhead *chain;=0A= - struct filter *f;=0A= =0A= /*=0A= * Flow control messages should come only=0A= @@ -314,17 +439,16 @@=0A= break;=0A= if (lasthook !=3D priv->downstream_hook)=0A= break;=0A= -=0A= /* Broadcast the event to all uplinks. */=0A= - for (i =3D 0, chain =3D priv->hashtable; i < HASHSIZE;=0A= - i++, chain++)=0A= - LIST_FOREACH(f, chain, next) {=0A= + for (i =3D 0; i < (EVL_VLID_MASK + 1); i ++) {=0A= + if (priv->vlan_hook[i] =3D=3D NULL)=0A= + continue;=0A= +=0A= NG_COPYMESSAGE(copy, msg, M_NOWAIT);=0A= if (copy =3D=3D NULL)=0A= - continue;=0A= - NG_SEND_MSG_HOOK(error, node, copy, f->hook, 0);=0A= + continue;=0A= + NG_SEND_MSG_HOOK(error, node, copy, priv->vlan_hook[i], 0);=0A= }=0A= -=0A= break;=0A= }=0A= default: /* Unknown type cookie. */=0A= @@ -343,16 +467,19 @@=0A= struct ether_header *eh;=0A= struct ether_vlan_header *evl =3D NULL;=0A= int error;=0A= - u_int16_t vlan;=0A= + uintptr_t hook_data;=0A= + u_int16_t vid;=0A= struct mbuf *m;=0A= - struct filter *f;=0A= + hook_p dst_hook;=0A= +=0A= =0A= - /* Make sure we have an entire header. */=0A= NGI_GET_M(item, m);=0A= - if (m->m_len < sizeof(*eh) &&=0A= - (m =3D m_pullup(m, sizeof(*eh))) =3D=3D NULL) {=0A= +=0A= + /* Make sure we have an entire header. */=0A= + error =3D m_chk(&m, ETHER_HDR_LEN);=0A= + if (error !=3D 0) {=0A= NG_FREE_ITEM(item);=0A= - return (EINVAL);=0A= + return (error);=0A= }=0A= eh =3D mtod(m, struct ether_header *);=0A= if (hook =3D=3D priv->downstream_hook) {=0A= @@ -360,75 +487,104 @@=0A= * If from downstream, select between a match hook=0A= * or the nomatch hook.=0A= */=0A= + dst_hook =3D priv->nomatch_hook;=0A= if (m->m_flags & M_VLANTAG ||=0A= - eh->ether_type =3D=3D htons(ETHERTYPE_VLAN)) {=0A= + eh->ether_type =3D=3D priv->encap_proto) {=0A= if (m->m_flags & M_VLANTAG) {=0A= /*=0A= * Packet is tagged, m contains a normal=0A= * Ethernet frame; tag is stored out-of-band.=0A= */=0A= - vlan =3D EVL_VLANOFTAG(m->m_pkthdr.ether_vtag);=0A= - } else {=0A= - if (m->m_len < sizeof(*evl) &&=0A= - (m =3D m_pullup(m, sizeof(*evl))) =3D=3D NULL) {=0A= + vid =3D EVL_VLANOFTAG(m->m_pkthdr.ether_vtag);=0A= + } else { /* eh->ether_type =3D=3D priv->encap_proto */=0A= + error =3D m_chk(&m, (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN));=0A= + if (error !=3D 0) {=0A= NG_FREE_ITEM(item);=0A= - return (EINVAL);=0A= + return (error);=0A= }=0A= evl =3D mtod(m, struct ether_vlan_header *);=0A= - vlan =3D EVL_VLANOFTAG(ntohs(evl->evl_tag));=0A= + vid =3D EVL_VLANOFTAG(ntohs(evl->evl_tag));=0A= }=0A= - if ((f =3D ng_vlan_findentry(priv, vlan)) !=3D NULL) {=0A= +=0A= + if (priv->vlan_hook[vid] !=3D NULL) {=0A= + dst_hook =3D priv->vlan_hook[vid];=0A= if (m->m_flags & M_VLANTAG) {=0A= m->m_pkthdr.ether_vtag =3D 0;=0A= m->m_flags &=3D ~M_VLANTAG;=0A= } else {=0A= - evl->evl_encap_proto =3D evl->evl_proto;=0A= - bcopy(mtod(m, caddr_t),=0A= - mtod(m, caddr_t) +=0A= - ETHER_VLAN_ENCAP_LEN,=0A= - ETHER_HDR_LEN);=0A= + /* =0A= + * move DstMAC and SrcMAC to ETHER_TYPE=0A= + * before: [dst_mac] [src_mac] [ether_type_encap(TPID)] = [PCP/CFI/VID] [ether_type] [payload]=0A= + * |-----------------| = >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> |---------------------|=0A= + * after: [free space] [dst_mac] [src_mac] [ether_type] [payload]=0A= + * |-----------------| |---------------------|=0A= + */=0A= + bcopy((char *)evl, ((char *)evl + ETHER_VLAN_ENCAP_LEN),=0A= + (ETHER_ADDR_LEN * 2));=0A= m_adj(m, ETHER_VLAN_ENCAP_LEN);=0A= }=0A= }=0A= - } else=0A= - f =3D NULL;=0A= - if (f !=3D NULL)=0A= - NG_FWD_NEW_DATA(error, item, f->hook, m);=0A= - else=0A= - NG_FWD_NEW_DATA(error, item, priv->nomatch_hook, m);=0A= + }=0A= } else {=0A= /*=0A= * It is heading towards the downstream.=0A= * If from nomatch, pass it unmodified.=0A= * Otherwise, do the VLAN encapsulation.=0A= */=0A= - if (hook !=3D priv->nomatch_hook) {=0A= - if ((f =3D NG_HOOK_PRIVATE(hook)) =3D=3D NULL) {=0A= + dst_hook =3D priv->downstream_hook;=0A= + if (dst_hook !=3D NULL && hook !=3D priv->nomatch_hook) {=0A= + hook_data =3D (uintptr_t)NG_HOOK_PRIVATE(hook);=0A= + if (IS_HOOK_VLAN_SET(hook_data) =3D=3D 0) {=0A= + m_freem(m);=0A= NG_FREE_ITEM(item);=0A= - NG_FREE_M(m);=0A= return (EOPNOTSUPP);=0A= }=0A= - M_PREPEND(m, ETHER_VLAN_ENCAP_LEN, M_DONTWAIT);=0A= - /* M_PREPEND takes care of m_len and m_pkthdr.len. */=0A= - if (m =3D=3D NULL || (m->m_len < sizeof(*evl) &&=0A= - (m =3D m_pullup(m, sizeof(*evl))) =3D=3D NULL)) {=0A= - NG_FREE_ITEM(item);=0A= - return (ENOMEM);=0A= + if (priv->encap_enable =3D=3D 0) {=0A= + /* just set packet header tag */=0A= + m->m_flags |=3D M_VLANTAG;=0A= + m->m_pkthdr.ether_vtag =3D (hook_data & VLAN_TAG_MASK);=0A= + } else {=0A= + /*=0A= + * Transform the Ethernet header into an Ethernet header=0A= + * with 802.1Q encapsulation.=0A= + * mod of: ether_vlanencap =0A= + */=0A= + M_PREPEND(m, ETHER_VLAN_ENCAP_LEN, M_DONTWAIT);=0A= + /* M_PREPEND takes care of m_len and m_pkthdr.len. */=0A= + if (m =3D=3D NULL)=0A= + error =3D ENOMEM;=0A= + else=0A= + error =3D m_chk(&m, (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN));=0A= + if (error !=3D 0) {=0A= + NG_FREE_ITEM(item);=0A= + return (error);=0A= + }=0A= + /* move DstMAC and SrcMAC from ETHER_TYPE=0A= + * before: [free - prepended space] [dst_mac] [src_mac] = [ether_type] [payload]=0A= + * <<<<<<<<<<<<<<<<<<<<<< |-----------------| = |--------------------|=0A= + * after: [dst_mac] [src_mac] [ether_type_encap(TPID)] = [PCP/CFI/VID] [ether_type] [payload]=0A= + * |-----------------| |----------- inserted tag = -----------| |--------------------| =0A= + */=0A= + evl =3D mtod(m, struct ether_vlan_header *);=0A= + bcopy(((char *)evl + ETHER_VLAN_ENCAP_LEN),=0A= + (char *)evl, (ETHER_ADDR_LEN * 2));=0A= + evl->evl_encap_proto =3D priv->encap_proto;=0A= + evl->evl_tag =3D htons((hook_data & VLAN_TAG_MASK));=0A= }=0A= - /*=0A= - * Transform the Ethernet header into an Ethernet header=0A= - * with 802.1Q encapsulation.=0A= - */=0A= - bcopy(mtod(m, char *) + ETHER_VLAN_ENCAP_LEN,=0A= - mtod(m, char *), ETHER_HDR_LEN);=0A= - evl =3D mtod(m, struct ether_vlan_header *);=0A= - evl->evl_proto =3D evl->evl_encap_proto;=0A= - evl->evl_encap_proto =3D htons(ETHERTYPE_VLAN);=0A= - evl->evl_tag =3D htons(f->vlan);=0A= }=0A= - NG_FWD_NEW_DATA(error, item, priv->downstream_hook, m);=0A= }=0A= - return (error);=0A= +=0A= + /* send packet */=0A= + if (dst_hook !=3D NULL) {=0A= + NG_FWD_NEW_DATA(error, item, dst_hook, m);=0A= + return (error);=0A= + }=0A= +=0A= + /* no hook to send */=0A= + m_freem(m);=0A= + NG_FREE_ITEM(item);=0A= +=0A= + return (ENETDOWN);=0A= }=0A= =0A= static int=0A= @@ -446,7 +602,7 @@=0A= ng_vlan_disconnect(hook_p hook)=0A= {=0A= const priv_p priv =3D NG_NODE_PRIVATE(NG_HOOK_NODE(hook));=0A= - struct filter *f;=0A= + uintptr_t hook_data;=0A= =0A= if (hook =3D=3D priv->downstream_hook)=0A= priv->downstream_hook =3D NULL;=0A= @@ -454,11 +610,9 @@=0A= priv->nomatch_hook =3D NULL;=0A= else {=0A= /* Purge a rule that refers to this hook. */=0A= - if ((f =3D NG_HOOK_PRIVATE(hook)) !=3D NULL) {=0A= - LIST_REMOVE(f, next);=0A= - priv->nent--;=0A= - free(f, M_NETGRAPH);=0A= - }=0A= + hook_data =3D (uintptr_t)NG_HOOK_PRIVATE(hook);=0A= + if (IS_HOOK_VLAN_SET(hook_data))=0A= + priv->vlan_hook[EVL_VLANOFTAG(hook_data)] =3D NULL;=0A= }=0A= NG_HOOK_SET_PRIVATE(hook, NULL);=0A= if ((NG_NODE_NUMHOOKS(NG_HOOK_NODE(hook)) =3D=3D 0) &&=0A= =0A= =0A= --- /usr/src/sys/netgraph/ng_vlan.h.orig 2009-08-03 17:13:06.000000000 = +0900=0A= +++ /usr/src/sys/netgraph/ng_vlan.h 2011-10-22 19:11:01.000000000 +0900=0A= @@ -1,5 +1,6 @@=0A= /*-=0A= * Copyright (c) 2003 IPNET Internet Communication Company=0A= + * Copyright (c) 2011 Rozhuk Ivan =0A= * All rights reserved.=0A= *=0A= * Redistribution and use in source and binary forms, with or without=0A= @@ -43,19 +44,28 @@=0A= enum {=0A= NGM_VLAN_ADD_FILTER =3D 1,=0A= NGM_VLAN_DEL_FILTER,=0A= - NGM_VLAN_GET_TABLE=0A= + NGM_VLAN_DEL_VID_FLT,=0A= + NGM_VLAN_GET_TABLE,=0A= + NGM_VLAN_GET_ENCAP,=0A= + NGM_VLAN_SET_ENCAP,=0A= + NGM_VLAN_GET_ENCAP_PROTO,=0A= + NGM_VLAN_SET_ENCAP_PROTO,=0A= };=0A= =0A= /* For NGM_VLAN_ADD_FILTER control message. */=0A= struct ng_vlan_filter {=0A= - char hook[NG_HOOKSIZ];=0A= - u_int16_t vlan;=0A= -}; =0A= + char hook_name[NG_HOOKSIZ];=0A= + u_int16_t vid; /* VID - VLAN Identifier */=0A= + u_int8_t pcp; /* PCP - Priority Code Point */=0A= + u_int8_t cfi; /* CFI - Canonical Format Indicator */=0A= +};=0A= =0A= /* Keep this in sync with the above structure definition. */=0A= #define NG_VLAN_FILTER_FIELDS { \=0A= - { "hook", &ng_parse_hookbuf_type }, \=0A= - { "vlan", &ng_parse_uint16_type }, \=0A= + { "hook", &ng_parse_hookbuf_type }, \=0A= + { "vid", &ng_parse_uint16_type }, \=0A= + { "pcp", &ng_parse_uint8_type }, \=0A= + { "cfi", &ng_parse_uint8_type }, \=0A= { NULL } \=0A= }=0A= =0A= ------=_NextPart_000_033B_01CC935E.7927C480-- From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 18:41:09 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE9BC106564A for ; Tue, 25 Oct 2011 18:41:09 +0000 (UTC) (envelope-from smahood@uwaterloo.ca) Received: from ecserv7.uwaterloo.ca (ecserv7.uwaterloo.ca [129.97.50.127]) by mx1.freebsd.org (Postfix) with ESMTP id 8A75A8FC0A for ; Tue, 25 Oct 2011 18:41:09 +0000 (UTC) Received: from ecserv7.uwaterloo.ca (localhost [127.0.0.1]) by ecserv7.uwaterloo.ca (8.14.3/8.14.3) with ESMTP id p9PIHk12012106 for ; Tue, 25 Oct 2011 14:17:46 -0400 (EDT) (envelope-from smahood@uwaterloo.ca) Received: (from www@localhost) by ecserv7.uwaterloo.ca (8.14.3/8.14.3/Submit) id p9PIHfbq012105 for freebsd-net@freebsd.org; Tue, 25 Oct 2011 14:17:41 -0400 (EDT) (envelope-from smahood@uwaterloo.ca) X-Authentication-Warning: ecserv7.uwaterloo.ca: www set sender to smahood@uwaterloo.ca using -f Received: from 64.7.137.182 ([64.7.137.182]) by www.nexusmail.uwaterloo.ca (Horde Framework) with HTTP; Tue, 25 Oct 2011 14:17:41 -0400 Message-ID: <20111025141741.26891k8ib38ekmkg@www.nexusmail.uwaterloo.ca> Date: Tue, 25 Oct 2011 14:17:41 -0400 From: Sean Mahood To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_1gddn2ugd0sg" Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.5) / FreeBSD-7.2 X-Originated-By: smahood@engmail.uwaterloo.ca X-Originating-IP: 64.7.137.182 X-Remote-Browser: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1 Subject: [PATCH] Have netstat test for sctp kernel support X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: smahood@engmail.uwaterloo.ca List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 18:41:09 -0000 This message is in MIME format. --=_1gddn2ugd0sg Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit Hello, I noticed that when doing a netstat -s (running on a kernel without SCTP support compiled in), I get the following message output to stderr: netstat: sysctl: net.inet.sctp.stats: No such file or directory Wondering why it would attempt to fetch SCTP stats when it was compiled out of the kernel, I noted that netstat could be compiled without SCTP support as well, but that there is no good way to link the two compilation conditions, or to have a generic binary that can run without error regardless of compiled kernel options. To that end, I added a method to check for environment compatibility in netstat. For SCTP specifically, I used the FEATURE macros and added SCTP as a FEATURE. Other protocols could also be configured to be checked at runtime if desired, though I haven't implemented any others at this time. -- Sean --=_1gddn2ugd0sg Content-Type: text/x-patch; charset=UTF-8; name="main.c.diff" Content-Disposition: attachment; filename="main.c.diff" Content-Transfer-Encoding: 7bit --- main.c@@/main/RELENG_8/RELENG_8_2/plt_ag_os/0 2011-10-21 11:35:33.000000000 -0400 +++ main.c 2011-10-24 14:30:13.000000000 -0400 @@ -198,106 +198,109 @@ struct protox { void (*pr_stats)(u_long, const char *, int, int); /* statistics printing routine */ void (*pr_istats)(char *); /* per/if statistics printing routine */ + int (*feature_test)(void); /* test routine to determine if kernel + suppport exists for protocol */ const char *pr_name; /* well-known name */ int pr_usesysctl; /* non-zero if we use sysctl, not kvm */ int pr_protocol; } protox[] = { { N_TCBINFO, N_TCPSTAT, 1, protopr, - tcp_stats, NULL, "tcp", 1, IPPROTO_TCP }, + tcp_stats, NULL, NULL, "tcp", 1, IPPROTO_TCP }, { N_UDBINFO, N_UDPSTAT, 1, protopr, - udp_stats, NULL, "udp", 1, IPPROTO_UDP }, + udp_stats, NULL, NULL, "udp", 1, IPPROTO_UDP }, #ifdef SCTP { -1, N_SCTPSTAT, 1, sctp_protopr, - sctp_stats, NULL, "sctp", 1, IPPROTO_SCTP }, + sctp_stats, NULL, sctp_feature_test, "sctp", 1, + IPPROTO_SCTP }, #endif { N_DIVCBINFO, -1, 1, protopr, - NULL, NULL, "divert", 1, IPPROTO_DIVERT }, + NULL, NULL, NULL, "divert", 1, IPPROTO_DIVERT }, { N_RIPCBINFO, N_IPSTAT, 1, protopr, - ip_stats, NULL, "ip", 1, IPPROTO_RAW }, + ip_stats, NULL, NULL, "ip", 1, IPPROTO_RAW }, { N_RIPCBINFO, N_ICMPSTAT, 1, protopr, - icmp_stats, NULL, "icmp", 1, IPPROTO_ICMP }, + icmp_stats, NULL, NULL, "icmp", 1, IPPROTO_ICMP }, { N_RIPCBINFO, N_IGMPSTAT, 1, protopr, - igmp_stats, NULL, "igmp", 1, IPPROTO_IGMP }, + igmp_stats, NULL, NULL, "igmp", 1, IPPROTO_IGMP }, #ifdef IPSEC { -1, N_IPSECSTAT, 1, NULL, /* keep as compat */ - ipsec_stats, NULL, "ipsec", 0, 0}, + ipsec_stats, NULL, NULL, "ipsec", 0, 0}, { -1, N_AHSTAT, 1, NULL, - ah_stats, NULL, "ah", 0, 0}, + ah_stats, NULL, NULL, "ah", 0, 0}, { -1, N_ESPSTAT, 1, NULL, - esp_stats, NULL, "esp", 0, 0}, + esp_stats, NULL, NULL, "esp", 0, 0}, { -1, N_IPCOMPSTAT, 1, NULL, - ipcomp_stats, NULL, "ipcomp", 0, 0}, + ipcomp_stats, NULL, NULL, "ipcomp", 0, 0}, #endif { N_RIPCBINFO, N_PIMSTAT, 1, protopr, - pim_stats, NULL, "pim", 1, IPPROTO_PIM }, + pim_stats, NULL, NULL, "pim", 1, IPPROTO_PIM }, { -1, N_CARPSTAT, 1, NULL, - carp_stats, NULL, "carp", 1, 0 }, + carp_stats, NULL, NULL, "carp", 1, 0 }, { -1, N_PFSYNCSTAT, 1, NULL, - pfsync_stats, NULL, "pfsync", 1, 0 }, + pfsync_stats, NULL, NULL, "pfsync", 1, 0 }, { -1, N_ARPSTAT, 1, NULL, - arp_stats, NULL, "arp", 1, 0 }, + arp_stats, NULL, NULL, "arp", 1, 0 }, { -1, -1, 0, NULL, - NULL, NULL, NULL, 0, 0 } + NULL, NULL, NULL, NULL, 0, 0 } }; #ifdef INET6 struct protox ip6protox[] = { { N_TCBINFO, N_TCPSTAT, 1, protopr, - tcp_stats, NULL, "tcp", 1, IPPROTO_TCP }, + tcp_stats, NULL, NULL, "tcp", 1, IPPROTO_TCP }, { N_UDBINFO, N_UDPSTAT, 1, protopr, - udp_stats, NULL, "udp", 1, IPPROTO_UDP }, + udp_stats, NULL, NULL, "udp", 1, IPPROTO_UDP }, { N_RIPCBINFO, N_IP6STAT, 1, protopr, - ip6_stats, ip6_ifstats, "ip6", 1, IPPROTO_RAW }, + ip6_stats, ip6_ifstats, NULL, "ip6", 1, IPPROTO_RAW }, { N_RIPCBINFO, N_ICMP6STAT, 1, protopr, - icmp6_stats, icmp6_ifstats, "icmp6", 1, IPPROTO_ICMPV6 }, + icmp6_stats, icmp6_ifstats, NULL, "icmp6", 1, IPPROTO_ICMPV6 }, #ifdef IPSEC { -1, N_IPSEC6STAT, 1, NULL, - ipsec_stats, NULL, "ipsec6", 0, 0 }, + ipsec_stats, NULL, NULL, "ipsec6", 0, 0 }, #endif #ifdef notyet { -1, N_PIM6STAT, 1, NULL, - pim6_stats, NULL, "pim6", 1, 0 }, + pim6_stats, NULL, NULL, "pim6", 1, 0 }, #endif { -1, N_RIP6STAT, 1, NULL, - rip6_stats, NULL, "rip6", 1, 0 }, + rip6_stats, NULL, NULL, "rip6", 1, 0 }, { -1, -1, 0, NULL, - NULL, NULL, NULL, 0, 0 } + NULL, NULL, NULL, NULL, 0, 0 } }; #endif /*INET6*/ #ifdef IPSEC struct protox pfkeyprotox[] = { { -1, N_PFKEYSTAT, 1, NULL, - pfkey_stats, NULL, "pfkey", 0, 0 }, + pfkey_stats, NULL, NULL, "pfkey", 0, 0 }, { -1, -1, 0, NULL, - NULL, NULL, NULL, 0, 0 } + NULL, NULL, NULL, NULL, 0, 0 } }; #endif struct protox atalkprotox[] = { { N_DDPCB, N_DDPSTAT, 1, atalkprotopr, - ddp_stats, NULL, "ddp", 0, 0 }, + ddp_stats, NULL, NULL, "ddp", 0, 0 }, { -1, -1, 0, NULL, - NULL, NULL, NULL, 0, 0 } + NULL, NULL, NULL, NULL, 0, 0 } }; #ifdef NETGRAPH struct protox netgraphprotox[] = { { N_NGSOCKS, -1, 1, netgraphprotopr, - NULL, NULL, "ctrl", 0, 0 }, + NULL, NULL, NULL, "ctrl", 0, 0 }, { N_NGSOCKS, -1, 1, netgraphprotopr, - NULL, NULL, "data", 0, 0 }, + NULL, NULL, NULL, "data", 0, 0 }, { -1, -1, 0, NULL, - NULL, NULL, NULL, 0, 0 } + NULL, NULL, NULL, NULL, 0, 0 } }; #endif #ifdef IPX struct protox ipxprotox[] = { { N_IPX, N_IPXSTAT, 1, ipxprotopr, - ipx_stats, NULL, "ipx", 0, 0 }, + ipx_stats, NULL, NULL, "ipx", 0, 0 }, { N_IPX, N_SPXSTAT, 1, ipxprotopr, - spx_stats, NULL, "spx", 0, 0 }, + spx_stats, NULL, NULL, "spx", 0, 0 }, { -1, -1, 0, NULL, - NULL, NULL, 0, 0, 0 } + NULL, NULL, NULL, 0, 0, 0 } }; #endif @@ -624,6 +627,12 @@ printproto(tp, name) void (*pr)(u_long, const char *, int, int); u_long off; + if (tp->feature_test != NULL && !(*tp->feature_test)()) { + if (pflag) + fprintf(stderr, "%s: not supported in kernel\n", tp->pr_name); + return; + } + if (sflag) { if (iflag) { if (tp->pr_istats) --=_1gddn2ugd0sg Content-Type: text/x-patch; charset=UTF-8; name="netstat.h.diff" Content-Disposition: attachment; filename="netstat.h.diff" Content-Transfer-Encoding: 7bit --- netstat.h@@/main/RELENG_8/RELENG_8_2/plt_ag_os/0 2011-10-21 12:16:04.000000000 -0400 +++ netstat.h 2011-10-24 14:30:13.000000000 -0400 @@ -73,6 +73,7 @@ void protopr(u_long, const char *, int, void tcp_stats(u_long, const char *, int, int); void udp_stats(u_long, const char *, int, int); #ifdef SCTP +int sctp_feature_test(void); void sctp_protopr(u_long, const char *, int, int); void sctp_stats(u_long, const char *, int, int); #endif --=_1gddn2ugd0sg Content-Type: text/x-patch; charset=UTF-8; name="sctp.c.diff" Content-Disposition: attachment; filename="sctp.c.diff" Content-Transfer-Encoding: 7bit --- sctp.c@@/main/RELENG_8/RELENG_8_2/plt_ag_os/0 2011-10-20 18:00:53.000000000 -0400 +++ sctp.c 2011-10-24 14:30:14.000000000 -0400 @@ -102,6 +102,13 @@ struct xraddr_entry { LIST_ENTRY(xraddr_entry) xraddr_entries; }; +int +sctp_feature_test(void) +{ + + return (feature_present("sctp")); +} + static int sctp_skip_xinpcb_ifneed(char *buf, const size_t buflen, size_t *offset) { --=_1gddn2ugd0sg Content-Type: text/x-patch; charset=UTF-8; name="sctp_sysctl.c.diff" Content-Disposition: attachment; filename="sctp_sysctl.c.diff" Content-Transfer-Encoding: 7bit --- sctp_sysctl.c@@/main/RELENG_8/RELENG_8_2/plt_ag_os/0 2011-10-20 17:36:01.000000000 -0400 +++ sctp_sysctl.c 2011-10-24 14:30:11.000000000 -0400 @@ -40,6 +40,8 @@ __FBSDID("$FreeBSD$"); #include #include +FEATURE(sctp, "Stream Control Transmission Protocol"); + /* * sysctl tunable variables */ --=_1gddn2ugd0sg-- From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 19:48:28 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F17B106566B for ; Tue, 25 Oct 2011 19:48:28 +0000 (UTC) (envelope-from quintessence@bobi.gateit.net) Received: from mail.gateit.net (mail4o.gateit.net [85.187.21.218]) by mx1.freebsd.org (Postfix) with ESMTP id 4D6208FC0C for ; Tue, 25 Oct 2011 19:48:27 +0000 (UTC) Received: from [127.0.0.1] (priv.gateit.net [212.43.51.203]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.gateit.net (Postfix) with ESMTPSA id A87E490800F for ; Tue, 25 Oct 2011 22:28:31 +0300 (EEST) Message-ID: <4EA70DD7.9040207@bobi.gateit.net> Date: Tue, 25 Oct 2011 22:28:23 +0300 From: Bojidara Marinchovska User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Subject: bce huge amount of input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 19:48:28 -0000 Hello, I'm running FreeBSD 8.2-STABLE #1: Thu May 19 15:05:33 EEST 2011 CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz (2400.10-MHz K8-class CPU) real memory = 25769803776 (24576 MB) FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads bce0: mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci2 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bce0: Ethernet address: xx:xx:xx:xx:xx:xx bce0: [ITHREAD] bce0@pci0:2:0:0: class=0x020000 card=0x7055103c chip=0x163914e4 rev=0x20 hdr=0x00 vendor = 'Broadcom Corporation' device = 'NetXtreme II Gigabit Ethernet (BCM5709)' class = network subclass = ethernet When bandwidth hits about 250Mb/s I get ~70 input errors/s, which causes packet loss. bce0: flags=8843 metric 0 mtu 1500 options=c01bb media: Ethernet autoselect (1000baseT ) status: active Settings: loader.conf: kern.hz=200 kern.maxusers=1024 sysctl.conf: kern.ipc.maxsockets=204800 kern.ipc.nmbclusters=131072 kern.ipc.somaxconn=16384 net.inet.tcp.maxtcptw=40960 About 4 million input errors on bce0. Today I changed kern.hz 200 -> 2000 and for ~7 hour uptime I get ~2 million input errors on bce0. vmstat: 1.) -i (fast forwarding is disabled) interrupt total rate irq1: atkbd0 6 0 irq17: atapci0 35 0 irq22: uhci2 uhci4 42643 1 cpu0: timer 48658417 1999 irq256: ciss0 11143030 457 irq257: bce0 198633062 8163 irq258: bce1 16803717 690 cpu1: timer 48649953 1999 cpu11: timer 48649228 1999 cpu3: timer 48649973 1999 cpu10: timer 48649217 1999 cpu6: timer 48649994 1999 cpu15: timer 48649262 1999 cpu14: timer 48649263 1999 cpu2: timer 48649866 1999 cpu12: timer 48649206 1999 cpu4: timer 48649976 1999 cpu8: timer 48649077 1999 cpu7: timer 48649994 1999 cpu9: timer 48649240 1999 cpu5: timer 48649994 1999 cpu13: timer 48649207 1999 Total 1005024360 41304 2.) -z (server does not hit any limit) ITEM SIZE LIMIT USED FREE REQUESTS FAILURES UMA Kegs: 208, 0, 102, 0, 102, 0 UMA Zones: 704, 0, 102, 3, 102, 0 UMA Slabs: 568, 0, 16658, 2, 24197, 0 UMA RCntSlabs: 568, 0, 4830, 0, 4830, 0 UMA Hash: 256, 0, 0, 15, 3, 0 16 Bucket: 152, 0, 192, 8, 192, 0 32 Bucket: 280, 0, 341, 9, 341, 0 64 Bucket: 536, 0, 481, 2, 493, 65 128 Bucket: 1048, 0, 2829, 0, 3226, 118368 VM OBJECT: 216, 0, 467472, 33576, 8255068, 0 MAP: 232, 0, 7, 25, 7, 0 KMAP ENTRY: 120, 827700, 47, 728, 25413, 0 MAP ENTRY: 120, 0, 665651, 3050, 3248202, 0 DP fakepg: 120, 0, 0, 0, 0, 0 SG fakepg: 120, 0, 0, 0, 0, 0 mt_zone: 2056, 0, 256, 7, 256, 0 16: 16, 0, 2837, 2539, 36713029, 0 32: 32, 0, 3229, 2225, 847494, 0 64: 64, 0, 12458, 3670, 769223924, 0 128: 128, 0, 14520, 2358, 7645834, 0 256: 256, 0, 8250, 3330, 2945807, 0 512: 512, 0, 8115, 1069, 7803084, 0 1024: 1024, 0, 75, 917, 124714, 0 2048: 2048, 0, 3961, 1033, 48456, 0 4096: 4096, 0, 4000, 819, 7480557, 0 Files: 80, 0, 11131, 1694, 77493053, 0 TURNSTILE: 136, 0, 4327, 273, 4327, 0 umtx pi: 96, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0 PROC: 1136, 0, 3660, 402, 17921, 0 THREAD: 1120, 0, 4140, 186, 4140, 0 SLEEPQUEUE: 80, 0, 4327, 545, 4327, 0 VMSPACE: 392, 0, 3639, 1101, 17899, 0 cpuset: 72, 0, 2, 98, 2, 0 audit_record: 952, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 1716, 4684, 475185964, 0 mbuf: 256, 0, 2560, 4480, 786007357, 0 mbuf_cluster: 2048, 131072, 6400, 770, 6400, 0 mbuf_jumbo_page: 4096, 33280, 126, 1119, 7766976, 0 mbuf_jumbo_9k: 9216, 16640, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 8320, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 2419, 3461, 152993298, 0 g_bio: 232, 0, 0, 3632, 44700603, 0 ttyinq: 160, 0, 135, 273, 600, 0 ttyoutq: 256, 0, 72, 138, 320, 0 ata_request: 320, 0, 0, 48, 20, 0 ata_composite: 336, 0, 0, 0, 0, 0 VNODE: 472, 0, 392140, 33452, 7823148, 0 VNODEPOLL: 112, 0, 0, 0, 0, 0 S VFS Cache: 108, 0, 404747, 33328, 7828800, 0 L VFS Cache: 328, 0, 450, 594, 4331, 0 NAMEI: 1024, 0, 0, 1024, 327010570, 0 DIRHASH: 1024, 0, 1856, 2872, 17086719, 0 NFSMOUNT: 632, 0, 0, 0, 0, 0 NFSNODE: 688, 0, 0, 0, 0, 0 pipe: 728, 0, 3, 407, 11069, 0 ksiginfo: 112, 0, 4048, 803, 5396, 0 itimer: 344, 0, 0, 0, 0, 0 KNOTE: 128, 0, 3622, 1221, 76994, 0 socket: 680, 204804, 40034, 5380, 18020394, 0 unpcb: 240, 204800, 15, 385, 759, 0 ipq: 56, 4158, 0, 0, 0, 0 udp_inpcb: 336, 204809, 2, 471, 19993, 0 udpcb: 16, 204960, 2, 2182, 19993, 0 tcp_inpcb: 336, 204809, 55675, 7223, 17999640, 0 tcpcb: 880, 204800, 18626, 4718, 17999640, 0 tcptw: 72, 41000, 37049, 3951, 11173499, 409 syncache: 144, 65546, 224, 1934, 17303968, 0 hostcache: 136, 65548, 22382, 3490, 89062, 0 tcpreass: 40, 8232, 162, 2274, 1817401, 0 sackhole: 32, 0, 106, 3328, 39148671, 0 sctp_ep: 1272, 66561, 0, 0, 0, 0 sctp_asoc: 2240, 40000, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 144, 2, 0 sctp_raddr: 600, 80004, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0 sctp_stream_msg_out: 96, 400026, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0 ripcb: 336, 204809, 0, 0, 0, 0 rtentry: 200, 0, 6, 51, 6, 0 pfsrctrpl: 152, 10000, 0, 0, 0, 0 pfrulepl: 912, 0, 20, 8, 20, 0 pfstatepl: 392, 10000, 764, 2416, 224673, 0 pfaltqpl: 240, 0, 0, 0, 0, 0 pfpooladdrpl: 88, 0, 8, 76, 8, 0 pfrktable: 1296, 1002, 2, 4, 3, 0 pfrkentry: 216, 100008, 1, 35, 1, 0 pfrkentry2: 216, 0, 0, 0, 0, 0 pffrent: 32, 5050, 0, 202, 8, 0 pffrag: 80, 0, 0, 90, 4, 0 pffrcache: 80, 10035, 0, 0, 0, 0 pffrcent: 24, 50022, 0, 0, 0, 0 pfstatescrub: 40, 0, 0, 0, 0, 0 pfiaddrpl: 120, 0, 2, 60, 2, 0 pfospfen: 112, 0, 696, 30, 696, 0 pfosfp: 40, 0, 407, 97, 407, 0 selfd: 56, 0, 6834, 3120, 157089498, 0 SWAPMETA: 288, 116519, 0, 0, 0, 0 Mountpoints: 752, 0, 6, 9, 6, 0 FFS inode: 168, 0, 392105, 33617, 7823018, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 392105, 33685, 7822979, 0 netstat 1) -m (a lot of free resources) 5055/8385/13440 mbufs in use (current/cache/total) 1907/5263/7170/131072 mbuf clusters in use (current/cache/total/max) 1907/4493 mbuf+clusters out of packet secondary zone in use (current/cache) 117/1128/1245/33280 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/16640 9k jumbo clusters in use (current/cache/total/max) 0/0/0/8320 16k jumbo clusters in use (current/cache/total/max) 5563K/17134K/22697K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 8664222 requests for I/O initiated by sendfile 0 calls to protocol drain routines 2.) -ni ( a lot of input errors ) Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll bce0 1500 {hidden} 401171627 2001104 0 254047107 0 0 bce0 1500 {hidden} {hidden} 405636437 - - 454597924 - - bce1 1500 {hidden} 17418213 209 0 14402350 0 0 bce1 1500 {hidden} {hidden} 17235746 - - 14211395 - - bce2* 1500 {hidden} 0 0 0 0 0 0 bce3* 1500 {hidden} 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 lo0 16384 7075150 0 0 7075326 0 0 lo0 16384 127.0.0.0/8 127.0.0.1 0 - - 7075400 - - pflog 33152 0 0 0 36748 0 0 sysctl: hw.bce.msi_enable: 1 hw.bce.tso_enable: 1 dev.bce.0.%desc: HP NC382i DP Multifunction Gigabit Server Adapter (C0) dev.bce.0.%driver: bce dev.bce.0.%location: slot=0 function=0 dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x103c subdevice=0x7055 class=0x020000 dev.bce.0.%parent: pci2 dev.bce.0.l2fhdr_error_count: 0 dev.bce.0.mbuf_alloc_failed_count: 0 dev.bce.0.mbuf_frag_count: 0 dev.bce.0.dma_map_addr_rx_failed_count: 0 dev.bce.0.dma_map_addr_tx_failed_count: 52 dev.bce.0.unexpected_attention_count: 0 dev.bce.0.stat_IfHcInOctets: 77735343678 dev.bce.0.stat_IfHCInBadOctets: 1453703 dev.bce.0.stat_IfHCOutOctets: 596056312299 dev.bce.0.stat_IfHCOutBadOctets: 0 dev.bce.0.stat_IfHCInUcastPkts: 404441471 dev.bce.0.stat_IfHCInMulticastPkts: 134 dev.bce.0.stat_IfHCInBroadcastPkts: 3076 dev.bce.0.stat_IfHCOutUcastPkts: 524112124 dev.bce.0.stat_IfHCOutMulticastPkts: 0 dev.bce.0.stat_IfHCOutBroadcastPkts: 56 dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0 dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0 dev.bce.0.stat_Dot3StatsFCSErrors: 0 dev.bce.0.stat_Dot3StatsAlignmentErrors: 0 dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0 dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0 dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0 dev.bce.0.stat_Dot3StatsLateCollisions: 0 dev.bce.0.stat_EtherStatsCollisions: 0 dev.bce.0.stat_EtherStatsFragments: 0 dev.bce.0.stat_EtherStatsJabbers: 0 dev.bce.0.stat_EtherStatsUndersizePkts: 0 dev.bce.0.stat_EtherStatsOversizePkts: 0 dev.bce.0.stat_EtherStatsPktsRx64Octets: 221079621 dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 116359437 dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 463526 dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 1310927 dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 58127062 dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 7104108 dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 0 dev.bce.0.stat_EtherStatsPktsTx64Octets: 38475943 dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 31773632 dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 32482350 dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 25713275 dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 20790569 dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 374876411 dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 0 dev.bce.0.stat_XonPauseFramesReceived: 0 dev.bce.0.stat_XoffPauseFramesReceived: 0 dev.bce.0.stat_OutXonSent: 0 dev.bce.0.stat_OutXoffSent: 0 dev.bce.0.stat_FlowControlDone: 0 dev.bce.0.stat_MacControlFramesReceived: 0 dev.bce.0.stat_XoffStateEntered: 0 dev.bce.0.stat_IfInFramesL2FilterDiscards: 19159 dev.bce.0.stat_IfInRuleCheckerDiscards: 0 dev.bce.0.stat_IfInFTQDiscards: 0 dev.bce.0.stat_IfInMBUFDiscards: 0 dev.bce.0.stat_IfInRuleCheckerP4Hit: 3210 dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0 dev.bce.0.stat_CatchupInFTQDiscards: 0 dev.bce.0.stat_CatchupInMBUFDiscards: 0 dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0 dev.bce.0.com_no_buffers: 2019766 RX/TX pages - 2 (default by driver), until now I didn't try to set a higher value (I saw in -current this option is available for changing via sysctl) load averages: 4.84, 4.81, 4.86 CPU: 5.4% user, 0.0% nice, 10.9% system, 4.4% interrupt, 79.3% idle Mem: 3179M Active, 11G Inact, 4253M Wired, 56K Cache, 2465M Buf, 5097M Free Swap: 24G Total, 24G Free I found in similar thread this patch as suggested: http://people.freebsd.org/~yongari/bce/bce.20100305.diff, but I didn't try it yet, because noone confirmed if this is a working solution. Anyone tested on -stable RX/TX pages = 8 ? Did anyone test suggested patch from Yongari and can confirm it is a working solution ? Thanks From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 20:02:55 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 871D2106564A for ; Tue, 25 Oct 2011 20:02:55 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2876F8FC0C for ; Tue, 25 Oct 2011 20:02:54 +0000 (UTC) Received: by wyi40 with SMTP id 40so1248045wyi.13 for ; Tue, 25 Oct 2011 13:02:54 -0700 (PDT) Received: by 10.227.203.198 with SMTP id fj6mr5601769wbb.24.1319572973617; Tue, 25 Oct 2011 13:02:53 -0700 (PDT) Received: from dfleuriot.local (angel.c-mal.com. [82.241.189.111]) by mx.google.com with ESMTPS id 11sm47014878wby.15.2011.10.25.13.02.51 (version=SSLv3 cipher=OTHER); Tue, 25 Oct 2011 13:02:52 -0700 (PDT) Message-ID: <4EA715EA.9070608@my.gd> Date: Tue, 25 Oct 2011 22:02:50 +0200 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4EA70DD7.9040207@bobi.gateit.net> In-Reply-To: <4EA70DD7.9040207@bobi.gateit.net> Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 7bit Subject: Re: bce huge amount of input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 20:02:55 -0000 On 10/25/11 9:28 PM, Bojidara Marinchovska wrote: > Hello, > > I'm running FreeBSD 8.2-STABLE #1: Thu May 19 15:05:33 EEST 2011 > [snip] > I found in similar thread this patch as suggested: > http://people.freebsd.org/~yongari/bce/bce.20100305.diff, but I didn't > try it yet, because noone confirmed if this is a working solution. > > Anyone tested on -stable RX/TX pages = 8 ? > Did anyone test suggested patch from Yongari and can confirm it is a > working solution ? > What does your port say, on the switch ? If you're using cisco, issue "show interface g0/x" and check for CRC errors and the like. From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 20:11:19 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2C48C106566B for ; Tue, 25 Oct 2011 20:11:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id B52EC8FC19 for ; Tue, 25 Oct 2011 20:11:18 +0000 (UTC) Received: by wyi40 with SMTP id 40so1259916wyi.13 for ; Tue, 25 Oct 2011 13:11:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=2uZgfpNI4vtcMpT8y7YNKBhunH2Ls2Mf/AxY9Ll27Gg=; b=nLg0swEvA5uBjDpftvXJ99GMJAw2r+xOeyqihwheuwHWfD0iPoXAsO3sLC5VtREOzd XRKsZDFwXGmLyFw4Fq5bdNWq0lr1was/ctekmUjR774pFVK8oZfDEJgZs0MShyC9fcep EBhSWVH0xsg/nhnmsrQbaUuYYH5zZLn8VOuWI= Received: by 10.227.198.85 with SMTP id en21mr5549248wbb.11.1319573477156; Tue, 25 Oct 2011 13:11:17 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id h9sm5318464wbp.14.2011.10.25.13.11.13 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 25 Oct 2011 13:11:15 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 25 Oct 2011 13:09:37 -0700 From: YongHyeon PYUN Date: Tue, 25 Oct 2011 13:09:37 -0700 To: Andrey Smagin Message-ID: <20111025200937.GA9348@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 20:11:19 -0000 On Tue, Oct 25, 2011 at 10:44:48AM +0400, Andrey Smagin wrote: > Hi ALL ! If I connect gigabit switch to my card - system hang until I unplug patchcord from device. > With 100Mbit switch card work good. Show me the output of dmesg and 'pciconf -lcbv'. > System: Current - r226163 > From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 20:55:57 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78054106568D for ; Tue, 25 Oct 2011 20:55:57 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-dy0-f54.google.com (mail-dy0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id CC1BB8FC16 for ; Tue, 25 Oct 2011 20:55:56 +0000 (UTC) Received: by dye36 with SMTP id 36so75914dye.13 for ; Tue, 25 Oct 2011 13:55:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bnQO0c8H0QLFtSiXKMVljV9yrCMgTlFKxM6LdZA9AJI=; b=ScxmtW4HiK/gJl8lChSv+1mgnPPceqle8XPkh+uGTvoPUtof0YuOCkjDhEO2xmX+sf hJ8oUQjFUjUklZ702pT6SE7XTQfXrm6tdx+bPbN86z+fgVY508e5gLmt5lM/5QPbmKqQ v0DIUtPsISjW2JNre/cllF6Tqc490vCfxpUKE= MIME-Version: 1.0 Received: by 10.182.59.5 with SMTP id v5mr1860763obq.78.1319576155115; Tue, 25 Oct 2011 13:55:55 -0700 (PDT) Received: by 10.182.35.193 with HTTP; Tue, 25 Oct 2011 13:55:55 -0700 (PDT) In-Reply-To: References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> Date: Tue, 25 Oct 2011 13:55:55 -0700 Message-ID: From: Jason Wolfe To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 20:55:57 -0000 On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe wrote: > Bumping rx/tx descriptors to 2048 was actually for performance reasons and > not to try to get around the issue. I did some fairly in depth testing and > found under heavy load it performed the best with those settings. > > As mentioned on the other thread I'll re enable MSI-X on a few servers here > and collect uptime and the kernel msgbuf in addition. I'll bump the > descriptors down to 512 to try and increase our chances and compile the > driver with EM_MULTIQUEUE also. > > Jason > Hi there, So I have a small pool of server running EM_MULTIQUEUE with lower descriptors as promised and just received an alert of an event. I have a fairly large pool of servers on the same hardware running the same OS/driver sans MSI-X and multiqueue with not a single 'wedge' event in about 2 months now, and it seems multiqueue has not changed the commonality of the issue. Here is my loader.conf followed by everything collected: net.inet.tcp.tcbhashsize="4096" net.inet.tcp.syncache.hashsize="1024" net.inet.tcp.syncache.bucketlimit="512" net.inet.tcp.syncache.cachelimit="65536" net.inet.tcp.hostcache.hashsize="1024" net.inet.tcp.hostcache.bucketlimit="512" net.inet.tcp.hostcache.cachelimit="65536" hw.em.rxd="512" hw.em.txd="512" cc_cubic_load="YES" I bounced em1 because dropped packets incremented 1386169 to 1386355 and the interface is not incrementing packets out. 1:30PM up 4 days, 6:19, 0 users, load averages: 0.18, 0.38, 0.42 interrupt total rate irq3: uart1 5816 0 cpu0: timer 736655476 2000 irq256: em0:rx 0 38122306 103 irq257: em0:tx 0 1605535054 4359 irq258: em0:link 1 0 irq259: em1:rx 0 2192460862 5952 irq260: em1:tx 0 1599049303 4341 irq261: em1:link 4172 0 irq262: mps0 212448927 576 cpu2: timer 736647277 2000 cpu3: timer 736647302 2000 cpu1: timer 736647302 2000 Total 8594223798 23333 27653/6022/33675 mbufs in use (current/cache/total) 3054/3196/6250/5700670 mbuf clusters in use (current/cache/total/max) 3054/1041 mbuf+clusters out of packet secondary zone in use (current/cache) 23266/1642/24908/2850335 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 106085K/14465K/120550K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 22 requests for I/O initiated by sendfile 0 calls to protocol drain routines Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop em0 1500 00:25:90:1f:f5:7d 38575296 0 0 6300959828 0 0 706638 em0 1500 fe80:1::225:9 fe80:1::225:90ff: 0 - - 3 - - - em1 1500 00:25:90:1f:f5:7d 6091053202 22415 0 6327642657 0 0 1386797 em1 1500 fe80:2::225:9 fe80:2::225:90ff: 0 - - 1 - - - lagg0 1500 00:25:90:1f:f5:7d 6129556798 0 0 12627493094 2093435 0 0 lagg0 1500 69.164.38.0/2 69.164.38.93 5429109508 - - 12630422599 - - - lagg0 1500 fe80:5::225:9 fe80:5::225:90ff: 12 - - 17 - - - lagg0 1500 2607:f4e8:310 2607:f4e8:310:12: 13655 - - 13663 - - - kern.msgbuf: Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE Oct 25 13:30:04 cds1043 kernel: em0: hw tdh = 105, hw tdt = 158 Oct 25 13:30:04 cds1043 kernel: em0: hw rdh = 191, hw rdt = 190 Oct 25 13:30:04 cds1043 kernel: em0: Tx Queue Status = 0 Oct 25 13:30:04 cds1043 kernel: em0: TX descriptors avail = 422 Oct 25 13:30:04 cds1043 kernel: em0: Tx Descriptors avail failure = 0 Oct 25 13:30:04 cds1043 kernel: em0: RX discarded packets = 0 Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Check = 192 Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Refresh = 191 Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE Oct 25 13:30:04 cds1043 kernel: em1: hw tdh = 159, hw tdt = 159 Oct 25 13:30:04 cds1043 kernel: em1: hw rdh = 193, hw rdt = 191 Oct 25 13:30:04 cds1043 kernel: em1: Tx Queue Status = 0 Oct 25 13:30:04 cds1043 kernel: em1: TX descriptors avail = 512 Oct 25 13:30:04 cds1043 kernel: em1: Tx Descriptors avail failure = 0 Oct 25 13:30:04 cds1043 kernel: em1: RX discarded packets = 0 Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Check = 407 Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Refresh = 436 net.inet.ip.intr_queue_maxlen: 512 net.inet.ip.intr_queue_drops: 0 dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.0.%parent: pci1 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.eee_control: 0 dev.em.0.link_irq: 1 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1074790984 dev.em.0.rx_control: 67141634 dev.em.0.fc_high_water: 18432 dev.em.0.fc_low_water: 16932 dev.em.0.queue0.txd_head: 208 dev.em.0.queue0.txd_tail: 208 dev.em.0.queue0.tx_irq: 1605545961 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 348 dev.em.0.queue0.rxd_tail: 347 dev.em.0.queue0.rx_irq: 38122461 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 38588018 dev.em.0.mac_stats.good_pkts_recvd: 38588018 dev.em.0.mac_stats.bcast_pkts_recvd: 38572739 dev.em.0.mac_stats.mcast_pkts_recvd: 1868 dev.em.0.mac_stats.rx_frames_64: 38572811 dev.em.0.mac_stats.rx_frames_65_127: 2762 dev.em.0.mac_stats.rx_frames_128_255: 12186 dev.em.0.mac_stats.rx_frames_256_511: 236 dev.em.0.mac_stats.rx_frames_512_1023: 23 dev.em.0.mac_stats.rx_frames_1024_1522: 0 dev.em.0.mac_stats.good_octets_recvd: 2470666426 dev.em.0.mac_stats.good_octets_txd: 8361514233625 dev.em.0.mac_stats.total_pkts_txd: 6301004697 dev.em.0.mac_stats.good_pkts_txd: 6301004695 dev.em.0.mac_stats.bcast_pkts_txd: 80 dev.em.0.mac_stats.mcast_pkts_txd: 2425 dev.em.0.mac_stats.tx_frames_64: 36413025 dev.em.0.mac_stats.tx_frames_65_127: 648631416 dev.em.0.mac_stats.tx_frames_128_255: 7701802 dev.em.0.mac_stats.tx_frames_256_511: 11499983 dev.em.0.mac_stats.tx_frames_512_1023: 56954995 dev.em.0.mac_stats.tx_frames_1024_1522: 5539803478 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 3 dev.em.0.interrupts.rx_pkt_timer: 0 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 0 dev.em.0.interrupts.tx_abs_timer: 0 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 0 dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.1.%driver: em dev.em.1.%location: slot=0 function=0 dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 subdevice=0x10d3 class=0x020000 dev.em.1.%parent: pci2 dev.em.1.nvm: -1 dev.em.1.debug: -1 dev.em.1.rx_int_delay: 0 dev.em.1.tx_int_delay: 66 dev.em.1.rx_abs_int_delay: 66 dev.em.1.tx_abs_int_delay: 66 dev.em.1.rx_processing_limit: 100 dev.em.1.flow_control: 3 dev.em.1.eee_control: 0 dev.em.1.link_irq: 4172 dev.em.1.mbuf_alloc_fail: 0 dev.em.1.cluster_alloc_fail: 0 dev.em.1.dropped: 0 dev.em.1.tx_dma_fail: 0 dev.em.1.rx_overruns: 0 dev.em.1.watchdog_timeouts: 0 dev.em.1.device_control: 1074790984 dev.em.1.rx_control: 67141634 dev.em.1.fc_high_water: 18432 dev.em.1.fc_low_water: 16932 dev.em.1.queue0.txd_head: 159 dev.em.1.queue0.txd_tail: 159 dev.em.1.queue0.tx_irq: 1599049292 dev.em.1.queue0.no_desc_avail: 0 dev.em.1.queue0.rxd_head: 122 dev.em.1.queue0.rxd_tail: 121 dev.em.1.queue0.rx_irq: 2190215040 dev.em.1.mac_stats.excess_coll: 0 dev.em.1.mac_stats.single_coll: 0 dev.em.1.mac_stats.multiple_coll: 0 dev.em.1.mac_stats.late_coll: 0 dev.em.1.mac_stats.collision_count: 0 dev.em.1.mac_stats.symbol_errors: 0 dev.em.1.mac_stats.sequence_errors: 0 dev.em.1.mac_stats.defer_count: 0 dev.em.1.mac_stats.missed_packets: 22415 dev.em.1.mac_stats.recv_no_buff: 11223 dev.em.1.mac_stats.recv_undersize: 0 dev.em.1.mac_stats.recv_fragmented: 0 dev.em.1.mac_stats.recv_oversize: 0 dev.em.1.mac_stats.recv_jabber: 0 dev.em.1.mac_stats.recv_errs: 0 dev.em.1.mac_stats.crc_errs: 0 dev.em.1.mac_stats.alignment_errs: 0 dev.em.1.mac_stats.coll_ext_errs: 0 dev.em.1.mac_stats.xon_recvd: 0 dev.em.1.mac_stats.xon_txd: 0 dev.em.1.mac_stats.xoff_recvd: 0 dev.em.1.mac_stats.xoff_txd: 0 dev.em.1.mac_stats.total_pkts_recvd: 6091088292 dev.em.1.mac_stats.good_pkts_recvd: 6091065877 dev.em.1.mac_stats.bcast_pkts_recvd: 38569835 dev.em.1.mac_stats.mcast_pkts_recvd: 1860 dev.em.1.mac_stats.rx_frames_64: 2329378133 dev.em.1.mac_stats.rx_frames_65_127: 2592499514 dev.em.1.mac_stats.rx_frames_128_255: 7898056 dev.em.1.mac_stats.rx_frames_256_511: 15815777 dev.em.1.mac_stats.rx_frames_512_1023: 44494959 dev.em.1.mac_stats.rx_frames_1024_1522: 1100979438 dev.em.1.mac_stats.good_octets_recvd: 2043621877185 dev.em.1.mac_stats.good_octets_txd: 8381779145365 dev.em.1.mac_stats.total_pkts_txd: 6327642657 dev.em.1.mac_stats.good_pkts_txd: 6327642657 dev.em.1.mac_stats.bcast_pkts_txd: 2149 dev.em.1.mac_stats.mcast_pkts_txd: 11 dev.em.1.mac_stats.tx_frames_64: 36904932 dev.em.1.mac_stats.tx_frames_65_127: 662019693 dev.em.1.mac_stats.tx_frames_128_255: 7256854 dev.em.1.mac_stats.tx_frames_256_511: 11840333 dev.em.1.mac_stats.tx_frames_512_1023: 57343575 dev.em.1.mac_stats.tx_frames_1024_1522: 5552277270 dev.em.1.mac_stats.tso_txd: 0 dev.em.1.mac_stats.tso_ctx_fail: 0 dev.em.1.interrupts.asserts: 3934 dev.em.1.interrupts.rx_pkt_timer: 3 dev.em.1.interrupts.rx_abs_timer: 0 dev.em.1.interrupts.tx_pkt_timer: 0 dev.em.1.interrupts.tx_abs_timer: 0 dev.em.1.interrupts.tx_queue_empty: 0 dev.em.1.interrupts.tx_queue_min_thresh: 0 dev.em.1.interrupts.rx_desc_min_thresh: 0 dev.em.1.interrupts.rx_overrun: 1 Jason From owner-freebsd-net@FreeBSD.ORG Tue Oct 25 21:22:04 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0E3F106566B; Tue, 25 Oct 2011 21:22:04 +0000 (UTC) (envelope-from tomelite82@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 7BA2D8FC08; Tue, 25 Oct 2011 21:22:04 +0000 (UTC) Received: by ggnq2 with SMTP id q2so1197200ggn.13 for ; Tue, 25 Oct 2011 14:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Kbi/fzghKC9XyG9iurZ6SfP/8uJup/V08JGXuVcWENY=; b=iaiGMvNaYO6/lyaC++nckmOB/4p/L/V4+LGqGvDXx5OBfRWt9prVUzZyrKSK94AUqo Bb7r8yn6GD5q1DxhAhZyNR4WtWbhFHM64icEnxI4h+QQfF4RbZVuhKDK7qrTxDyTIdnN 1rB7xImxnGGAeQjaN3v22t4WgtSaKKyZXIdjk= MIME-Version: 1.0 Received: by 10.151.87.19 with SMTP id p19mr5795603ybl.71.1319577723792; Tue, 25 Oct 2011 14:22:03 -0700 (PDT) Sender: tomelite82@gmail.com Received: by 10.151.44.14 with HTTP; Tue, 25 Oct 2011 14:22:03 -0700 (PDT) In-Reply-To: References: Date: Tue, 25 Oct 2011 14:22:03 -0700 X-Google-Sender-Auth: SQawIKIy35MXcWVzrELv13mgS28 Message-ID: From: Qing Li To: freebsd-net@freebsd.org, FreeBSD Stable List Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Qing Li Subject: FreeBSD 9 IPv6 conformance test report X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Oct 2011 21:22:05 -0000 Since I have been receiving private reports on the test failures, I thought I just post the test report as executed against the upcoming FreeBSD 9.0 release in my own setup, so you can see where the baseline is. More test reports will be posted here. =A0 =A0 =A0 =A0http://people.freebsd.org/~qingli/ct-freebsd-9/ I am going over each failure and providing the fix. Please let me know if you already have a suggested patch that you are willing to share. Thanks. -- Qing From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 04:35:24 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DBB81065674 for ; Wed, 26 Oct 2011 04:35:24 +0000 (UTC) (envelope-from yilinjing2006@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 4DE3C8FC0C for ; Wed, 26 Oct 2011 04:35:24 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1RIvD1-0004t4-KP for freebsd-net@freebsd.org; Tue, 25 Oct 2011 21:35:23 -0700 Date: Tue, 25 Oct 2011 21:35:23 -0700 (PDT) From: jyl_2006 To: freebsd-net@freebsd.org Message-ID: <1319603723611-4938682.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Does "Transparent TCP-to-SCTP Translation Shim Layer" available? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 04:35:24 -0000 Hi, Does anyone know Shim Layer about TCP-to-SCTP is available or not? Or any schedule about this? -- View this message in context: http://freebsd.1045724.n5.nabble.com/Does-Transparent-TCP-to-SCTP-Translation-Shim-Layer-available-tp4938682p4938682.html Sent from the freebsd-net mailing list archive at Nabble.com. From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 07:23:56 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78D421065674 for ; Wed, 26 Oct 2011 07:23:56 +0000 (UTC) (envelope-from quintessence@bobi.gateit.net) Received: from mail.gateit.net (mail4o.gateit.net [85.187.21.218]) by mx1.freebsd.org (Postfix) with ESMTP id 3736E8FC08 for ; Wed, 26 Oct 2011 07:23:55 +0000 (UTC) Received: from [127.0.0.1] (priv.gateit.net [212.43.51.203]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.gateit.net (Postfix) with ESMTPSA id 8307F90800F for ; Wed, 26 Oct 2011 10:23:54 +0300 (EEST) Message-ID: <4EA7B57F.5010604@bobi.gateit.net> Date: Wed, 26 Oct 2011 10:23:43 +0300 From: Bojidara Marinchovska User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4EA70DD7.9040207@bobi.gateit.net> <4EA715EA.9070608@my.gd> In-Reply-To: <4EA715EA.9070608@my.gd> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: bce huge amount of input errors X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 07:23:56 -0000 Hello, it says nothing to be worry about 10 input errors, 6 CRC, 0 frame, 0 overrun, 4 ignored 0 watchdog, 64973071 multicast, 0 pause input 0 input packets with dribble condition detected 4 output errors, 0 collisions, 2 interface resets 0 babbles, 0 late collision, 0 deferred 4 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out I see same behaviour with bce input error on other servers running different FreeBSD amd64 versions (small bandwidth, small amount of errors) on the same switch, but I also see no input errors on other FreeBSD amd64 servers which are on the same switch(small bandwitdh, no errors). All servers don't hit any limits. Only for information without providing any other details versions are as follows: 1.)8.2-STABLE FreeBSD 8.2-STABLE #0: Fri Sep 30 19:59:31 EEST 2011 - no input errors - very small bandwidth usage 2.)8.2-STABLE FreeBSD 8.2-STABLE #0: Tue Jul 19 15:38:26 EEST 2011 - 6249 input errors since 40 days - small bandwidth usage 3.)8.2-STABLE FreeBSD 8.2-STABLE #0: Mon May 16 14:57:35 EEST 2011 - 2 interfaces: 177 errors since boot, 331444 errors since boot - same bandwidth usage as server 2 4.)8.2-STABLE FreeBSD 8.2-STABLE #0: Mon May 16 17:16:30 EEST 2011 - no input errors - bandwidth usage as server 1 5.)8.1-STABLE FreeBSD 8.1-STABLE #0: Tue Oct 12 10:12:02 EEST 2010 - 2711 ierrors in 6 months - ~50-60Mb/s bandwidth 6.)7.1-STABLE FreeBSD 7.1-STABLE #0: Thu Jan 29 10:21:42 EET 2009 - no errors - same bandwidth usage as server 1 7.)8.2-STABLE FreeBSD 8.2-STABLE #1: Wed Apr 20 10:43:34 EEST 2011 - 3505 ierrors since boot - same bandwidth as server 5 (50-60Mb/s) 8.)8.0-STABLE FreeBSD 8.0-STABLE #0: Wed Dec 30 12:30:03 EET 2009 - 8015 ierrors since boot - same bandwidth usage as server 2 9.)8.2-STABLE FreeBSD 8.2-STABLE #1: Thu May 19 15:05:33 EEST 2011 - 2748140 ierrors in 20 hours uptime - ~250Mb/s bandwidth 10.)8.1-STABLE FreeBSD 8.1-STABLE #0: Thu Aug 19 13:49:35 EEST 2010 - no ierrors - bandwidth usage as server 1 11.)7.2-STABLE FreeBSD 7.2-STABLE #1: Fri Dec 11 15:26:14 EET 2009 - 2 interfaces - 1835 ierrors since boot, 5392022 errors since boot - ~80Mb/s bandwidth 12.)8.0-STABLE FreeBSD 8.0-STABLE #0: Tue May 11 13:58:54 EEST 2010 - 114 ierrors since boot - bandwidth usage as server 1 13.) 7.4-STABLE FreeBSD 7.4-STABLE #3: Thu Jun 2 10:56:15 EEST 2011 - no ierrors - bandwidth usage as server 1 14.)8.0-STABLE FreeBSD 8.0-STABLE #1: Wed Jan 6 17:57:14 EET 2010 - no ierrors - bandwidth usage as server 1 Also on the same switch are servers with FreeBSD amd64 with BCM5702X (bge driver) - small bandwidth usage ~65000 input errors On 25.10.2011 23:02 ÷., Damien Fleuriot wrote: > > What does your port say, on the switch ? > > If you're using cisco, issue "show interface g0/x" and check for CRC > errors and the like. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 07:55:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EABC6106566B; Wed, 26 Oct 2011 07:55:16 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 4AC0C8FC14; Wed, 26 Oct 2011 07:55:16 +0000 (UTC) Received: from localhost (58.wheelsystems.com [83.12.187.58]) by mail.dawidek.net (Postfix) with ESMTPSA id 177781E2; Wed, 26 Oct 2011 09:55:14 +0200 (CEST) Date: Wed, 26 Oct 2011 09:54:31 +0200 From: Pawel Jakub Dawidek To: John Baldwin Message-ID: <20111026075431.GB1672@garage.freebsd.pl> References: <20111022084931.GD1697@garage.freebsd.pl> <20111023084445.GB50300@deviant.kiev.zoral.com.ua> <20111023155827.GH1697@garage.freebsd.pl> <201110240814.22368.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0eh6TmSyL6TZE2Uz" Content-Disposition: inline In-Reply-To: <201110240814.22368.jhb@freebsd.org> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Kostik Belousov , Lawrence Stewart , freebsd-current@freebsd.org, Andre Oppermann , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 07:55:17 -0000 --0eh6TmSyL6TZE2Uz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 24, 2011 at 08:14:22AM -0400, John Baldwin wrote: > On Sunday, October 23, 2011 11:58:28 am Pawel Jakub Dawidek wrote: > > On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: > > > On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: > > > > My suggestion would be that if we won't be able to fix it before 9.= 0, > > > > we should turn this assertion off, as the system seems to be able to > > > > recover. > > >=20 > > > Shipped kernels have all assertions turned off. > >=20 > > Yes, I'm aware of that, but many people compile their production kernels > > with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. > > corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing > > it into a printf, so it will be visible. >=20 > No, the kernel is corrupting things in other places when this is true, so > if you are running with INVARIANTS, we want to know about it. Specifica= lly, > in several places in TCP we assume that rcv_adv >=3D rcv_nxt, and depend = on > being able to do 'rcv_adv - rcv_nxt'. >=20 > In this case, it looks like the difference is consistently less than one= =20 > frame. I suspect the other end of the connection is sending just beyond = the=20 > end of the advertised window (it probably assumes it is better to send a = full=20 > frame if it has that much pending data even though part of it is beyond t= he=20 > window edge vs sending a truncated packet that just fills the window) and= that > that frame is accepted ok in the header prediction case and it's ACK is= =20 > delayed, but the next packet to arrive then trips over this assumption. >=20 > Since 'win' is guaranteed to be non-negative and we explicitly cast > 'rcv_adv - rcv_nxt' to (int) in the following line that the assert is che= cking > for: >=20 > tp->rcv_wnd =3D imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); >=20 > I think we already handle this case ok and perhaps the assertion can just= be > removed? Not sure if others feel that it warrants a comment to note that= this > is the case being handled. I added debug to the places where rcv_adv and rcv_nxt are modified. Here is what happens before the panic occurs: tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223615= 48 rcv_adv 4022360100 diff -1448 tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223622= 98 rcv_adv 4022361548 diff -750 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223637= 46 rcv_adv 4022362298 diff -1448 tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223648= 36 rcv_adv 4022363746 diff -1090 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223662= 84 rcv_adv 4022364836 diff -1448 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223706= 28 rcv_adv 4022369690 diff -938 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223791= 40 rcv_adv 4022377692 diff -1448 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223877= 92 rcv_adv 4022386344 diff -1448 tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223888= 90 rcv_adv 4022387792 diff -1098 tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223903= 38 rcv_adv 4022388890 diff -1448 tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 40223945= 63 rcv_adv 4022394342 diff -221 panic: tcp_input negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 = rcv_adv 4022394342 win=3D0 diff -221 I can send you the full log if you want, I've plenty of messages where rcv_adv < rcv_nxt, not all of them trigger this assertion. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --0eh6TmSyL6TZE2Uz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk6nvLYACgkQForvXbEpPzS/igCgh6TcluOieBa/tfP90xC4Gy1t 5GMAoNQG1Mc8LyevKm/3rQFJoC02DriH =LzuI -----END PGP SIGNATURE----- --0eh6TmSyL6TZE2Uz-- From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 10:45:28 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 25F84106566C for ; Wed, 26 Oct 2011 10:45:28 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 991FC8FC17 for ; Wed, 26 Oct 2011 10:45:27 +0000 (UTC) Received: by wwi18 with SMTP id 18so2129573wwi.31 for ; Wed, 26 Oct 2011 03:45:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=pfs6OZpFKxWLJ9U6gUytcmyMRyOyoXxea8gJTeuamW4=; b=u1G0VLxtXkdFuMXC5J+0wjnIY3tT62Fv6Cl7YO8Ge6l6t4ldZG14AFc3kC061TN+XN q+rK7+1B6sWgXv5q/3aWxZVr0IB6VmJHCJJ6QptfImSgiBlqP3e8gd/irhZeT2PbMbb6 1OWkkENr7DywuIgjB2YA/5fGCVpFJLs3LdyDQ= Received: by 10.227.207.130 with SMTP id fy2mr9254875wbb.8.1319625926097; Wed, 26 Oct 2011 03:45:26 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id es5sm2361672wbb.11.2011.10.26.03.45.21 (version=SSLv3 cipher=OTHER); Wed, 26 Oct 2011 03:45:25 -0700 (PDT) Message-ID: <4EA7E4B8.9050003@gmail.com> Date: Wed, 26 Oct 2011 14:15:12 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Jason Wolfe References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 10:45:28 -0000 Hi Jason Have you tried: hw.em.fc_setting="0" (in loader.conf) ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol with MSIX and no multiqueue. Advanced features has always been a source of problem. It is worth a try and help to narrow down possibilities. It would also be helpful if you provide 'ifconfig' output when the problem happens. And a question: Does interface RX also hangs or it is just TX? On 10/26/2011 12:25 AM, Jason Wolfe wrote: > On Fri, Oct 7, 2011 at 2:14 PM, Jason Wolfe wrote: > >> Bumping rx/tx descriptors to 2048 was actually for performance reasons and >> not to try to get around the issue. I did some fairly in depth testing and >> found under heavy load it performed the best with those settings. >> >> As mentioned on the other thread I'll re enable MSI-X on a few servers here >> and collect uptime and the kernel msgbuf in addition. I'll bump the >> descriptors down to 512 to try and increase our chances and compile the >> driver with EM_MULTIQUEUE also. >> >> Jason >> > Hi there, > > So I have a small pool of server running EM_MULTIQUEUE with lower > descriptors as promised and just received an alert of an event. I have a > fairly large pool of servers on the same hardware running the same OS/driver > sans MSI-X and multiqueue with not a single 'wedge' event in about 2 months > now, and it seems multiqueue has not changed the commonality of the issue. > Here is my loader.conf followed by everything collected: > > net.inet.tcp.tcbhashsize="4096" > net.inet.tcp.syncache.hashsize="1024" > net.inet.tcp.syncache.bucketlimit="512" > net.inet.tcp.syncache.cachelimit="65536" > net.inet.tcp.hostcache.hashsize="1024" > net.inet.tcp.hostcache.bucketlimit="512" > net.inet.tcp.hostcache.cachelimit="65536" > hw.em.rxd="512" > hw.em.txd="512" > cc_cubic_load="YES" > > I bounced em1 because dropped packets incremented 1386169 to 1386355 and the > interface is not incrementing packets out. > > 1:30PM up 4 days, 6:19, 0 users, load averages: 0.18, 0.38, 0.42 > > interrupt total rate > irq3: uart1 5816 0 > cpu0: timer 736655476 2000 > irq256: em0:rx 0 38122306 103 > irq257: em0:tx 0 1605535054 4359 > irq258: em0:link 1 0 > irq259: em1:rx 0 2192460862 5952 > irq260: em1:tx 0 1599049303 4341 > irq261: em1:link 4172 0 > irq262: mps0 212448927 576 > cpu2: timer 736647277 2000 > cpu3: timer 736647302 2000 > cpu1: timer 736647302 2000 > Total 8594223798 23333 > > 27653/6022/33675 mbufs in use (current/cache/total) > 3054/3196/6250/5700670 mbuf clusters in use (current/cache/total/max) > 3054/1041 mbuf+clusters out of packet secondary zone in use (current/cache) > 23266/1642/24908/2850335 4k (page size) jumbo clusters in use > (current/cache/total/max) > 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) > 106085K/14465K/120550K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 22 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > em0 1500 00:25:90:1f:f5:7d 38575296 0 0 6300959828 0 0 706638 > em0 1500 fe80:1::225:9 fe80:1::225:90ff: 0 - - 3 - - - > em1 1500 00:25:90:1f:f5:7d 6091053202 22415 0 6327642657 0 0 > 1386797 > em1 1500 fe80:2::225:9 fe80:2::225:90ff: 0 - - 1 - - - > lagg0 1500 00:25:90:1f:f5:7d 6129556798 0 0 12627493094 2093435 0 0 > > lagg0 1500 69.164.38.0/2 69.164.38.93 5429109508 - - 12630422599 - - - > lagg0 1500 fe80:5::225:9 fe80:5::225:90ff: 12 - - 17 - - - > lagg0 1500 2607:f4e8:310 2607:f4e8:310:12: 13655 - - 13663 - - - > > kern.msgbuf: > > Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE > Oct 25 13:30:04 cds1043 kernel: em0: hw tdh = 105, hw tdt = 158 > Oct 25 13:30:04 cds1043 kernel: em0: hw rdh = 191, hw rdt = 190 > Oct 25 13:30:04 cds1043 kernel: em0: Tx Queue Status = 0 > Oct 25 13:30:04 cds1043 kernel: em0: TX descriptors avail = 422 > Oct 25 13:30:04 cds1043 kernel: em0: Tx Descriptors avail failure = 0 > Oct 25 13:30:04 cds1043 kernel: em0: RX discarded packets = 0 > Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Check = 192 > Oct 25 13:30:04 cds1043 kernel: em0: RX Next to Refresh = 191 > Oct 25 13:30:04 cds1043 kernel: Interface is RUNNING and INACTIVE > Oct 25 13:30:04 cds1043 kernel: em1: hw tdh = 159, hw tdt = 159 > Oct 25 13:30:04 cds1043 kernel: em1: hw rdh = 193, hw rdt = 191 > Oct 25 13:30:04 cds1043 kernel: em1: Tx Queue Status = 0 > Oct 25 13:30:04 cds1043 kernel: em1: TX descriptors avail = 512 > Oct 25 13:30:04 cds1043 kernel: em1: Tx Descriptors avail failure = 0 > Oct 25 13:30:04 cds1043 kernel: em1: RX discarded packets = 0 > Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Check = 407 > Oct 25 13:30:04 cds1043 kernel: em1: RX Next to Refresh = 436 > > net.inet.ip.intr_queue_maxlen: 512 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.0.%parent: pci1 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 0 > dev.em.0.tx_int_delay: 66 > dev.em.0.rx_abs_int_delay: 66 > dev.em.0.tx_abs_int_delay: 66 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 1 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1074790984 > dev.em.0.rx_control: 67141634 > dev.em.0.fc_high_water: 18432 > dev.em.0.fc_low_water: 16932 > dev.em.0.queue0.txd_head: 208 > dev.em.0.queue0.txd_tail: 208 > dev.em.0.queue0.tx_irq: 1605545961 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 348 > dev.em.0.queue0.rxd_tail: 347 > dev.em.0.queue0.rx_irq: 38122461 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 0 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 0 > dev.em.0.mac_stats.xon_txd: 0 > dev.em.0.mac_stats.xoff_recvd: 0 > dev.em.0.mac_stats.xoff_txd: 0 > dev.em.0.mac_stats.total_pkts_recvd: 38588018 > dev.em.0.mac_stats.good_pkts_recvd: 38588018 > dev.em.0.mac_stats.bcast_pkts_recvd: 38572739 > dev.em.0.mac_stats.mcast_pkts_recvd: 1868 > dev.em.0.mac_stats.rx_frames_64: 38572811 > dev.em.0.mac_stats.rx_frames_65_127: 2762 > dev.em.0.mac_stats.rx_frames_128_255: 12186 > dev.em.0.mac_stats.rx_frames_256_511: 236 > dev.em.0.mac_stats.rx_frames_512_1023: 23 > dev.em.0.mac_stats.rx_frames_1024_1522: 0 > dev.em.0.mac_stats.good_octets_recvd: 2470666426 > dev.em.0.mac_stats.good_octets_txd: 8361514233625 > dev.em.0.mac_stats.total_pkts_txd: 6301004697 > dev.em.0.mac_stats.good_pkts_txd: 6301004695 > dev.em.0.mac_stats.bcast_pkts_txd: 80 > dev.em.0.mac_stats.mcast_pkts_txd: 2425 > dev.em.0.mac_stats.tx_frames_64: 36413025 > dev.em.0.mac_stats.tx_frames_65_127: 648631416 > dev.em.0.mac_stats.tx_frames_128_255: 7701802 > dev.em.0.mac_stats.tx_frames_256_511: 11499983 > dev.em.0.mac_stats.tx_frames_512_1023: 56954995 > dev.em.0.mac_stats.tx_frames_1024_1522: 5539803478 > dev.em.0.mac_stats.tso_txd: 0 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 3 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > dev.em.1.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.1.%driver: em > dev.em.1.%location: slot=0 function=0 > dev.em.1.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x15d9 > subdevice=0x10d3 class=0x020000 > dev.em.1.%parent: pci2 > dev.em.1.nvm: -1 > dev.em.1.debug: -1 > dev.em.1.rx_int_delay: 0 > dev.em.1.tx_int_delay: 66 > dev.em.1.rx_abs_int_delay: 66 > dev.em.1.tx_abs_int_delay: 66 > dev.em.1.rx_processing_limit: 100 > dev.em.1.flow_control: 3 > dev.em.1.eee_control: 0 > dev.em.1.link_irq: 4172 > dev.em.1.mbuf_alloc_fail: 0 > dev.em.1.cluster_alloc_fail: 0 > dev.em.1.dropped: 0 > dev.em.1.tx_dma_fail: 0 > dev.em.1.rx_overruns: 0 > dev.em.1.watchdog_timeouts: 0 > dev.em.1.device_control: 1074790984 > dev.em.1.rx_control: 67141634 > dev.em.1.fc_high_water: 18432 > dev.em.1.fc_low_water: 16932 > dev.em.1.queue0.txd_head: 159 > dev.em.1.queue0.txd_tail: 159 > dev.em.1.queue0.tx_irq: 1599049292 > dev.em.1.queue0.no_desc_avail: 0 > dev.em.1.queue0.rxd_head: 122 > dev.em.1.queue0.rxd_tail: 121 > dev.em.1.queue0.rx_irq: 2190215040 > dev.em.1.mac_stats.excess_coll: 0 > dev.em.1.mac_stats.single_coll: 0 > dev.em.1.mac_stats.multiple_coll: 0 > dev.em.1.mac_stats.late_coll: 0 > dev.em.1.mac_stats.collision_count: 0 > dev.em.1.mac_stats.symbol_errors: 0 > dev.em.1.mac_stats.sequence_errors: 0 > dev.em.1.mac_stats.defer_count: 0 > dev.em.1.mac_stats.missed_packets: 22415 > dev.em.1.mac_stats.recv_no_buff: 11223 > dev.em.1.mac_stats.recv_undersize: 0 > dev.em.1.mac_stats.recv_fragmented: 0 > dev.em.1.mac_stats.recv_oversize: 0 > dev.em.1.mac_stats.recv_jabber: 0 > dev.em.1.mac_stats.recv_errs: 0 > dev.em.1.mac_stats.crc_errs: 0 > dev.em.1.mac_stats.alignment_errs: 0 > dev.em.1.mac_stats.coll_ext_errs: 0 > dev.em.1.mac_stats.xon_recvd: 0 > dev.em.1.mac_stats.xon_txd: 0 > dev.em.1.mac_stats.xoff_recvd: 0 > dev.em.1.mac_stats.xoff_txd: 0 > dev.em.1.mac_stats.total_pkts_recvd: 6091088292 > dev.em.1.mac_stats.good_pkts_recvd: 6091065877 > dev.em.1.mac_stats.bcast_pkts_recvd: 38569835 > dev.em.1.mac_stats.mcast_pkts_recvd: 1860 > dev.em.1.mac_stats.rx_frames_64: 2329378133 > dev.em.1.mac_stats.rx_frames_65_127: 2592499514 > dev.em.1.mac_stats.rx_frames_128_255: 7898056 > dev.em.1.mac_stats.rx_frames_256_511: 15815777 > dev.em.1.mac_stats.rx_frames_512_1023: 44494959 > dev.em.1.mac_stats.rx_frames_1024_1522: 1100979438 > dev.em.1.mac_stats.good_octets_recvd: 2043621877185 > dev.em.1.mac_stats.good_octets_txd: 8381779145365 > dev.em.1.mac_stats.total_pkts_txd: 6327642657 > dev.em.1.mac_stats.good_pkts_txd: 6327642657 > dev.em.1.mac_stats.bcast_pkts_txd: 2149 > dev.em.1.mac_stats.mcast_pkts_txd: 11 > dev.em.1.mac_stats.tx_frames_64: 36904932 > dev.em.1.mac_stats.tx_frames_65_127: 662019693 > dev.em.1.mac_stats.tx_frames_128_255: 7256854 > dev.em.1.mac_stats.tx_frames_256_511: 11840333 > dev.em.1.mac_stats.tx_frames_512_1023: 57343575 > dev.em.1.mac_stats.tx_frames_1024_1522: 5552277270 > dev.em.1.mac_stats.tso_txd: 0 > dev.em.1.mac_stats.tso_ctx_fail: 0 > dev.em.1.interrupts.asserts: 3934 > dev.em.1.interrupts.rx_pkt_timer: 3 > dev.em.1.interrupts.rx_abs_timer: 0 > dev.em.1.interrupts.tx_pkt_timer: 0 > dev.em.1.interrupts.tx_abs_timer: 0 > dev.em.1.interrupts.tx_queue_empty: 0 > dev.em.1.interrupts.tx_queue_min_thresh: 0 > dev.em.1.interrupts.rx_desc_min_thresh: 0 > dev.em.1.interrupts.rx_overrun: 1 > > Jason > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 09:49:17 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB21E1065673 for ; Wed, 26 Oct 2011 09:49:17 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: from f223.mail.ru (f223.mail.ru [217.69.128.156]) by mx1.freebsd.org (Postfix) with ESMTP id 45B668FC15 for ; Wed, 26 Oct 2011 09:49:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Message-Id:Content-Transfer-Encoding:Content-Type:Reply-To:In-Reply-To:References:Date:Mime-Version:Subject:Cc:To:From; bh=eCyhebKt80Dy/D/aDKiN1yB7SaNQNREBB7RP2RBzeuw=; b=cbNVvk+e33QlanYMcMv/PrWldk0b2hmkZ3pEjU0GtQ5a/tUQUqjMaJDkhnzZ+iq2Dq9yir+sa/fnmzpdHhvtkVFHuKGl8lJyUYDi2mQEWym5Y71cxModGw70CQvmAaZZ; Received: from mail by f223.mail.ru with local id 1RJ06k-0003eP-00; Wed, 26 Oct 2011 13:49:14 +0400 Received: from [77.45.196.15] by e.mail.ru with HTTP; Wed, 26 Oct 2011 13:49:14 +0400 From: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= To: pyunyh@gmail.com Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [77.45.196.15] Date: Wed, 26 Oct 2011 13:49:14 +0400 References: <20111025200937.GA9348@michelle.cdnetworks.com> In-Reply-To: <20111025200937.GA9348@michelle.cdnetworks.com> X-Priority: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Message-Id: X-Spam: Not detected X-Mras: Ok X-Mailman-Approved-At: Wed, 26 Oct 2011 10:45:34 +0000 Cc: freebsd-net@freebsd.org Subject: Re[2]: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 09:49:17 -0000 SGkgIQp2Z2UwQHBjaTA6MjowOjA6ICAgICAgICBjbGFzcz0weDAyMDAwMCBjYXJkPTB4MDExMDEx MDYgY2hpcD0weDMxMTkxMTA2IHJldj0weDgyIGhkcj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ1ZJ QSBUZWNobm9sb2dpZXMsIEluYy4nCiAgICBkZXZpY2UgICAgID0gJ1ZUNjEyMC9WVDYxMjEvVlQ2 MTIyIEdpZ2FiaXQgRXRoZXJuZXQgQWRhcHRlcicKICAgIGNsYXNzICAgICAgPSBuZXR3b3JrCiAg ICBzdWJjbGFzcyAgID0gZXRoZXJuZXQKICAgIGJhciAgIFsxMF0gPSB0eXBlIEkvTyBQb3J0LCBy YW5nZSAzMiwgYmFzZSAweDgwMDAsIHNpemUgMjU2LCBlbmFibGVkCiAgICBiYXIgICBbMTRdID0g dHlwZSBNZW1vcnksIHJhbmdlIDY0LCBiYXNlIDB4ZDQxMDAwMDAsIHNpemUgMjU2LCBlbmFibGVk CiAgICBjYXAgMDFbNTBdID0gcG93ZXJzcGVjIDMgIHN1cHBvcnRzIEQwIEQxIEQyIEQzICBjdXJy ZW50IEQwCiAgICBjYXAgMTBbOTBdID0gUENJLUV4cHJlc3MgMSBlbmRwb2ludCBtYXggZGF0YSAx MjgoMTI4KSBsaW5rIHgxKHgxKQogICAgY2FwIDA1W2MwXSA9IE1TSSBzdXBwb3J0cyAxIG1lc3Nh Z2UsIDY0IGJpdCwgdmVjdG9yIG1hc2tzIGVuYWJsZWQgd2l0aCAxIG1lc3NhZ2UKZWNhcCAwMDAx WzEwMF0gPSBBRVIgMSAwIGZhdGFsIDEgbm9uLWZhdGFsIDYgY29ycmVjdGVkCmVjYXAgMDAwM1sx MzBdID0gU2VyaWFsIDEgMDAwMDExMDYwMDAwMDAwMAp2Z2UxQHBjaTA6MTowOjA6ICAgICAgICBj bGFzcz0weDAyMDAwMCBjYXJkPTB4MDExMDExMDYgY2hpcD0weDMxMTkxMTA2IHJldj0weDgyIGhk cj0weDAwCiAgICB2ZW5kb3IgICAgID0gJ1ZJQSBUZWNobm9sb2dpZXMsIEluYy4nCiAgICBkZXZp Y2UgICAgID0gJ1ZUNjEyMC9WVDYxMjEvVlQ2MTIyIEdpZ2FiaXQgRXRoZXJuZXQgQWRhcHRlcicK ICAgIGNsYXNzICAgICAgPSBuZXR3b3JrCiAgICBzdWJjbGFzcyAgID0gZXRoZXJuZXQKICAgIGJh ciAgIFsxMF0gPSB0eXBlIEkvTyBQb3J0LCByYW5nZSAzMiwgYmFzZSAweDcwMDAsIHNpemUgMjU2 LCBlbmFibGVkCiAgICBiYXIgICBbMTRdID0gdHlwZSBNZW1vcnksIHJhbmdlIDY0LCBiYXNlIDB4 ZDQwMDAwMDAsIHNpemUgMjU2LCBlbmFibGVkCiAgICBjYXAgMDFbNTBdID0gcG93ZXJzcGVjIDMg IHN1cHBvcnRzIEQwIEQxIEQyIEQzICBjdXJyZW50IEQwCiAgICBjYXAgMTBbOTBdID0gUENJLUV4 cHJlc3MgMSBlbmRwb2ludCBtYXggZGF0YSAxMjgoMTI4KSBsaW5rIHgxKHgxKQogICAgY2FwIDA1 W2MwXSA9IE1TSSBzdXBwb3J0cyAxIG1lc3NhZ2UsIDY0IGJpdCwgdmVjdG9yIG1hc2tzIGVuYWJs ZWQgd2l0aCAxIG1lc3NhZ2UKZWNhcCAwMDAxWzEwMF0gPSBBRVIgMSAxIGZhdGFsIDEgbm9uLWZh dGFsIDYgY29ycmVjdGVkCmVjYXAgMDAwM1sxMzBdID0gU2VyaWFsIDEgMDAwMDExMDYwMDAwMDAw MAoKZG1lc2cgaXMgZW1wdHkKCjI2INC+0LrRgtGP0LHRgNGPIDIwMTEsIDAwOjExINC+0YIgWW9u Z0h5ZW9uIFBZVU4gPHB5dW55aEBnbWFpbC5jb20+Ogo+IE9uIFR1ZSwgT2N0IDI1LCAyMDExIGF0 IDEwOjQ0OjQ4QU0gKzA0MDAsIEFuZHJleSBTbWFnaW4gd3JvdGU6Cj4gPiBIaSBBTEwgISAgSWYg SSBjb25uZWN0IGdpZ2FiaXQgc3dpdGNoIHRvIG15IGNhcmQgLSBzeXN0ZW0gaGFuZyB1bnRpbCBJ IHVucGx1ZyBwYXRjaGNvcmQgZnJvbSBkZXZpY2UuCj4gPiBXaXRoIDEwME1iaXQgc3dpdGNoIGNh cmQgd29yayBnb29kLgo+IAo+IFNob3cgbWUgdGhlIG91dHB1dCBvZiBkbWVzZyBhbmQgJ3BjaWNv bmYgLWxjYnYnLgo+IAo+ID4gU3lzdGVtOiAgQ3VycmVudCAtIHIyMjYxNjMKPiA+Cj4g From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 11:33:46 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07728106566B for ; Wed, 26 Oct 2011 11:33:46 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id B08A08FC0A for ; Wed, 26 Oct 2011 11:33:45 +0000 (UTC) Received: by vcbfo13 with SMTP id fo13so2015637vcb.13 for ; Wed, 26 Oct 2011 04:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/KqvgwTldAMQ1Q1IwSpkfL4Tms3+uGzuQJoO1LmduFk=; b=xd3hJiAu2e9CWQLYvnXdDYlAAHqwzuKNIcs4i6bnrWETvCtvggjgbc2mMA5jFG66W7 7mmqXtANkjO8zqXQbz1pnlG9q8wHtNRRS18LGpwkVLtJz4vAUJAADMMb5N3Zr/STkWDT ssAvwRUFIkAEkPZQh3Ef4bTBGGB6EuIR1i9xY= MIME-Version: 1.0 Received: by 10.182.59.5 with SMTP id v5mr2396508obq.78.1319628824788; Wed, 26 Oct 2011 04:33:44 -0700 (PDT) Received: by 10.182.35.193 with HTTP; Wed, 26 Oct 2011 04:33:44 -0700 (PDT) In-Reply-To: <4EA7E203.3020306@sepehrs.com> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> Date: Wed, 26 Oct 2011 04:33:44 -0700 Message-ID: From: Jason Wolfe To: Hooman Fazaeli Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 11:33:46 -0000 Hooman, I have run with dev.em.X.flow_control=0, which should have the same result as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the remaining options would be able to produce the scenario I'm seeing, but I'm open to giving it a try with no options on the interfaces. I've also added ifconfig output to the collection. options=219b ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol options=88 It's always TX, but these servers push ~12x what they receive, so I'm guessing it could happen to either buffer given the right traffic patterns. While looking through commits I also found a patch to add a couple sysctls for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 Thanks, Jason On Wed, Oct 26, 2011 at 3:33 AM, Hooman Fazaeli wrote: > Hi Jason > > Have you tried: > > hw.em.fc_setting="0" (in loader.conf) > ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol > > with MSIX and no multiqueue. > > Advanced features has always been a source of problem. > It is worth a try and help to narrow down possibilities. > > It would also be helpful if you provide 'ifconfig' output > when the problem happens. > > And a question: Does interface RX also hangs or it is just TX? From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 11:53:40 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92EEE106567F; Wed, 26 Oct 2011 11:53:40 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 565178FC2A; Wed, 26 Oct 2011 11:53:40 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 4BACF46B1A; Wed, 26 Oct 2011 07:53:39 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 9F0BE8A037; Wed, 26 Oct 2011 07:53:38 -0400 (EDT) From: John Baldwin To: Pawel Jakub Dawidek Date: Wed, 26 Oct 2011 07:53:36 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20111022084931.GD1697@garage.freebsd.pl> <201110240814.22368.jhb@freebsd.org> <20111026075431.GB1672@garage.freebsd.pl> In-Reply-To: <20111026075431.GB1672@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201110260753.37264.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Wed, 26 Oct 2011 07:53:38 -0400 (EDT) Cc: Kostik Belousov , Lawrence Stewart , freebsd-current@freebsd.org, Andre Oppermann , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 11:53:40 -0000 On Wednesday, October 26, 2011 3:54:31 am Pawel Jakub Dawidek wrote: > On Mon, Oct 24, 2011 at 08:14:22AM -0400, John Baldwin wrote: > > On Sunday, October 23, 2011 11:58:28 am Pawel Jakub Dawidek wrote: > > > On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: > > > > On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: > > > > > My suggestion would be that if we won't be able to fix it before 9.0, > > > > > we should turn this assertion off, as the system seems to be able to > > > > > recover. > > > > > > > > Shipped kernels have all assertions turned off. > > > > > > Yes, I'm aware of that, but many people compile their production kernels > > > with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. > > > corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing > > > it into a printf, so it will be visible. > > > > No, the kernel is corrupting things in other places when this is true, so > > if you are running with INVARIANTS, we want to know about it. Specifically, > > in several places in TCP we assume that rcv_adv >= rcv_nxt, and depend on > > being able to do 'rcv_adv - rcv_nxt'. > > > > In this case, it looks like the difference is consistently less than one > > frame. I suspect the other end of the connection is sending just beyond the > > end of the advertised window (it probably assumes it is better to send a full > > frame if it has that much pending data even though part of it is beyond the > > window edge vs sending a truncated packet that just fills the window) and that > > that frame is accepted ok in the header prediction case and it's ACK is > > delayed, but the next packet to arrive then trips over this assumption. > > > > Since 'win' is guaranteed to be non-negative and we explicitly cast > > 'rcv_adv - rcv_nxt' to (int) in the following line that the assert is checking > > for: > > > > tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); > > > > I think we already handle this case ok and perhaps the assertion can just be > > removed? Not sure if others feel that it warrants a comment to note that this > > is the case being handled. > > I added debug to the places where rcv_adv and rcv_nxt are modified. Here > is what happens before the panic occurs: > > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022361548 rcv_adv 4022360100 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022362298 rcv_adv 4022361548 diff -750 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022363746 rcv_adv 4022362298 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022364836 rcv_adv 4022363746 diff -1090 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022366284 rcv_adv 4022364836 diff -1448 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022370628 rcv_adv 4022369690 diff -938 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022379140 rcv_adv 4022377692 diff -1448 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022387792 rcv_adv 4022386344 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022388890 rcv_adv 4022387792 diff -1098 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022390338 rcv_adv 4022388890 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 diff -221 > panic: tcp_input negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 win=0 diff -221 > > I can send you the full log if you want, I've plenty of messages where > rcv_adv < rcv_nxt, not all of them trigger this assertion. The assertion would be triggered when the next packet arrives (as I said above). Try modifying your debugging output to also log if the ACK is delayed. I suspect it is not delayed until the last one. (Pushing out an ACK will reset rcv_adv to be beyond rcv_nxt in tcp_output(), so in the case of an immediate ACK, rcv_nxt > rcv_adv is only a transient condition all under a single lock invocation so never visible to other consumers of the protocol control block.) If that is what you see, then that confirms what I guessed above and I will likely just remove the assertion in tcp_input() and patch the timewait code to handle this case. -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 13:16:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D2601065675 for ; Wed, 26 Oct 2011 13:16:20 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id A5C578FC14 for ; Wed, 26 Oct 2011 13:16:19 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id p9QDGFbj089388; Wed, 26 Oct 2011 09:16:15 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <4EA80818.3030504@sentex.net> Date: Wed, 26 Oct 2011 09:16:08 -0400 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jason Wolfe References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jack Vogel Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 13:16:20 -0000 On 10/26/2011 7:33 AM, Jason Wolfe wrote: > Hooman, > > I have run with dev.em.X.flow_control=0, which should have the same result > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the > remaining options would be able to produce the scenario I'm seeing, but I'm > open to giving it a try with no options on the interfaces. I've also added > ifconfig output to the collection. > > options=219b > ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol > options=88 > > It's always TX, but these servers push ~12x what they receive, so I'm > guessing it could happen to either buffer given the right traffic patterns. > While looking through commits I also found a patch to add a couple sysctls > for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 Can you provide some more details as to how traffic flows on these servers ? Are they going through them, or are they generating the traffic ? I wonder given a smaller CPU, it would be easier to trigger the condition somehow. We recently got a Soekris 6501 which has onboard 4 such em nics, but has just a 1Ghz Atom. em0@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa1000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa1020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em1@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa2000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa2020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em2@pci0:10:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa3000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x4000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa3020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled em3@pci0:11:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82574L Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xa4000000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x5000, size 32, enabled bar [1c] = type Memory, range 32, base 0xa4020000, size 16384, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled soekris6501# vmstat -i interrupt total rate irq4: uart0 4993 0 cpu0:timer 84947808 71 irq256: ahci0 59195 0 irq257: em0:rx 0 1946546 1 irq258: em0:tx 0 629707 0 irq259: em0:link 2 0 Total 87588251 73 soekris6501# -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 14:04:50 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E0D5106564A for ; Wed, 26 Oct 2011 14:04:50 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id 2BEFD8FC13 for ; Wed, 26 Oct 2011 14:04:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id 285441C0867; Wed, 26 Oct 2011 17:48:17 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.52]) by mailgate.pik.ru (Postfix) with ESMTP id E61A81C0864; Wed, 26 Oct 2011 17:48:14 +0400 (MSD) Received: from [192.168.148.9] ([192.168.148.9]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Wed, 26 Oct 2011 17:48:11 +0400 Message-ID: <4EA80F88.4000400@hotplug.ru> Date: Wed, 26 Oct 2011 17:47:52 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: Mike Tancsa References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> In-Reply-To: <4EA80818.3030504@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 26 Oct 2011 13:48:11.0505 (UTC) FILETIME=[E6C02E10:01CC93E5] Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jack Vogel , Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 14:04:50 -0000 >> Hooman, >> >> I have run with dev.em.X.flow_control=0, which should have the same result >> as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the >> remaining options would be able to produce the scenario I'm seeing, but I'm >> open to giving it a try with no options on the interfaces. I've also added >> ifconfig output to the collection. >> >> options=219b >> ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol >> options=88 >> >> It's always TX, but these servers push ~12x what they receive, so I'm >> guessing it could happen to either buffer given the right traffic patterns. >> While looking through commits I also found a patch to add a couple sysctls >> for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 > Can you provide some more details as to how traffic flows on these servers ? Are they going through them, or are they generating the traffic ? I wonder given a smaller CPU, it would be easier to trigger the condition somehow. We recently got a Soekris 6501 which has onboard 4 such em nics, but has just a 1Ghz Atom. > Hi All, I've got almost the same problem with intel 82574L based nic. My platform is nvidia ion running Atom 1.6 and nic is an external PCI-express adapter. Unlike Jason's case mine is always stuck in receiving traffic, it's Ierrs increasing while Ipkts not. Thanks to Jason's script I can see those locks and interface flapping every several hours. My system is not a heavy loaded server but just a home nas/router, usually routing at 100 mbps or less. Nither disabling MSIX nor tuning txd rxd doesn't help me. From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 14:07:41 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0799A106566B for ; Wed, 26 Oct 2011 14:07:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id A448D8FC0A for ; Wed, 26 Oct 2011 14:07:40 +0000 (UTC) Received: by vcbfo13 with SMTP id fo13so2234951vcb.13 for ; Wed, 26 Oct 2011 07:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=SH52ziEWybgkCbpXQKWFJgnzxE9eorQGoXiRenz0F0A=; b=rLRZOBek4TeycriF2SpBFxe6Yv0W0N+RYpyQmYqGm9WOYpp+k0c3DM+Nj83wPfI9BV Nxiw9Tlk5aq8gw4tSuFX8OOCHZtpjN8BqsYhTWXKVcH0oyCCpz7V5I9fyem0E0pf8s2N kRGRxLDK9ET7MLMd30w8dPIJIrmx9JeRZztjc= MIME-Version: 1.0 Received: by 10.220.1.18 with SMTP id 18mr351788vcd.62.1319638059880; Wed, 26 Oct 2011 07:07:39 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.52.176.1 with HTTP; Wed, 26 Oct 2011 07:07:39 -0700 (PDT) In-Reply-To: <4EA80F88.4000400@hotplug.ru> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> Date: Wed, 26 Oct 2011 22:07:39 +0800 X-Google-Sender-Auth: Ar6LrUEJFemKwSiSTl1bMiXMatc Message-ID: From: Adrian Chadd To: Emil Muratov Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jack Vogel , Jason Wolfe , Mike Tancsa Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 14:07:41 -0000 On 26 October 2011 21:47, Emil Muratov wrote: > Hi All, > I've got almost the same problem with intel 82574L based nic. My platform is > nvidia ion running Atom 1.6 and nic is an external PCI-express adapter. > Unlike Jason's case mine is always stuck in receiving traffic, it's Ierrs > increasing while Ipkts not. Thanks to Jason's script I can see those locks > and interface flapping every several hours. My system is not a heavy loaded > server but just a home nas/router, usually routing at 100 mbps or less. > Nither disabling MSIX nor tuning txd rxd doesn't help me. > Hi, When this occurs, does this completely lock up with RX traffic? Ie, _no_ valid RX traffic occurs? If so, does the error count increase 1:1 with what traffic you're trying to send to it? Adrian From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 15:24:10 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 81263106566B for ; Wed, 26 Oct 2011 15:24:10 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 354C78FC14 for ; Wed, 26 Oct 2011 15:24:09 +0000 (UTC) Received: by gyd8 with SMTP id 8so2191649gyd.13 for ; Wed, 26 Oct 2011 08:24:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=+PYaEuufbB+lePm/oGJuKyJPiVymI0JnDRBnXhxuH30=; b=GsnW+f9w8gEy02PoNQbT+EKeCGd3cINswjiCnzfg3tucy+BQwoMZJ11fyt6GmMFctD I3pUO22iZFHzk28b+WXpnyoFKfDBZxBFwbocwYGqFbvQD4j/yeQ6z4tCbva38WzmyYGk i23jQwBw8kRUqPc8eZdIzZeiRHiGQFSF4oXxw= Received: by 10.150.199.18 with SMTP id w18mr17156659ybf.1.1319642649514; Wed, 26 Oct 2011 08:24:09 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id g38sm5886215ann.4.2011.10.26.08.24.07 (version=SSLv3 cipher=OTHER); Wed, 26 Oct 2011 08:24:09 -0700 (PDT) Message-ID: <4EA82610.6090207@gmail.com> Date: Wed, 26 Oct 2011 18:54:00 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Jason Wolfe References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Hooman Fazaeli Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 15:24:10 -0000 Dear Jason I was actually interested in "status:" and "flags=" line from ifconfig output _at_ the time problem happens. From your previous post, it is clear that when the driver stops transmitting, output errors are zero and "Drops" increasing. This happens when the stack uses IFQ_ENQUEUE to put packet in interface output queue and the queue is full. The are a few possibilities for interface queue being full: - if_start has not been called regularly by stack. This is unlikely in your case since it affect all drivers not just em. - if_em has not been fast enough to IFQ_DRV_DEQUEUE the packets. This is also unlikely because you would otherwise see occasional drops not total TX hang. The other reasons may be found by looking at em_start() function: - Driver may be OACTIVE, that is, it is busy sending packets. This may not be your case because 'sysctl dev.em.1.debug=1' shows that driver is INACTIVE. - Number of available TX descriptors falls below EM_MAX_SCATTER (64) and driver never recovers. This possibility is also rules out because em debug output shows that 1014 descriptors are available. - Link may not be active (or at least, the may PHY believe so). In this case, IFQ_DRV_DEQUEUE is never called and interface queue becomes full. This was the reason I asked you ifconfig output. You may add the following line to the end of em_print_debug_info function to report link status when the problem happens: device_printf(dev, "Link state: %s\n", adapter->link_active? "active": "inactive"); On 10/26/2011 3:03 PM, Jason Wolfe wrote: > Hooman, > > I have run with dev.em.X.flow_control=0, which should have the same result > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the > remaining options would be able to produce the scenario I'm seeing, but I'm > open to giving it a try with no options on the interfaces. I've also added > ifconfig output to the collection. > > options=219b > ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol > options=88 > > It's always TX, but these servers push ~12x what they receive, so I'm > guessing it could happen to either buffer given the right traffic patterns. > While looking through commits I also found a patch to add a couple sysctls > for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 > > Thanks, > Jason > > On Wed, Oct 26, 2011 at 3:33 AM, Hooman Fazaeli wrote: > >> Hi Jason >> >> Have you tried: >> >> hw.em.fc_setting="0" (in loader.conf) >> ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol >> >> with MSIX and no multiqueue. >> >> Advanced features has always been a source of problem. >> It is worth a try and help to narrow down possibilities. >> >> It would also be helpful if you provide 'ifconfig' output >> when the problem happens. >> >> And a question: Does interface RX also hangs or it is just TX? > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 15:28:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F77E106566B for ; Wed, 26 Oct 2011 15:28:33 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id E57348FC08 for ; Wed, 26 Oct 2011 15:28:32 +0000 (UTC) Received: by ggnq2 with SMTP id q2so2202438ggn.13 for ; Wed, 26 Oct 2011 08:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=g/59Ox9ZGuVT14wtRS2NV0awNAUYAmu3vcgcRTTF5jU=; b=lIEJp1q+5yng+azjUgqzdWvxVjpjBYZwuh+JAhtsF71Zt5vS5R2I2NvW3T86ULLeMR Ovle954g0biu16/Wqh+S61g9I0gsbRP4R5/Wm8uF6X320ixCNjT+WSHW8++/sqFBz1T2 TdXRv7VuTt6N9vALrfB0/74SC07r7FKbJ/SLU= Received: by 10.101.193.31 with SMTP id v31mr4171148anp.159.1319642912174; Wed, 26 Oct 2011 08:28:32 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id k3sm4349626ann.0.2011.10.26.08.28.26 (version=SSLv3 cipher=OTHER); Wed, 26 Oct 2011 08:28:29 -0700 (PDT) Message-ID: <4EA82715.2000404@gmail.com> Date: Wed, 26 Oct 2011 18:58:21 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Emil Muratov References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> In-Reply-To: <4EA80F88.4000400@hotplug.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Jack Vogel , Jason Wolfe , Mike Tancsa Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 15:28:33 -0000 Hi, Can yan you pls post the output of these command _when_ the problem happens? uname -a sysctl dev.em netstat -ind ifconfig > I've got almost the same problem with intel 82574L based nic. My platform is nvidia ion running Atom 1.6 and nic is an external PCI-express adapter. Unlike Jason's case mine is always stuck in > receiving traffic, it's Ierrs increasing while Ipkts not. Thanks to Jason's script I can see those locks and interface flapping every several hours. My system is not a heavy loaded server but just a > home nas/router, usually routing at 100 mbps or less. Nither disabling MSIX nor tuning txd rxd doesn't help me. > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 15:32:06 2011 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFE341065670 for ; Wed, 26 Oct 2011 15:32:06 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from cell.glebius.int.ru (glebius.int.ru [81.19.64.117]) by mx1.freebsd.org (Postfix) with ESMTP id 3067D8FC0A for ; Wed, 26 Oct 2011 15:32:05 +0000 (UTC) Received: from cell.glebius.int.ru (localhost [127.0.0.1]) by cell.glebius.int.ru (8.14.4/8.14.4) with ESMTP id p9QFW4j3052735 for ; Wed, 26 Oct 2011 19:32:04 +0400 (MSD) (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by cell.glebius.int.ru (8.14.4/8.14.4/Submit) id p9QFW4Lf052734 for net@FreeBSD.org; Wed, 26 Oct 2011 19:32:04 +0400 (MSD) (envelope-from glebius@FreeBSD.org) X-Authentication-Warning: cell.glebius.int.ru: glebius set sender to glebius@FreeBSD.org using -f Date: Wed, 26 Oct 2011 19:32:04 +0400 From: Gleb Smirnoff To: net@FreeBSD.org Message-ID: <20111026153204.GD36064@glebius.int.ru> References: <20110810160526.GO43567@FreeBSD.org> <20111013160216.GU94905@glebius.int.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline In-Reply-To: <20111013160216.GU94905@glebius.int.ru> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Subject: Re: new CARP implementation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 15:32:06 -0000 Hello networkers, I've rolled out a new patch & README here: http://people.freebsd.org/~glebius/newcarp/ The most important change since last version is not sending spurious graturious ARP announce on startup. -- Totus tuus, Glebius. From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 15:49:52 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FE05106564A for ; Wed, 26 Oct 2011 15:49:52 +0000 (UTC) (envelope-from freebsd-net@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 195FB8FC16 for ; Wed, 26 Oct 2011 15:49:51 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1RJ5jg-0005jh-5M for freebsd-net@freebsd.org; Wed, 26 Oct 2011 17:49:48 +0200 Received: from 208.85.208.53 ([208.85.208.53]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 26 Oct 2011 17:49:48 +0200 Received: from atkin901 by 208.85.208.53 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 26 Oct 2011 17:49:48 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-net@freebsd.org From: Mark Atkinson Date: Wed, 26 Oct 2011 08:49:29 -0700 Lines: 32 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 208.85.208.53 User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 In-Reply-To: X-Enigmail-Version: undefined Subject: Re: FreeBSD 9 IPv6 conformance test report X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 15:49:52 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/25/2011 14:22, Qing Li wrote: > Since I have been receiving private reports on the test failures, I > thought I just post the test report as executed against the > upcoming FreeBSD 9.0 release in my own setup, so you can see where > the baseline is. > > More test reports will be posted here. > > http://people.freebsd.org/~qingli/ct-freebsd-9/ > > > I am going over each failure and providing the fix. > > Please let me know if you already have a suggested patch that you > are willing to share. Are the ct-2.1.1 suite results still a good baseline? From what I understand the v6 ready logo phase 2 tests are what people are comparing their compliance against currently. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6oLAkACgkQrDN5kXnx8ybkUACeIWdJfUHnMmLnipiJGN8eMqm8 MYEAnAuOIxwfi7ZeqYo40VHxN3b8NuNc =0n4k -----END PGP SIGNATURE----- From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 16:49:41 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 89810106564A for ; Wed, 26 Oct 2011 16:49:41 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 154378FC0A for ; Wed, 26 Oct 2011 16:49:40 +0000 (UTC) Received: by wwi18 with SMTP id 18so2662243wwi.31 for ; Wed, 26 Oct 2011 09:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=B/TyuVbD6EqAkhk4xNQBeYhdmRFvYAGRFDlJcm+Zdhk=; b=wnH74n/FUMAS1RkV3KxKh3JNRKInMeh4O4yNhmGt4KsxyBF/O9ZyKlzAtOGc+Q2gRO PTEjYcT+JlcFYkqUzs4xkpn2sqppOda1G5lmd0dMaKgiEFCe5tq2xzQgIewyGm+trzWJ e0lt78AJN4ba21glp5ZiZi0/TYF3I/VuPmAqU= Received: by 10.227.205.76 with SMTP id fp12mr12100881wbb.17.1319647779961; Wed, 26 Oct 2011 09:49:39 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id ei16sm4161744wbb.21.2011.10.26.09.49.36 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 26 Oct 2011 09:49:38 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 26 Oct 2011 09:48:00 -0700 From: YongHyeon PYUN Date: Wed, 26 Oct 2011 09:48:00 -0700 To: Andrey Smagin Message-ID: <20111026164800.GA13234@michelle.cdnetworks.com> References: <20111025200937.GA9348@michelle.cdnetworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 16:49:41 -0000 On Wed, Oct 26, 2011 at 01:49:14PM +0400, Andrey Smagin wrote: > Hi ! > vge0@pci0:2:0:0: class=0x020000 card=0x01101106 chip=0x31191106 rev=0x82 hdr=0x00 > vendor = 'VIA Technologies, Inc.' > device = 'VT6120/VT6121/VT6122 Gigabit Ethernet Adapter' > class = network > subclass = ethernet > bar [10] = type I/O Port, range 32, base 0x8000, size 256, enabled > bar [14] = type Memory, range 64, base 0xd4100000, size 256, enabled > cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0 > cap 10[90] = PCI-Express 1 endpoint max data 128(128) link x1(x1) > cap 05[c0] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message > ecap 0001[100] = AER 1 0 fatal 1 non-fatal 6 corrected > ecap 0003[130] = Serial 1 0000110600000000 > vge1@pci0:1:0:0: class=0x020000 card=0x01101106 chip=0x31191106 rev=0x82 hdr=0x00 > vendor = 'VIA Technologies, Inc.' > device = 'VT6120/VT6121/VT6122 Gigabit Ethernet Adapter' > class = network > subclass = ethernet > bar [10] = type I/O Port, range 32, base 0x7000, size 256, enabled > bar [14] = type Memory, range 64, base 0xd4000000, size 256, enabled > cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0 > cap 10[90] = PCI-Express 1 endpoint max data 128(128) link x1(x1) > cap 05[c0] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message > ecap 0001[100] = AER 1 1 fatal 1 non-fatal 6 corrected > ecap 0003[130] = Serial 1 0000110600000000 > > dmesg is empty Hmm, check whether you have old dmesg file in /var/log directory. At least, I need output of 'devinfo -rv' to know which PHY you have. By chance, are you using manual link configuration instead of relying on auto-negotiation? > > 26 октÑÐ±Ñ€Ñ 2011, 00:11 от YongHyeon PYUN : > > On Tue, Oct 25, 2011 at 10:44:48AM +0400, Andrey Smagin wrote: > > > Hi ALL ! If I connect gigabit switch to my card - system hang until I unplug patchcord from device. > > > With 100Mbit switch card work good. > > > > Show me the output of dmesg and 'pciconf -lcbv'. > > > > > System: Current - r226163 > > > > > > From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 18:11:18 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29A5F1065673; Wed, 26 Oct 2011 18:11:18 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id E3A968FC15; Wed, 26 Oct 2011 18:11:17 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p9QHb4aN057563 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 26 Oct 2011 10:37:07 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4EA8453B.2090808@freebsd.org> Date: Wed, 26 Oct 2011 10:36:59 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <20111022084931.GD1697@garage.freebsd.pl> <20111023084445.GB50300@deviant.kiev.zoral.com.ua> <20111023155827.GH1697@garage.freebsd.pl> <201110240814.22368.jhb@freebsd.org> <20111026075431.GB1672@garage.freebsd.pl> In-Reply-To: <20111026075431.GB1672@garage.freebsd.pl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Andre Oppermann , John Baldwin , freebsd-net@freebsd.org, freebsd-current@freebsd.org, Kostik Belousov , Lawrence Stewart Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 18:11:18 -0000 On 10/26/11 12:54 AM, Pawel Jakub Dawidek wrote: > On Mon, Oct 24, 2011 at 08:14:22AM -0400, John Baldwin wrote: >> On Sunday, October 23, 2011 11:58:28 am Pawel Jakub Dawidek wrote: >>> On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: >>>> On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: >>>>> My suggestion would be that if we won't be able to fix it before 9.0, >>>>> we should turn this assertion off, as the system seems to be able to >>>>> recover. >>>> Shipped kernels have all assertions turned off. >>> Yes, I'm aware of that, but many people compile their production kernels >>> with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. >>> corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing >>> it into a printf, so it will be visible. >> No, the kernel is corrupting things in other places when this is true, so >> if you are running with INVARIANTS, we want to know about it. Specifically, >> in several places in TCP we assume that rcv_adv>= rcv_nxt, and depend on >> being able to do 'rcv_adv - rcv_nxt'. >> >> In this case, it looks like the difference is consistently less than one >> frame. I suspect the other end of the connection is sending just beyond the >> end of the advertised window (it probably assumes it is better to send a full >> frame if it has that much pending data even though part of it is beyond the >> window edge vs sending a truncated packet that just fills the window) and that >> that frame is accepted ok in the header prediction case and it's ACK is >> delayed, but the next packet to arrive then trips over this assumption. >> >> Since 'win' is guaranteed to be non-negative and we explicitly cast >> 'rcv_adv - rcv_nxt' to (int) in the following line that the assert is checking >> for: >> >> tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); >> >> I think we already handle this case ok and perhaps the assertion can just be >> removed? Not sure if others feel that it warrants a comment to note that this >> is the case being handled. > I added debug to the places where rcv_adv and rcv_nxt are modified. Here > is what happens before the panic occurs: > > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022361548 rcv_adv 4022360100 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022362298 rcv_adv 4022361548 diff -750 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022363746 rcv_adv 4022362298 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022364836 rcv_adv 4022363746 diff -1090 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022366284 rcv_adv 4022364836 diff -1448 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022370628 rcv_adv 4022369690 diff -938 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022379140 rcv_adv 4022377692 diff -1448 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022387792 rcv_adv 4022386344 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022388890 rcv_adv 4022387792 diff -1098 > tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022390338 rcv_adv 4022388890 diff -1448 > tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 diff -221 > panic: tcp_input negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 win=0 diff -221 > > I can send you the full log if you want, I've plenty of messages where > rcv_adv< rcv_nxt, not all of them trigger this assertion. > Might be a good place to use the new sifter tool. From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 21:18:17 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 572D8106566B for ; Wed, 26 Oct 2011 21:18:17 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id CF6668FC16 for ; Wed, 26 Oct 2011 21:18:16 +0000 (UTC) Received: by bkbzt4 with SMTP id zt4so2438639bkb.13 for ; Wed, 26 Oct 2011 14:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cfwOm/y5DU33cN3LSXbnGv+ZZyCz2fH+GQTj1TMjz9E=; b=RRfE0x88vk73ZjMPYIEw5Vjw2Ic9+d4qOkFlu+XaAcCoiERLh4gsFcFqiKPBNVdRWp 6s/Lo0BS9LHRVwpHk7G4pzXJnVo4+Uo1dFzQRvsiiRvZ4noyGJ3kk4ubxsrUdVPKoRCu GYVEZJTsqrH0pY1kTnFinZBB5zOJ1aDxmSOh8= MIME-Version: 1.0 Received: by 10.182.115.40 with SMTP id jl8mr5751690obb.8.1319663894767; Wed, 26 Oct 2011 14:18:14 -0700 (PDT) Received: by 10.182.35.193 with HTTP; Wed, 26 Oct 2011 14:18:14 -0700 (PDT) In-Reply-To: <4EA82610.6090207@gmail.com> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA82610.6090207@gmail.com> Date: Wed, 26 Oct 2011 14:18:14 -0700 Message-ID: From: Jason Wolfe To: Hooman Fazaeli Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Hooman Fazaeli Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 21:18:17 -0000 Hooman, I've added ifconfig to my collection script (as well as sysctl hw.em) that will run at the time of the next wedge, the ifconfig lines in my email were just to show the before and after as I've also modified the running options on emX in this test group as you proposed. Thank you for the details. I know the team has fixed many bugs related to this chip in the past year or so and most folks are running great, so I'm hoping we can squash this seemingly last and rare one related to interface hangs. If not a bug and rather something about my implementation, I can at least serve as a cautionary tale :) I've added the dev_printf you proposed also, as of today the test pool is running with that as well as what's mentioned above. Thanks, Jason On Wed, Oct 26, 2011 at 8:24 AM, Hooman Fazaeli wrote: > > Dear Jason > > I was actually interested in "status:" and "flags=" line from ifconfig > output > _at_ the time problem happens. > > From your previous post, it is clear that when the driver stops > transmitting, > output errors are zero and "Drops" increasing. This happens when the stack > uses > IFQ_ENQUEUE to put packet in interface output queue and the queue is full. > The > are a few possibilities for interface queue being full: > > - if_start has not been called regularly by stack. This is unlikely in > your case since it affect all drivers not just em. > > - if_em has not been fast enough to IFQ_DRV_DEQUEUE the packets. This is > also unlikely because you would otherwise see occasional drops not total > TX hang. > > The other reasons may be found by looking at em_start() function: > > - Driver may be OACTIVE, that is, it is busy sending packets. This may not > be > your case because 'sysctl dev.em.1.debug=1' shows that driver is INACTIVE. > > - Number of available TX descriptors falls below EM_MAX_SCATTER (64) and > driver > never recovers. This possibility is also rules out because > em debug output shows that 1014 descriptors are available. > > - Link may not be active (or at least, the may PHY believe so). > In this case, IFQ_DRV_DEQUEUE is never called and interface queue becomes > full. > This was the reason I asked you ifconfig output. You may add the > following line to the end of em_print_debug_info function > to report link status when the problem happens: > > device_printf(dev, "Link state: %s\n", > adapter->link_active? "active": "inactive"); > > > > > On 10/26/2011 3:03 PM, Jason Wolfe wrote: > >> Hooman, >> >> I have run with dev.em.X.flow_control=0, which should have the same result >> as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the >> remaining options would be able to produce the scenario I'm seeing, but >> I'm >> open to giving it a try with no options on the interfaces. I've also >> added >> ifconfig output to the collection. >> >> >> options=219b >> ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol >> options=88 >> >> It's always TX, but these servers push ~12x what they receive, so I'm >> guessing it could happen to either buffer given the right traffic >> patterns. >> While looking through commits I also found a patch to add a couple >> sysctls >> for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 >> >> Thanks, >> Jason >> >> On Wed, Oct 26, 2011 at 3:33 AM, Hooman Fazaeli >> wrote: >> >> Hi Jason >>> >>> Have you tried: >>> >>> hw.em.fc_setting="0" (in loader.conf) >>> ifconfig emX -tso -lro -rxcsum -txcsum -vlanhwtag -wol >>> >>> with MSIX and no multiqueue. >>> >>> Advanced features has always been a source of problem. >>> It is worth a try and help to narrow down possibilities. >>> >>> It would also be helpful if you provide 'ifconfig' output >>> when the problem happens. >>> >>> And a question: Does interface RX also hangs or it is just TX? >>> >> _______________________________________________ >> >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > From owner-freebsd-net@FreeBSD.ORG Wed Oct 26 22:04:45 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FA601065675 for ; Wed, 26 Oct 2011 22:04:45 +0000 (UTC) (envelope-from nitroboost@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id BC1588FC14 for ; Wed, 26 Oct 2011 22:04:44 +0000 (UTC) Received: by eyd10 with SMTP id 10so2600967eyd.13 for ; Wed, 26 Oct 2011 15:04:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1yWAAv/q1ZSn7Z7wIlVpW6QLcHOUn+Bsk04AUg+TXls=; b=vNT3ufBYCAclQqpvQaDYHszshtlzSoUMzN5YzRDmhOGuSfmKWQrOIOwDBpk6QlgXjm GTnE9REnG9V02p3cKdDuYpkHmtm1IeVALzNCsb5XeKRWO3HFn/bT6BFSA8q71xkIn/Qj 1IYy9EBxbzJpHNxsudqpEYxcD+srfHyBFlTCg= MIME-Version: 1.0 Received: by 10.182.226.33 with SMTP id rp1mr2225719obc.18.1319666683344; Wed, 26 Oct 2011 15:04:43 -0700 (PDT) Received: by 10.182.35.193 with HTTP; Wed, 26 Oct 2011 15:04:43 -0700 (PDT) In-Reply-To: <4EA80818.3030504@sentex.net> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> Date: Wed, 26 Oct 2011 15:04:43 -0700 Message-ID: From: Jason Wolfe To: Mike Tancsa Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jack Vogel Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Oct 2011 22:04:45 -0000 On Wed, Oct 26, 2011 at 6:16 AM, Mike Tancsa wrote: > On 10/26/2011 7:33 AM, Jason Wolfe wrote: > > Hooman, > > > > I have run with dev.em.X.flow_control=0, which should have the same > result > > as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0. I'm not sure the > > remaining options would be able to produce the scenario I'm seeing, but > I'm > > open to giving it a try with no options on the interfaces. I've also > added > > ifconfig output to the collection. > > > > > options=219b > > ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol > > options=88 > > > > It's always TX, but these servers push ~12x what they receive, so I'm > > guessing it could happen to either buffer given the right traffic > patterns. > > While looking through commits I also found a patch to add a couple > sysctls > > for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676 > > Can you provide some more details as to how traffic flows on these servers > ? Are they going through them, or are they generating the traffic ? I > wonder given a smaller CPU, it would be easier to trigger the condition > somehow. We recently got a Soekris 6501 which has onboard 4 such em nics, > but has just a 1Ghz Atom. > > > em0@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa1000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa1020000, size 16384, > enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em1@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa2000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa2020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em2@pci0:10:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa3000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x4000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa3020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > em3@pci0:11:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xa4000000, size 131072, > enabled > bar [18] = type I/O Port, range 32, base 0x5000, size 32, enabled > bar [1c] = type Memory, range 32, base 0xa4020000, size 16384, enabled > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > cap 05[d0] = MSI supports 1 message, 64 bit > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > > soekris6501# vmstat -i > interrupt total rate > irq4: uart0 4993 0 > cpu0:timer 84947808 71 > irq256: ahci0 59195 0 > irq257: em0:rx 0 1946546 1 > irq258: em0:tx 0 629707 0 > irq259: em0:link 2 0 > Total 87588251 73 > soekris6501# > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/ > Mike, There is actually a mix of traffic going through the machine and traffic originating from the box itself, though it is ~75% coming from the local disk. Let us know if you have any issues with that Soekris once it's loaded up. Though in my case the hang is still somewhat rare, it works out to roughly once every 4 months, though each box has 2 x 82574L. I've just set aside enough boxes to reproduce about once a week, which gives them time to settle in and not have the quest take up too large a chunk of my time :) Jason From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 06:30:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91C19106564A for ; Thu, 27 Oct 2011 06:30:21 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id 280478FC1B for ; Thu, 27 Oct 2011 06:30:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id 2F6551C084A; Thu, 27 Oct 2011 10:30:18 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.52]) by mailgate.pik.ru (Postfix) with ESMTP id 673681C0840; Thu, 27 Oct 2011 10:30:15 +0400 (MSD) Received: from [192.168.148.9] ([192.168.148.9]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Oct 2011 10:29:39 +0400 Message-ID: <4EA8FA40.7010504@hotplug.ru> Date: Thu, 27 Oct 2011 10:29:20 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: Hooman Fazaeli References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> In-Reply-To: <4EA82715.2000404@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 27 Oct 2011 06:29:39.0035 (UTC) FILETIME=[CDB5A2B0:01CC9471] Cc: freebsd-net@freebsd.org, Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 06:30:21 -0000 > Hi, > > Can yan you pls post the output of these command _when_ the problem > happens? > > uname -a > sysctl dev.em > netstat -ind > ifconfig > Hi Hooman Here is what I've got when the script triggered just in time when the interface was locked 11.10.26-23:39:10 ... interface em0 is down... FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 20:20:25 MSD 2011 root@epia.home .lan:/usr/obj/usr/src/sys/ION6debug amd64 11:39PM up 1:12, 2 users, load averages: 0.26, 0.48, 0.58 == vmstat -i == interrupt total rate irq22: nfe0 16644480 3865 cpu0: timer 8610122 1999 irq256: ahci0 606705 140 irq257: em0:rx 0 3896622 904 irq258: em0:tx 0 2762957 641 irq259: em0:link 620 0 cpu3: timer 8609499 1999 cpu1: timer 8609499 1999 cpu2: timer 8609499 1999 Total 58350003 13550 == netstat -ind == Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop usbus 0 0 0 0 0 0 0 0 usbus 0 0 0 0 0 0 0 0 nfe0 1500 00:25:22:21:86:89 7157140 0 0 12266747 0 0 0 nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - - 85 - - - nfe0 1500 10.16.128.0/1 10.16.189.71 0 - - 48135 - - - em0 9000 00:1b:21:ab:bf:4a 5465087 623 0 2862028 0 0 113 em0 9000 192.168.168.0 192.168.168.1 764085 - - 1005078 - - - em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - 252 - - - em0 9000 2002:d58d:871 2002:d58d:8715:1: 73 - - 38 - - - wifi 1500 00:1b:21:ab:bf:4a 347 0 0 350 0 0 0 wifi 1500 192.168.168.6 192.168.168.65 0 - - 0 - - - wifi 1500 fe80::225:x fe80::225:x:x 0 - - 349 - - - wifi 1500 2002:x:x 2002:x:x:2: 0 - - 0 - - - wifio 1500 00:1b:21:ab:bf:4a 59559 0 0 114639 0 0 0 wifio 1500 192.168.168.8 192.168.168.81 0 - - 160 - - - wifio 1500 fe80::225:x fe80::225:x:x 0 - - 0 - - - stf0 1280 5725 0 0 6125 420 0 0 stf0 1280 2002:x:x 2002:x:x::1 1878 - - 1121 - - - ng0* 1500 0 0 0 0 0 0 0 ng1* 1500 0 0 0 0 0 0 0 ng2 1492 7143733 0 0 12234436 0 0 0 ng2 1492 213.141.x.x 213.141.x.x 4735932 - - 8480089 - - - ng2 1492 fe80::x:x fe80::x:x:x 0 - - 1 - - - tun0 1455 350 0 0 172 0 0 0 tun0 1455 fe80::225:x fe80::225:x:x 0 - - 2 - - - tun0 1455 192.168.169.1 192.168.169.1 117 - - 167 - - - Oct 26 23:39:11 ion kernel: em0: hw tdh = 975, hw tdt = 944 Oct 26 23:39:11 ion kernel: em0: hw rdh = 960, hw rdt = 959 Oct 26 23:39:11 ion kernel: em0: Tx Queue Status = 1 Oct 26 23:39:11 ion kernel: em0: TX descriptors avail = 31 Oct 26 23:39:11 ion kernel: em0: Tx Descriptors avail failure = 0 Oct 26 23:39:11 ion kernel: em0: RX discarded packets = 0 Oct 26 23:39:11 ion kernel: em0: RX Next to Check = 960 Oct 26 23:39:11 ion kernel: em0: RX Next to Refresh = 959 net.inet.ip.intr_queue_maxlen: 4096 net.inet.ip.intr_queue_drops: 0 dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 dev.em.0.%driver: em dev.em.0.%location: slot=0 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 dev.em.0.%parent: pci2 dev.em.0.nvm: -1 dev.em.0.debug: -1 dev.em.0.rx_int_delay: 200 dev.em.0.tx_int_delay: 200 dev.em.0.rx_abs_int_delay: 4096 dev.em.0.tx_abs_int_delay: 4096 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.eee_control: 0 dev.em.0.link_irq: 648 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1477444168 dev.em.0.rx_control: 100827170 dev.em.0.fc_high_water: 11264 dev.em.0.fc_low_water: 9764 dev.em.0.queue0.txd_head: 975 dev.em.0.queue0.txd_tail: 944 dev.em.0.queue0.tx_irq: 2762762 dev.em.0.queue0.no_desc_avail: 0 dev.em.0.queue0.rxd_head: 960 dev.em.0.queue0.rxd_tail: 959 dev.em.0.queue0.rx_irq: 3895860 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 647 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 438789 dev.em.0.mac_stats.xon_txd: 366 dev.em.0.mac_stats.xoff_recvd: 438789 dev.em.0.mac_stats.xoff_txd: 1013 dev.em.0.mac_stats.total_pkts_recvd: 5465524 dev.em.0.mac_stats.good_pkts_recvd: 4587299 dev.em.0.mac_stats.bcast_pkts_recvd: 1102 dev.em.0.mac_stats.mcast_pkts_recvd: 162 dev.em.0.mac_stats.rx_frames_64: 325765 dev.em.0.mac_stats.rx_frames_65_127: 1029229 dev.em.0.mac_stats.rx_frames_128_255: 118432 dev.em.0.mac_stats.rx_frames_256_511: 11360 dev.em.0.mac_stats.rx_frames_512_1023: 100708 dev.em.0.mac_stats.rx_frames_1024_1522: 3001805 dev.em.0.mac_stats.good_octets_recvd: 4648591681 dev.em.0.mac_stats.good_octets_txd: 2203060494 dev.em.0.mac_stats.total_pkts_txd: 3780652 dev.em.0.mac_stats.good_pkts_txd: 3779273 dev.em.0.mac_stats.bcast_pkts_txd: 89 dev.em.0.mac_stats.mcast_pkts_txd: 534 dev.em.0.mac_stats.tx_frames_64: 1323163 dev.em.0.mac_stats.tx_frames_65_127: 850801 dev.em.0.mac_stats.tx_frames_128_255: 193136 dev.em.0.mac_stats.tx_frames_256_511: 64088 dev.em.0.mac_stats.tx_frames_512_1023: 47149 dev.em.0.mac_stats.tx_frames_1024_1522: 1300936 dev.em.0.mac_stats.tso_txd: 429804 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.interrupts.asserts: 44 dev.em.0.interrupts.rx_pkt_timer: 0 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.tx_pkt_timer: 0 dev.em.0.interrupts.tx_abs_timer: 0 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_queue_min_thresh: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.rx_overrun: 0 ifconfig em0 em0: flags=8c43 metric 0 mtu 9000 description: LAN options=219b ether 00:1b:21:ab:bf:4a inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 inet6 2002:x:x:1::1 prefixlen 64 nd6 options=1 media: Ethernet autoselect (1000baseT ) status: active > >> I've got almost the same problem with intel 82574L based nic. My >> platform is nvidia ion running Atom 1.6 and nic is an external >> PCI-express adapter. Unlike Jason's case mine is always stuck in >> receiving traffic, it's Ierrs increasing while Ipkts not. Thanks to >> Jason's script I can see those locks and interface flapping every >> several hours. My system is not a heavy loaded server but just a home >> nas/router, usually routing at 100 mbps or less. Nither disabling >> MSIX nor tuning txd rxd doesn't help me. >> From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 07:01:08 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6DB91106564A for ; Thu, 27 Oct 2011 07:01:08 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id C3F5F8FC13 for ; Thu, 27 Oct 2011 07:01:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id 5F1651C08BD; Thu, 27 Oct 2011 11:01:06 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.51]) by mailgate.pik.ru (Postfix) with ESMTP id D346E1C08A2; Thu, 27 Oct 2011 11:01:05 +0400 (MSD) Received: from EX21PIK.PICompany.ru ([192.168.156.131]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Oct 2011 11:00:24 +0400 Received: from [192.168.148.9] (192.168.148.9) by EX21PIK.PICompany.ru (192.168.156.131) with Microsoft SMTP Server id 14.1.218.12; Thu, 27 Oct 2011 10:51:09 +0400 Message-ID: <4EA8FF4B.3070704@hotplug.ru> Date: Thu, 27 Oct 2011 10:50:51 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: Adrian Chadd References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 27 Oct 2011 07:00:24.0420 (UTC) FILETIME=[19A50A40:01CC9476] Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jason, Wolfe , Mike Tancsa Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 07:01:08 -0000 >> Hi All, >> I've got almost the same problem with intel 82574L based nic. My platform is >> nvidia ion running Atom 1.6 and nic is an external PCI-express adapter. >> Unlike Jason's case mine is always stuck in receiving traffic, it's Ierrs >> increasing while Ipkts not. Thanks to Jason's script I can see those locks >> and interface flapping every several hours. My system is not a heavy loaded >> server but just a home nas/router, usually routing at 100 mbps or less. >> Nither disabling MSIX nor tuning txd rxd doesn't help me. >> > Hi, > > When this occurs, does this completely lock up with RX traffic? Ie, > _no_ valid RX traffic occurs? > If so, does the error count increase 1:1 with what traffic you're > trying to send to it? > Hi Adrian Yes, it's completely locked - ping packet loss to the interface is 100%, I also managed to run tcpdump -ni em0 and there were only outgoing packets from the interface but no packets arrived to the interface. As for the error counter ratio - I'm not sure about it, don't know how check this besides comparing counters on the switch with interface, but it's a little bit complicated to catch in time. From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 08:37:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53CBE106564A for ; Thu, 27 Oct 2011 08:37:21 +0000 (UTC) (envelope-from hoomanfazaeli@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 05B0E8FC15 for ; Thu, 27 Oct 2011 08:37:20 +0000 (UTC) Received: by ggnq2 with SMTP id q2so3206848ggn.13 for ; Thu, 27 Oct 2011 01:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=Q87UQGNphDJx1rZbZbDlOk/NZVxQRKAs0dctJQ3A0S4=; b=d1gx3D+1AErsytdtQbE7/drWaPNMVcxQkD2UMqyTnJf+vrf4rr5+UDDv8IGhyemJ7O jjY5EDDYm+Zh/gT7IUVGT348OC40o2UxMJU6ER4fzwDmVeSnXOWKNYDsb8YjKftzLeI7 rvvcT7ZGY2Ovde/V+rwuHPP2Ft8qeee9/E0Os= Received: by 10.151.149.4 with SMTP id b4mr31417254ybo.33.1319704640074; Thu, 27 Oct 2011 01:37:20 -0700 (PDT) Received: from [127.0.0.1] ([84.241.57.181]) by mx.google.com with ESMTPS id l27sm12875024ani.21.2011.10.27.01.37.17 (version=SSLv3 cipher=OTHER); Thu, 27 Oct 2011 01:37:19 -0700 (PDT) Message-ID: <4EA91836.2040508@gmail.com> Date: Thu, 27 Oct 2011 12:07:10 +0330 From: Hooman Fazaeli User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: Emil Muratov References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> In-Reply-To: <4EA8FA40.7010504@hotplug.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 08:37:21 -0000 On 10/27/2011 9:59 AM, Emil Muratov wrote: > > Hi Hooman > > Here is what I've got when the script triggered just in time when the interface was locked > > > 11.10.26-23:39:10 ... interface em0 is down... > > FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 20:20:25 MSD 2011 root@epia.home > .lan:/usr/obj/usr/src/sys/ION6debug amd64 > 11:39PM up 1:12, 2 users, load averages: 0.26, 0.48, 0.58 > > > == vmstat -i == > interrupt total rate > irq22: nfe0 16644480 3865 > cpu0: timer 8610122 1999 > irq256: ahci0 606705 140 > irq257: em0:rx 0 3896622 904 > irq258: em0:tx 0 2762957 641 > irq259: em0:link 620 0 > cpu3: timer 8609499 1999 > cpu1: timer 8609499 1999 > cpu2: timer 8609499 1999 > Total 58350003 13550 > > == netstat -ind == > Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop > usbus 0 0 0 0 0 0 0 0 > usbus 0 0 0 0 0 0 0 0 > nfe0 1500 00:25:22:21:86:89 7157140 0 0 12266747 0 0 0 > nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - - 85 - - - > nfe0 1500 10.16.128.0/1 10.16.189.71 0 - - 48135 - - - > em0 9000 00:1b:21:ab:bf:4a 5465087 623 0 2862028 0 0 113 > em0 9000 192.168.168.0 192.168.168.1 764085 - - 1005078 - - - > em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - 252 - - - > em0 9000 2002:d58d:871 2002:d58d:8715:1: 73 - - 38 - - - > wifi 1500 00:1b:21:ab:bf:4a 347 0 0 350 0 0 0 > wifi 1500 192.168.168.6 192.168.168.65 0 - - 0 - - - > wifi 1500 fe80::225:x fe80::225:x:x 0 - - 349 - - - > wifi 1500 2002:x:x 2002:x:x:2: 0 - - 0 - - - > wifio 1500 00:1b:21:ab:bf:4a 59559 0 0 114639 0 0 0 > wifio 1500 192.168.168.8 192.168.168.81 0 - - 160 - - - > wifio 1500 fe80::225:x fe80::225:x:x 0 - - 0 - - - > stf0 1280 5725 0 0 6125 420 0 0 > stf0 1280 2002:x:x 2002:x:x::1 1878 - - 1121 - - - > ng0* 1500 0 0 0 0 0 0 0 > ng1* 1500 0 0 0 0 0 0 0 > ng2 1492 7143733 0 0 12234436 0 0 0 > ng2 1492 213.141.x.x 213.141.x.x 4735932 - - 8480089 - - - > ng2 1492 fe80::x:x fe80::x:x:x 0 - - 1 - - - > tun0 1455 350 0 0 172 0 0 0 > tun0 1455 fe80::225:x fe80::225:x:x 0 - - 2 - - - > tun0 1455 192.168.169.1 192.168.169.1 117 - - 167 - - - > > Oct 26 23:39:11 ion kernel: em0: hw tdh = 975, hw tdt = 944 > Oct 26 23:39:11 ion kernel: em0: hw rdh = 960, hw rdt = 959 > Oct 26 23:39:11 ion kernel: em0: Tx Queue Status = 1 > Oct 26 23:39:11 ion kernel: em0: TX descriptors avail = 31 > Oct 26 23:39:11 ion kernel: em0: Tx Descriptors avail failure = 0 > Oct 26 23:39:11 ion kernel: em0: RX discarded packets = 0 > Oct 26 23:39:11 ion kernel: em0: RX Next to Check = 960 > Oct 26 23:39:11 ion kernel: em0: RX Next to Refresh = 959 > > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=0 function=0 > dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0xa01f class=0x020000 > dev.em.0.%parent: pci2 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 200 > dev.em.0.tx_int_delay: 200 > dev.em.0.rx_abs_int_delay: 4096 > dev.em.0.tx_abs_int_delay: 4096 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 648 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1477444168 > dev.em.0.rx_control: 100827170 > dev.em.0.fc_high_water: 11264 > dev.em.0.fc_low_water: 9764 > dev.em.0.queue0.txd_head: 975 > dev.em.0.queue0.txd_tail: 944 > dev.em.0.queue0.tx_irq: 2762762 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 960 > dev.em.0.queue0.rxd_tail: 959 > dev.em.0.queue0.rx_irq: 3895860 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 647 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 438789 > dev.em.0.mac_stats.xon_txd: 366 > dev.em.0.mac_stats.xoff_recvd: 438789 > dev.em.0.mac_stats.xoff_txd: 1013 > dev.em.0.mac_stats.total_pkts_recvd: 5465524 > dev.em.0.mac_stats.good_pkts_recvd: 4587299 > dev.em.0.mac_stats.bcast_pkts_recvd: 1102 > dev.em.0.mac_stats.mcast_pkts_recvd: 162 > dev.em.0.mac_stats.rx_frames_64: 325765 > dev.em.0.mac_stats.rx_frames_65_127: 1029229 > dev.em.0.mac_stats.rx_frames_128_255: 118432 > dev.em.0.mac_stats.rx_frames_256_511: 11360 > dev.em.0.mac_stats.rx_frames_512_1023: 100708 > dev.em.0.mac_stats.rx_frames_1024_1522: 3001805 > dev.em.0.mac_stats.good_octets_recvd: 4648591681 > dev.em.0.mac_stats.good_octets_txd: 2203060494 > dev.em.0.mac_stats.total_pkts_txd: 3780652 > dev.em.0.mac_stats.good_pkts_txd: 3779273 > dev.em.0.mac_stats.bcast_pkts_txd: 89 > dev.em.0.mac_stats.mcast_pkts_txd: 534 > dev.em.0.mac_stats.tx_frames_64: 1323163 > dev.em.0.mac_stats.tx_frames_65_127: 850801 > dev.em.0.mac_stats.tx_frames_128_255: 193136 > dev.em.0.mac_stats.tx_frames_256_511: 64088 > dev.em.0.mac_stats.tx_frames_512_1023: 47149 > dev.em.0.mac_stats.tx_frames_1024_1522: 1300936 > dev.em.0.mac_stats.tso_txd: 429804 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 44 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > > ifconfig em0 > em0: flags=8c43 metric 0 mtu 9000 > description: LAN > options=219b > ether 00:1b:21:ab:bf:4a > inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 > inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 > inet6 2002:x:x:1::1 prefixlen 64 > nd6 options=1 > media: Ethernet autoselect (1000baseT ) > status: active > What device is at the other end of link? The input errors netstat reports correspond to the missed_packets reported by the driver. Also the driver has been OACTIVE when the problem occurred. There are also a strange number of xon/xoff frames transmitted and received by the driver. You may try these settings and see if they help: - hw.em.fc_setting=0 (in /boot/loader.conf) - hw.em.rxd="4096" (in /boot/loader.conf) - hw.em.txd="4096" (in /boot/loader.conf) - Fix speed and duplex at both link sides. After doing that, confirm on the freebsd box (with ifconfig) and the other device (with whatever command it provides) that the same speed and duplex is used by both devices. you also have high values for dev.em.0.rx/tx_[abs]_int_delay. If you have set them manually, remove them or replace them with these in loader.conf: hw.em.rx_int_delay=0 hw.em.tx_int_delay=66 hw.em.tx_abs_int_delay=66 hw.em.rx_abs_int_delay=66 these may be set via corresponding sysctls too. From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 09:26:20 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61BF0106564A for ; Thu, 27 Oct 2011 09:26:20 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 26D118FC0A for ; Thu, 27 Oct 2011 09:26:20 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:2c41:2903:6d8f:ca1c]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id EE29D4AC1C; Thu, 27 Oct 2011 13:26:18 +0400 (MSD) Date: Thu, 27 Oct 2011 13:26:14 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <628597845.20111027132614@serebryakov.spb.ru> To: Mike Tancsa In-Reply-To: <4E8F157A.40702@sentex.net> References: <4E8F157A.40702@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 09:26:20 -0000 Hello, Mike. You wrote 7 =EE=EA=F2=FF=E1=F0=FF 2011 =E3., 19:06:34: > This sure sounds like the issue I was seeing with the 7.1.9 drive= r... > However, it has been fixed for me by going to 7.2.3, which is in > RELENG_8. Is it possible you have a couple of issues going on since you > are using lagg as well ? Another problem some folks have reported is > that in the BIOS, if you have an option for ASPM, make sure its disabled. I had a lot of such problems with 7.1.9 on my 82566DM, and I thought, that new driver is Ok, but yesterday it happens again with 7.2.3. No packets could be sent, buffers are overfilled, only full reset helps (after "ifconfig wm0 down && ifconfig em0 up" ping starts to report "Host is down" for any remote host, instead of "No buffer space available")... 8-STABLE, 7.2.3 driver, amd64, 82566DM LOM chip. --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 10:00:43 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4B8E1065678; Thu, 27 Oct 2011 10:00:43 +0000 (UTC) (envelope-from prvs=1281525081=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 29FC08FC08; Thu, 27 Oct 2011 10:00:42 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 27 Oct 2011 10:49:33 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 27 Oct 2011 10:49:33 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50015873990.msg; Thu, 27 Oct 2011 10:49:32 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1281525081=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: , "Mike Tancsa" References: <4E8F157A.40702@sentex.net> <628597845.20111027132614@serebryakov.spb.ru> Date: Thu, 27 Oct 2011 10:49:29 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="windows-1251"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6109 Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIXenabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 10:00:43 -0000 What did netstat -m show? Regards Steve ----- Original Message ----- From: "Lev Serebryakov" To: "Mike Tancsa" Cc: Sent: Thursday, October 27, 2011 10:26 AM Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIXenabled Hello, Mike. You wrote 7 îêòÿáðÿ 2011 ã., 19:06:34: > This sure sounds like the issue I was seeing with the 7.1.9 driver... > However, it has been fixed for me by going to 7.2.3, which is in > RELENG_8. Is it possible you have a couple of issues going on since you > are using lagg as well ? Another problem some folks have reported is > that in the BIOS, if you have an option for ASPM, make sure its disabled. I had a lot of such problems with 7.1.9 on my 82566DM, and I thought, that new driver is Ok, but yesterday it happens again with 7.2.3. No packets could be sent, buffers are overfilled, only full reset helps (after "ifconfig wm0 down && ifconfig em0 up" ping starts to report "Host is down" for any remote host, instead of "No buffer space available")... 8-STABLE, 7.2.3 driver, amd64, 82566DM LOM chip. -- // Black Lion AKA Lev Serebryakov _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 11:59:30 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 135FE1065673 for ; Thu, 27 Oct 2011 11:59:30 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: from f11.mail.ru (f11.mail.ru [217.69.129.69]) by mx1.freebsd.org (Postfix) with ESMTP id 5E9238FC17 for ; Thu, 27 Oct 2011 11:59:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail; h=Message-Id:Content-Transfer-Encoding:Content-Type:Reply-To:In-Reply-To:References:Date:Mime-Version:Subject:Cc:To:From; bh=hj3N2fXNm2SRzK8BXjNvyeX6i29gXWIaNVVTZoFVMP8=; b=EMAe7MSXfFGI33rQhq8ZWPHkbrKXfxyTSD++LGIjnZnGgysRcs8yhaz2jkZU3f+dhus+4SyXqsSBSt/8pd8AFTZnSboE6cvFylY2T8M2VLSV0mBAv0orsj+Pmmr9Dy46; Received: from mail by f11.mail.ru with local id 1RJOcH-0008Ja-00; Thu, 27 Oct 2011 15:59:25 +0400 Received: from [77.45.195.203] by e.mail.ru with HTTP; Thu, 27 Oct 2011 15:59:25 +0400 From: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= To: pyunyh@gmail.com Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [77.45.195.203] Date: Thu, 27 Oct 2011 15:59:25 +0400 References: <20111026164800.GA13234@michelle.cdnetworks.com> In-Reply-To: <20111026164800.GA13234@michelle.cdnetworks.com> X-Priority: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Message-Id: X-Mras: Ok X-Mailman-Approved-At: Thu, 27 Oct 2011 12:13:40 +0000 Cc: freebsd-net@freebsd.org Subject: Re[2]: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 11:59:30 -0000 CgoKMjYg0L7QutGC0Y/QsdGA0Y8gMjAxMSwgMjA6NDkg0L7RgiBZb25nSHllb24gUFlVTiA8cHl1 bnloQGdtYWlsLmNvbT46Cj4gT24gV2VkLCBPY3QgMjYsIDIwMTEgYXQgMDE6NDk6MTRQTSArMDQw MCwgQW5kcmV5IFNtYWdpbiB3cm90ZToKPiA+IEhpICEKPiA+IHZnZTBAcGNpMDoyOjA6MDogICAg ICAgIGNsYXNzPTB4MDIwMDAwIGNhcmQ9MHgwMTEwMTEwNiBjaGlwPTB4MzExOTExMDYgcmV2PTB4 ODIgaGRyPTB4MDAKPiA+ICAgICB2ZW5kb3IgICAgID0gJ1ZJQSBUZWNobm9sb2dpZXMsIEluYy4n Cj4gPiAgICAgZGV2aWNlICAgICA9ICdWVDYxMjAvVlQ2MTIxL1ZUNjEyMiBHaWdhYml0IEV0aGVy bmV0IEFkYXB0ZXInCj4gPiAgICAgY2xhc3MgICAgICA9IG5ldHdvcmsKPiA+ICAgICBzdWJjbGFz cyAgID0gZXRoZXJuZXQKPiA+ICAgICBiYXIgICBbMTBdID0gdHlwZSBJL08gUG9ydCwgcmFuZ2Ug MzIsIGJhc2UgMHg4MDAwLCBzaXplIDI1NiwgZW5hYmxlZAo+ID4gICAgIGJhciAgIFsxNF0gPSB0 eXBlIE1lbW9yeSwgcmFuZ2UgNjQsIGJhc2UgMHhkNDEwMDAwMCwgc2l6ZSAyNTYsIGVuYWJsZWQK PiA+ICAgICBjYXAgMDFbNTBdID0gcG93ZXJzcGVjIDMgIHN1cHBvcnRzIEQwIEQxIEQyIEQzICBj dXJyZW50IEQwCj4gPiAgICAgY2FwIDEwWzkwXSA9IFBDSS1FeHByZXNzIDEgZW5kcG9pbnQgbWF4 IGRhdGEgMTI4KDEyOCkgbGluayB4MSh4MSkKPiA+ICAgICBjYXAgMDVbYzBdID0gTVNJIHN1cHBv cnRzIDEgbWVzc2FnZSwgNjQgYml0LCB2ZWN0b3IgbWFza3MgZW5hYmxlZCB3aXRoIDEgbWVzc2Fn ZQo+ID4gZWNhcCAwMDAxWzEwMF0gPSBBRVIgMSAwIGZhdGFsIDEgbm9uLWZhdGFsIDYgY29ycmVj dGVkCj4gPiBlY2FwIDAwMDNbMTMwXSA9IFNlcmlhbCAxIDAwMDAxMTA2MDAwMDAwMDAKPiA+IHZn ZTFAcGNpMDoxOjA6MDogICAgICAgIGNsYXNzPTB4MDIwMDAwIGNhcmQ9MHgwMTEwMTEwNiBjaGlw PTB4MzExOTExMDYgcmV2PTB4ODIgaGRyPTB4MDAKPiA+ICAgICB2ZW5kb3IgICAgID0gJ1ZJQSBU ZWNobm9sb2dpZXMsIEluYy4nCj4gPiAgICAgZGV2aWNlICAgICA9ICdWVDYxMjAvVlQ2MTIxL1ZU NjEyMiBHaWdhYml0IEV0aGVybmV0IEFkYXB0ZXInCj4gPiAgICAgY2xhc3MgICAgICA9IG5ldHdv cmsKPiA+ICAgICBzdWJjbGFzcyAgID0gZXRoZXJuZXQKPiA+ICAgICBiYXIgICBbMTBdID0gdHlw ZSBJL08gUG9ydCwgcmFuZ2UgMzIsIGJhc2UgMHg3MDAwLCBzaXplIDI1NiwgZW5hYmxlZAo+ID4g ICAgIGJhciAgIFsxNF0gPSB0eXBlIE1lbW9yeSwgcmFuZ2UgNjQsIGJhc2UgMHhkNDAwMDAwMCwg c2l6ZSAyNTYsIGVuYWJsZWQKPiA+ICAgICBjYXAgMDFbNTBdID0gcG93ZXJzcGVjIDMgIHN1cHBv cnRzIEQwIEQxIEQyIEQzICBjdXJyZW50IEQwCj4gPiAgICAgY2FwIDEwWzkwXSA9IFBDSS1FeHBy ZXNzIDEgZW5kcG9pbnQgbWF4IGRhdGEgMTI4KDEyOCkgbGluayB4MSh4MSkKPiA+ICAgICBjYXAg MDVbYzBdID0gTVNJIHN1cHBvcnRzIDEgbWVzc2FnZSwgNjQgYml0LCB2ZWN0b3IgbWFza3MgZW5h YmxlZCB3aXRoIDEgbWVzc2FnZQo+ID4gZWNhcCAwMDAxWzEwMF0gPSBBRVIgMSAxIGZhdGFsIDEg bm9uLWZhdGFsIDYgY29ycmVjdGVkCj4gPiBlY2FwIDAwMDNbMTMwXSA9IFNlcmlhbCAxIDAwMDAx MTA2MDAwMDAwMDAKPiA+Cj4gPiBkbWVzZyBpcyBlbXB0eQoKCiAKPiBIbW0sIGNoZWNrIHdoZXRo ZXIgeW91IGhhdmUgb2xkIGRtZXNnIGZpbGUgaW4gL3Zhci9sb2cgZGlyZWN0b3J5Lgo+IEF0IGxl YXN0LCBJIG5lZWQgb3V0cHV0IG9mICdkZXZpbmZvIC1ydicgdG8ga25vdyB3aGljaCBQSFkgeW91 IGhhdmUuCgogICAgICAgIHBjaWI0IHBucGluZm8gdmVuZG9yPTB4MTBkZSBkZXZpY2U9MHgwMDVk IHN1YnZlbmRvcj0weDAwMDAgc3ViZGV2aWNlPTB4MDAwMCBjbGFzcz0weDA2MDQwMCBhdCBzbG90 PTEzIGZ1bmN0aW9uPTAgaGFuZGxlPVxfU0JfLlBDSTAuWFZSMQogICAgICAgICAgICBJL08gcG9y dHM6CiAgICAgICAgICAgICAgICAweDgwMDAtMHg4ZmZmCiAgICAgICAgICAgIEkvTyBtZW1vcnkg YWRkcmVzc2VzOgogICAgICAgICAgICAgICAgMHhkNDEwMDAwMC0weGQ0MWZmZmZmCiAgICAgICAg ICBwY2kyCiAgICAgICAgICAgIHZnZTAgcG5waW5mbyB2ZW5kb3I9MHgxMTA2IGRldmljZT0weDMx MTkgc3VidmVuZG9yPTB4MTEwNiBzdWJkZXZpY2U9MHgwMTEwIGNsYXNzPTB4MDIwMDAwIGF0IHNs b3Q9MCBmdW5jdGlvbj0wCiAgICAgICAgICAgICAgICBJbnRlcnJ1cHQgcmVxdWVzdCBsaW5lczoK ICAgICAgICAgICAgICAgICAgICAyNTYKICAgICAgICAgICAgICAgIHBjaWI0IEkvTyBwb3J0IHdp bmRvdzoKICAgICAgICAgICAgICAgICAgICAweDgwMDAtMHg4MGZmCiAgICAgICAgICAgICAgICBw Y2liNCBtZW1vcnkgd2luZG93OgogICAgICAgICAgICAgICAgICAgIDB4ZDQxMDAwMDAtMHhkNDEw MDBmZgogICAgICAgICAgICAgIG1paWJ1czEKICAgICAgICAgICAgICAgIGlwMTAwMHBoeTAgcG5w aW5mbyBvdWk9MHg5YzMgbW9kZWw9MHgxOSByZXY9MHgwIGF0IHBoeW5vPTEKICAgICAgICBwY2li NSBwbnBpbmZvIHZlbmRvcj0weDEwZGUgZGV2aWNlPTB4MDA1ZCBzdWJ2ZW5kb3I9MHgwMDAwIHN1 YmRldmljZT0weDAwMDAgY2xhc3M9MHgwNjA0MDAgYXQgc2xvdD0xNCBmdW5jdGlvbj0wIGhhbmRs ZT1cX1NCXy5QQ0kwLlhWUjAKICAgICAgICAgICAgSS9PIHBvcnRzOgogICAgICAgICAgICAgICAg MHg3MDAwLTB4N2ZmZgogICAgICAgICAgICBJL08gbWVtb3J5IGFkZHJlc3NlczoKICAgICAgICAg ICAgICAgIDB4ZDQwMDAwMDAtMHhkNDBmZmZmZgogICAgICAgICAgcGNpMQogICAgICAgICAgICB2 Z2UxIHBucGluZm8gdmVuZG9yPTB4MTEwNiBkZXZpY2U9MHgzMTE5IHN1YnZlbmRvcj0weDExMDYg c3ViZGV2aWNlPTB4MDExMCBjbGFzcz0weDAyMDAwMCBhdCBzbG90PTAgZnVuY3Rpb249MAogICAg ICAgICAgICAgICAgSW50ZXJydXB0IHJlcXVlc3QgbGluZXM6CiAgICAgICAgICAgICAgICAgICAg MjU3CiAgICAgICAgICAgICAgICBwY2liNSBJL08gcG9ydCB3aW5kb3c6CiAgICAgICAgICAgICAg ICAgICAgMHg3MDAwLTB4NzBmZgogICAgICAgICAgICAgICAgcGNpYjUgbWVtb3J5IHdpbmRvdzoK ICAgICAgICAgICAgICAgICAgICAweGQ0MDAwMDAwLTB4ZDQwMDAwZmYKICAgICAgICAgICAgICBt aWlidXMyCiAgICAgICAgICAgICAgICBpcDEwMDBwaHkxIHBucGluZm8gb3VpPTB4OWMzIG1vZGVs PTB4MTkgcmV2PTB4MCBhdCBwaHlubz0xCgo+IEJ5IGNoYW5jZSwgYXJlIHlvdSB1c2luZyBtYW51 YWwgbGluayBjb25maWd1cmF0aW9uIGluc3RlYWQgb2YKPiByZWx5aW5nIG9uIGF1dG8tbmVnb3Rp YXRpb24/CgpJIHRyaWVkICBtYW51YWwgbGluayBhbmQgYXV0by1uZWdvdGlhdGlvbi4gTm93IEkg bm90IHJlbWVtYmVyIHdoaWNoIG1vZGUgCmhhbmcgc3lzdGVtLiBJIHdpbGwgdHJ5IGFuZCB3cml0 ZSByZXN1bHQuCgoKCj4gCj4gPgo+ID4gMjYg0L7QutGC0Y/QsdGA0Y8gMjAxMSwgMDA6MTEg0L7R giBZb25nSHllb24gUFlVTiA8cHl1bnloQGdtYWlsLmNvbT46Cj4gPiA+IE9uIFR1ZSwgT2N0IDI1 LCAyMDExIGF0IDEwOjQ0OjQ4QU0gKzA0MDAsIEFuZHJleSBTbWFnaW4gd3JvdGU6Cj4gPiA+ID4g SGkgQUxMICEgIElmIEkgY29ubmVjdCBnaWdhYml0IHN3aXRjaCB0byBteSBjYXJkIC0gc3lzdGVt IGhhbmcgdW50aWwgSSB1bnBsdWcgcGF0Y2hjb3JkIGZyb20gZGV2aWNlLgo+ID4gPiA+IFdpdGgg MTAwTWJpdCBzd2l0Y2ggY2FyZCB3b3JrIGdvb2QuCj4gPiA+Cj4gPiA+IFNob3cgbWUgdGhlIG91 dHB1dCBvZiBkbWVzZyBhbmQgJ3BjaWNvbmYgLWxjYnYnLgo+ID4gPgo+ID4gPiA+IFN5c3RlbTog IEN1cnJlbnQgLSByMjI2MTYzCj4gPiA+ID4KPiA+ID4KPiA+Cj4g From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 12:34:37 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7FE1106566C; Thu, 27 Oct 2011 12:34:37 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 629778FC1B; Thu, 27 Oct 2011 12:34:37 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:2c41:2903:6d8f:ca1c]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 7B5DF4AC2D; Thu, 27 Oct 2011 16:34:35 +0400 (MSD) Date: Thu, 27 Oct 2011 16:34:30 +0400 From: Lev Serebryakov Organization: FreeBSD Project X-Priority: 3 (Normal) Message-ID: <189569520.20111027163430@serebryakov.spb.ru> To: "Steven Hartland" In-Reply-To: References: <4E8F157A.40702@sentex.net> <628597845.20111027132614@serebryakov.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, lev@FreeBSD.org, Mike Tancsa Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIXenabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 12:34:37 -0000 Hello, Steven. You wrote 27 =EE=EA=F2=FF=E1=F0=FF 2011 =E3., 13:49:29: > What did netstat -m show? Nothing criminal :( 13414/2921/16335 mbufs in use (current/cache/total) 4997/533/5530/204800 mbuf clusters in use (current/cache/total/max) 4626/329 mbuf+clusters out of packet secondary zone in use (current/cache) 78/2976/3054/192000 4k (page size) jumbo clusters in use (current/cache/tot= al/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 13659K/13700K/27359K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines --=20 // Black Lion AKA Lev Serebryakov From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 13:18:38 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C3C771065680 for ; Thu, 27 Oct 2011 13:18:38 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id 219888FC18 for ; Thu, 27 Oct 2011 13:18:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id 2F2A91C0874; Thu, 27 Oct 2011 17:18:35 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.51]) by mailgate.pik.ru (Postfix) with ESMTP id 2CF021C0873; Thu, 27 Oct 2011 17:18:35 +0400 (MSD) Received: from EX21PIK.PICompany.ru ([192.168.156.131]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Oct 2011 17:17:53 +0400 Received: from [192.168.148.9] (192.168.148.9) by EX21PIK.PICompany.ru (192.168.156.131) with Microsoft SMTP Server id 14.1.218.12; Thu, 27 Oct 2011 17:17:52 +0400 Message-ID: <4EA959EE.2070806@hotplug.ru> Date: Thu, 27 Oct 2011 17:17:34 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: Hooman Fazaeli References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EA91836.2040508@gmail.com> In-Reply-To: <4EA91836.2040508@gmail.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 27 Oct 2011 13:17:53.0478 (UTC) FILETIME=[D5890660:01CC94AA] Cc: freebsd-net@freebsd.org Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 13:18:38 -0000 >> >> Hi Hooman >> >> Here is what I've got when the script triggered just in time when the >> interface was locked >> >> >> 11.10.26-23:39:10 ... interface em0 is down... >> >> FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 >> 20:20:25 MSD 2011 root@epia.home >> .lan:/usr/obj/usr/src/sys/ION6debug amd64 >> 11:39PM up 1:12, 2 users, load averages: 0.26, 0.48, 0.58 >> >> >> == vmstat -i == >> interrupt total rate >> irq22: nfe0 16644480 3865 >> cpu0: timer 8610122 1999 >> irq256: ahci0 606705 140 >> irq257: em0:rx 0 3896622 904 >> irq258: em0:tx 0 2762957 641 >> irq259: em0:link 620 0 >> cpu3: timer 8609499 1999 >> cpu1: timer 8609499 1999 >> cpu2: timer 8609499 1999 >> Total 58350003 13550 >> >> == netstat -ind == >> Name Mtu Network Address Ipkts Ierrs Idrop >> Opkts Oerrs Coll Drop >> usbus 0 0 0 >> 0 0 0 0 0 >> usbus 0 0 0 >> 0 0 0 0 0 >> nfe0 1500 00:25:22:21:86:89 7157140 0 0 >> 12266747 0 0 0 >> nfe0 1500 fe80::225:22f fe80::225:22ff:fe 0 - >> - 85 - - - >> nfe0 1500 10.16.128.0/1 10.16.189.71 0 - - >> 48135 - - - >> em0 9000 00:1b:21:ab:bf:4a 5465087 623 0 >> 2862028 0 0 113 >> em0 9000 192.168.168.0 192.168.168.1 764085 - - >> 1005078 - - - >> em0 9000 fe80::21b:21f fe80::21b:21ff:fe 45 - - >> 252 - - - >> em0 9000 2002:d58d:871 2002:d58d:8715:1: 73 - >> - 38 - - - >> wifi 1500 00:1b:21:ab:bf:4a 347 0 0 >> 350 0 0 0 >> wifi 1500 192.168.168.6 192.168.168.65 0 - >> - 0 - - - >> wifi 1500 fe80::225:x fe80::225:x:x 0 - - >> 349 - - - >> wifi 1500 2002:x:x 2002:x:x:2: 0 - - 0 >> - - - >> wifio 1500 00:1b:21:ab:bf:4a 59559 0 0 >> 114639 0 0 0 >> wifio 1500 192.168.168.8 192.168.168.81 0 - - >> 160 - - - >> wifio 1500 fe80::225:x fe80::225:x:x 0 - - >> 0 - - - >> stf0 1280 5725 0 0 >> 6125 420 0 0 >> stf0 1280 2002:x:x 2002:x:x::1 1878 - - 1121 >> - - - >> ng0* 1500 0 0 >> 0 0 0 0 0 >> ng1* 1500 0 0 >> 0 0 0 0 0 >> ng2 1492 7143733 0 0 >> 12234436 0 0 0 >> ng2 1492 213.141.x.x 213.141.x.x 4735932 - - >> 8480089 - - - >> ng2 1492 fe80::x:x fe80::x:x:x 0 - - 1 >> - - - >> tun0 1455 350 0 0 >> 172 0 0 0 >> tun0 1455 fe80::225:x fe80::225:x:x 0 - - >> 2 - - - >> tun0 1455 192.168.169.1 192.168.169.1 117 - - >> 167 - - - >> >> Oct 26 23:39:11 ion kernel: em0: hw tdh = 975, hw tdt = 944 >> Oct 26 23:39:11 ion kernel: em0: hw rdh = 960, hw rdt = 959 >> Oct 26 23:39:11 ion kernel: em0: Tx Queue Status = 1 >> Oct 26 23:39:11 ion kernel: em0: TX descriptors avail = 31 >> Oct 26 23:39:11 ion kernel: em0: Tx Descriptors avail failure = 0 >> Oct 26 23:39:11 ion kernel: em0: RX discarded packets = 0 >> Oct 26 23:39:11 ion kernel: em0: RX Next to Check = 960 >> Oct 26 23:39:11 ion kernel: em0: RX Next to Refresh = 959 >> >> net.inet.ip.intr_queue_maxlen: 4096 >> net.inet.ip.intr_queue_drops: 0 >> dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 >> dev.em.0.%driver: em >> dev.em.0.%location: slot=0 function=0 >> dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 >> subdevice=0xa01f class=0x020000 >> dev.em.0.%parent: pci2 >> dev.em.0.nvm: -1 >> dev.em.0.debug: -1 >> dev.em.0.rx_int_delay: 200 >> dev.em.0.tx_int_delay: 200 >> dev.em.0.rx_abs_int_delay: 4096 >> dev.em.0.tx_abs_int_delay: 4096 >> dev.em.0.rx_processing_limit: 100 >> dev.em.0.flow_control: 3 >> dev.em.0.eee_control: 0 >> dev.em.0.link_irq: 648 >> dev.em.0.mbuf_alloc_fail: 0 >> dev.em.0.cluster_alloc_fail: 0 >> dev.em.0.dropped: 0 >> dev.em.0.tx_dma_fail: 0 >> dev.em.0.rx_overruns: 0 >> dev.em.0.watchdog_timeouts: 0 >> dev.em.0.device_control: 1477444168 >> dev.em.0.rx_control: 100827170 >> dev.em.0.fc_high_water: 11264 >> dev.em.0.fc_low_water: 9764 >> dev.em.0.queue0.txd_head: 975 >> dev.em.0.queue0.txd_tail: 944 >> dev.em.0.queue0.tx_irq: 2762762 >> dev.em.0.queue0.no_desc_avail: 0 >> dev.em.0.queue0.rxd_head: 960 >> dev.em.0.queue0.rxd_tail: 959 >> dev.em.0.queue0.rx_irq: 3895860 >> dev.em.0.mac_stats.excess_coll: 0 >> dev.em.0.mac_stats.single_coll: 0 >> dev.em.0.mac_stats.multiple_coll: 0 >> dev.em.0.mac_stats.late_coll: 0 >> dev.em.0.mac_stats.collision_count: 0 >> dev.em.0.mac_stats.symbol_errors: 0 >> dev.em.0.mac_stats.sequence_errors: 0 >> dev.em.0.mac_stats.defer_count: 0 >> dev.em.0.mac_stats.missed_packets: 647 >> dev.em.0.mac_stats.recv_no_buff: 0 >> dev.em.0.mac_stats.recv_undersize: 0 >> dev.em.0.mac_stats.recv_fragmented: 0 >> dev.em.0.mac_stats.recv_fragmented: 0 >> dev.em.0.mac_stats.recv_oversize: 0 >> dev.em.0.mac_stats.recv_jabber: 0 >> dev.em.0.mac_stats.recv_errs: 0 >> dev.em.0.mac_stats.crc_errs: 0 >> dev.em.0.mac_stats.alignment_errs: 0 >> dev.em.0.mac_stats.coll_ext_errs: 0 >> dev.em.0.mac_stats.xon_recvd: 438789 >> dev.em.0.mac_stats.xon_txd: 366 >> dev.em.0.mac_stats.xoff_recvd: 438789 >> dev.em.0.mac_stats.xoff_txd: 1013 >> dev.em.0.mac_stats.total_pkts_recvd: 5465524 >> dev.em.0.mac_stats.good_pkts_recvd: 4587299 >> dev.em.0.mac_stats.bcast_pkts_recvd: 1102 >> dev.em.0.mac_stats.mcast_pkts_recvd: 162 >> dev.em.0.mac_stats.rx_frames_64: 325765 >> dev.em.0.mac_stats.rx_frames_65_127: 1029229 >> dev.em.0.mac_stats.rx_frames_128_255: 118432 >> dev.em.0.mac_stats.rx_frames_256_511: 11360 >> dev.em.0.mac_stats.rx_frames_512_1023: 100708 >> dev.em.0.mac_stats.rx_frames_1024_1522: 3001805 >> dev.em.0.mac_stats.good_octets_recvd: 4648591681 >> dev.em.0.mac_stats.good_octets_txd: 2203060494 >> dev.em.0.mac_stats.total_pkts_txd: 3780652 >> dev.em.0.mac_stats.good_pkts_txd: 3779273 >> dev.em.0.mac_stats.bcast_pkts_txd: 89 >> dev.em.0.mac_stats.mcast_pkts_txd: 534 >> dev.em.0.mac_stats.tx_frames_64: 1323163 >> dev.em.0.mac_stats.tx_frames_65_127: 850801 >> dev.em.0.mac_stats.tx_frames_128_255: 193136 >> dev.em.0.mac_stats.tx_frames_256_511: 64088 >> dev.em.0.mac_stats.tx_frames_512_1023: 47149 >> dev.em.0.mac_stats.tx_frames_1024_1522: 1300936 >> dev.em.0.mac_stats.tso_txd: 429804 >> dev.em.0.mac_stats.tso_ctx_fail: 0 >> dev.em.0.interrupts.asserts: 44 >> dev.em.0.interrupts.rx_pkt_timer: 0 >> dev.em.0.interrupts.rx_abs_timer: 0 >> dev.em.0.interrupts.tx_pkt_timer: 0 >> dev.em.0.interrupts.tx_abs_timer: 0 >> dev.em.0.interrupts.tx_queue_empty: 0 >> dev.em.0.interrupts.tx_queue_min_thresh: 0 >> dev.em.0.interrupts.rx_desc_min_thresh: 0 >> dev.em.0.interrupts.rx_overrun: 0 >> >> ifconfig em0 >> em0: flags=8c43 >> metric 0 mtu 9000 >> description: LAN >> >> options=219b >> ether 00:1b:21:ab:bf:4a >> inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.168.63 >> inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x4 >> inet6 2002:x:x:1::1 prefixlen 64 >> nd6 options=1 >> media: Ethernet autoselect (1000baseT ) >> status: active >> > What device is at the other end of link? > Just a simple soho router/switch/wifi AP. I use it as a gigabit switch with dot1q support and wifi AP. It's management capabilities and features are very limited. Here is what I've got form the switch info, it's rather obscure, for ex. I have no idea what does Dot3StatsFCSErrors means. But at least I had no problems with this switch and another nic which is nvidia LOM running nfe driver root@tplink:~# swconfig dev rtl8366rb port 1 show Port 1: link: port:1 link:up speed:1000baseT full-duplex tx-pause rx-pause mib: Port 1 MIB counters IfInOctets : 7538704547344 EtherStatsOctets : 7538876056305 EtherStatsUnderSizePkts : 0 EtherFragments : 3 EtherStatsPkts64Octets : 662581957 EtherStatsPkts65to127Octets : 2535622995 EtherStatsPkts128to255Octets : 96290375 EtherStatsPkts256to511Octets : 42103721 EtherStatsPkts512to1023Octets : 83464044 EtherStatsPkts1024to1518Octets : 384590702 EtherOversizeStats : 22227404 EtherStatsJabbers : 38641 IfInUcastPkts : 3757127256 EtherStatsMulticastPkts : 69494916 EtherStatsBroadcastPkts : 109605 EtherStatsDropEvents : 69198572 Dot3StatsFCSErrors : 149421 Dot3StatsSymbolErrors : 90698 Dot3InPauseFrames : 68996650 Dot3ControlInUnknownOpcodes : 0 IfOutOctets : 8216482296676 Dot3StatsSingleCollisionFrames : 0 Dot3StatMultipleCollisionFrames : 0 Dot3sDeferredTransmissions : 1474621219 Dot3StatsLateCollisions : 0 EtherStatsCollisions : 0 Dot3StatsExcessiveCollisions : 0 Dot3OutPauseFrames : 1512177530 Dot1dBasePortDelayExceededDiscards : 0 Dot1dTpPortInDiscards : 9143 IfOutUcastPkts : 3920068852 IfOutMulticastPkts : 9715117 IfOutBroadcastPkts : 1006622 > The input errors netstat reports correspond to the missed_packets > reported by the driver. > Also the driver has been OACTIVE when the problem occurred. > There are also a strange number of xon/xoff frames transmitted and > received by the driver. > > You may try these settings and see if they help: > > - hw.em.fc_setting=0 (in /boot/loader.conf) > - hw.em.rxd="4096" (in /boot/loader.conf) > - hw.em.txd="4096" (in /boot/loader.conf) > - Fix speed and duplex at both link sides. After doing that, confirm > on the freebsd > box (with ifconfig) and the other device (with whatever command it > provides) that > the same speed and duplex is used by both devices. > Thanks for taking a look at this and for the tips, I'll do the changes and will see if this helps. > you also have high values for dev.em.0.rx/tx_[abs]_int_delay. If you > have set them manually, remove them or replace them with these in > loader.conf: > > hw.em.rx_int_delay=0 > hw.em.tx_int_delay=66 > hw.em.tx_abs_int_delay=66 > hw.em.rx_abs_int_delay=66 > > Yes, indeed it was my blind tuning trying to change anything here and there because of those locks. Will remove this to the default. From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 14:50:57 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 13E541065676 for ; Thu, 27 Oct 2011 14:50:57 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8D2398FC13 for ; Thu, 27 Oct 2011 14:50:56 +0000 (UTC) Received: by wyi40 with SMTP id 40so3867214wyi.13 for ; Thu, 27 Oct 2011 07:50:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=h8pfh8UeoQI0wNMmRUqtzk+kgL8879z3M24AjHD4Opo=; b=G18tWyNawh+lu9H3Z+RJXBEH4S0QH+ACEVtc2/wlmjRnEy08FtNdWmkJOVjLrWMrae Kq/+KDLsb44KBTbdujZF8MidguE5aN1pHycjw77s4Q0oN923/TGFN9atInwFDZUdP9TN rKHnlTMpFqhpeFM8THPYYGozedMlYizlAClDY= MIME-Version: 1.0 Received: by 10.227.203.132 with SMTP id fi4mr17057702wbb.6.1319727055142; Thu, 27 Oct 2011 07:50:55 -0700 (PDT) Received: by 10.180.105.162 with HTTP; Thu, 27 Oct 2011 07:50:54 -0700 (PDT) In-Reply-To: <4EA8FA40.7010504@hotplug.ru> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> Date: Thu, 27 Oct 2011 10:50:54 -0400 Message-ID: From: Arnaud Lacombe To: Emil Muratov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, Jason Wolfe , Hooman Fazaeli Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 14:50:57 -0000 Hi, On Thu, Oct 27, 2011 at 2:29 AM, Emil Muratov wrote: > > >> Hi, >> >> Can yan you pls post the output of these command _when_ the problem >> happens? >> >> uname -a >> sysctl dev.em >> netstat -ind >> ifconfig >> > > Hi Hooman > > Here is what I've got when the script triggered just in time when the > interface was locked > > > 11.10.26-23:39:10 ... interface em0 is down... > > FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 20:20= :25 > Please upgrade to 8-STABLE, similar issues have been fixed there. Thanks, - Arnaud > MSD 2011 =A0 =A0 root@epia.home > .lan:/usr/obj/usr/src/sys/ION6debug =A0amd64 > 11:39PM =A0up =A01:12, 2 users, load averages: 0.26, 0.48, 0.58 > > > =A0=3D=3D vmstat -i =3D=3D > interrupt =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0total =A0 = =A0 =A0 rate > irq22: nfe0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 16644480 =A0 =A0 =A0 = 3865 > cpu0: timer =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08610122 =A0 =A0 = =A0 1999 > irq256: ahci0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 606705 =A0 =A0 =A0 = =A0140 > irq257: em0:rx 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 3896622 =A0 =A0 =A0 =A09= 04 > irq258: em0:tx 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 2762957 =A0 =A0 =A0 =A06= 41 > irq259: em0:link =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 620 =A0 =A0 =A0 = =A0 =A00 > cpu3: timer =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08609499 =A0 =A0 = =A0 1999 > cpu1: timer =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08609499 =A0 =A0 = =A0 1999 > cpu2: timer =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A08609499 =A0 =A0 = =A0 1999 > Total =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 58350003 =A0 = =A0 =A013550 > > =A0=3D=3D netstat -ind =3D=3D > Name =A0 =A0Mtu Network =A0 =A0 =A0 Address =A0 =A0 =A0 =A0 =A0 =A0 =A0Ip= kts Ierrs Idrop =A0 =A0Opkts > Oerrs =A0Coll Drop > usbus =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 =A0 =A00 > =A0 0 =A0 =A0 0 =A0 =A00 > usbus =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 =A0 =A00 > =A0 0 =A0 =A0 0 =A0 =A00 > nfe0 =A0 1500 =A0 =A0 =A000:25:22:21:86:89 =A07157140 =A0 =A0 0 = =A0 =A0 0 12266747 > =A0 0 =A0 =A0 0 =A0 =A00 > nfe0 =A0 1500 fe80::225:22f fe80::225:22ff:fe =A0 =A0 =A0 =A00 =A0 =A0 - = =A0 =A0 - =A0 =A0 =A0 85 > =A0 - =A0 =A0 - =A0 =A0- > nfe0 =A0 1500 10.16.128.0/1 10.16.189.71 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 - =A0 =A0 - =A0 =A048135 > =A0 - =A0 =A0 - =A0 =A0- > em0 =A0 =A09000 =A0 =A0 =A000:1b:21:ab:bf:4a =A05465087 =A0 623 = =A0 =A0 0 =A02862028 > =A0 0 =A0 =A0 0 =A0113 > em0 =A0 =A09000 192.168.168.0 192.168.168.1 =A0 =A0 =A0 764085 =A0 =A0 - = =A0 =A0 - =A01005078 > =A0 - =A0 =A0 - =A0 =A0- > em0 =A0 =A09000 fe80::21b:21f fe80::21b:21ff:fe =A0 =A0 =A0 45 =A0 =A0 - = =A0 =A0 - =A0 =A0 =A0252 > =A0 - =A0 =A0 - =A0 =A0- > em0 =A0 =A09000 2002:d58d:871 2002:d58d:8715:1: =A0 =A0 =A0 73 =A0 =A0 - = =A0 =A0 - =A0 =A0 =A0 38 > =A0 - =A0 =A0 - =A0 =A0- > wifi =A0 1500 =A0 =A0 =A000:1b:21:ab:bf:4a =A0 =A0 =A0347 =A0 = =A0 0 =A0 =A0 0 =A0 =A0 =A0350 > =A0 0 =A0 =A0 0 =A0 =A00 > wifi =A0 1500 192.168.168.6 192.168.168.65 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = - =A0 =A0 - =A0 =A0 =A0 =A00 > =A0 - =A0 =A0 - =A0 =A0- > wifi =A0 1500 fe80::225:x fe80::225:x:x =A0 =A0 =A0 =A00 =A0 =A0 - =A0 = =A0 - =A0 =A0 =A0349 =A0 =A0 - > =A0 - =A0 =A0- > wifi =A0 1500 2002:x:x 2002:x:x:2: =A0 =A0 =A0 =A00 =A0 =A0 - =A0 =A0 - = =A0 =A0 =A0 =A00 =A0 =A0 - =A0 =A0 - > =A0- > wifio =A01500 =A0 =A0 =A000:1b:21:ab:bf:4a =A0 =A059559 =A0 =A0 = 0 =A0 =A0 0 =A0 114639 > =A0 0 =A0 =A0 0 =A0 =A00 > wifio =A01500 192.168.168.8 192.168.168.81 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = - =A0 =A0 - =A0 =A0 =A0160 > =A0 - =A0 =A0 - =A0 =A0- > wifio =A01500 fe80::225:x fe80::225:x:x =A0 =A0 =A0 =A00 =A0 =A0 - =A0 = =A0 - =A0 =A0 =A0 =A00 =A0 =A0 - > =A0 - =A0 =A0- > stf0 =A0 1280 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A05725 =A0 =A0 0 =A0 =A0 0 =A0 =A0 6125 > 420 =A0 =A0 0 =A0 =A00 > stf0 =A0 1280 2002:x:x 2002:x:x::1 =A0 =A0 1878 =A0 =A0 - =A0 =A0 - =A0 = =A0 1121 =A0 =A0 - =A0 =A0 - > =A0- > ng0* =A0 1500 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A00 =A0 =A0 0 =A0 =A0 0 =A0 =A0 =A0 =A00 > =A0 0 =A0 =A0 0 =A0 =A00 > ng1* =A0 1500 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A00 =A0 =A0 0 =A0 =A0 0 =A0 =A0 =A0 =A00 > =A0 0 =A0 =A0 0 =A0 =A00 > ng2 =A0 =A01492 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= 7143733 =A0 =A0 0 =A0 =A0 0 12234436 > =A0 0 =A0 =A0 0 =A0 =A00 > ng2 =A0 =A01492 213.141.x.x 213.141.x.x =A0 =A0 4735932 =A0 =A0 - =A0 =A0= - =A08480089 =A0 =A0 - > =A0 - =A0 =A0- > ng2 =A0 =A01492 fe80::x:x fe80::x:x:x =A0 =A0 =A0 =A00 =A0 =A0 - =A0 =A0 = - =A0 =A0 =A0 =A01 =A0 =A0 - =A0 =A0 - > =A0 =A0- > tun0 =A0 1455 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0350 =A0 =A0 0 =A0 =A0 0 =A0 =A0 =A0172 > =A0 0 =A0 =A0 0 =A0 =A00 > tun0 =A0 1455 fe80::225:x fe80::225:x:x =A0 =A0 =A0 =A00 =A0 =A0 - =A0 = =A0 - =A0 =A0 =A0 =A02 =A0 =A0 - > =A0 - =A0 =A0- > tun0 =A0 1455 192.168.169.1 192.168.169.1 =A0 =A0 =A0 =A0 =A0117 =A0 =A0 = - =A0 =A0 - =A0 =A0 =A0167 > =A0 - =A0 =A0 - =A0 =A0- > > Oct 26 23:39:11 ion kernel: em0: hw tdh =3D 975, hw tdt =3D 944 > Oct 26 23:39:11 ion kernel: em0: hw rdh =3D 960, hw rdt =3D 959 > Oct 26 23:39:11 ion kernel: em0: Tx Queue Status =3D 1 > Oct 26 23:39:11 ion kernel: em0: TX descriptors avail =3D 31 > Oct 26 23:39:11 ion kernel: em0: Tx Descriptors avail failure =3D 0 > Oct 26 23:39:11 ion kernel: em0: RX discarded packets =3D 0 > Oct 26 23:39:11 ion kernel: em0: RX Next to Check =3D 960 > Oct 26 23:39:11 ion kernel: em0: RX Next to Refresh =3D 959 > > net.inet.ip.intr_queue_maxlen: 4096 > net.inet.ip.intr_queue_drops: 0 > dev.em.0.%desc: Intel(R) PRO/1000 Network Connection 7.2.3 > dev.em.0.%driver: em > dev.em.0.%location: slot=3D0 function=3D0 > dev.em.0.%pnpinfo: vendor=3D0x8086 device=3D0x10d3 subvendor=3D0x8086 > subdevice=3D0xa01f class=3D0x020000 > dev.em.0.%parent: pci2 > dev.em.0.nvm: -1 > dev.em.0.debug: -1 > dev.em.0.rx_int_delay: 200 > dev.em.0.tx_int_delay: 200 > dev.em.0.rx_abs_int_delay: 4096 > dev.em.0.tx_abs_int_delay: 4096 > dev.em.0.rx_processing_limit: 100 > dev.em.0.flow_control: 3 > dev.em.0.eee_control: 0 > dev.em.0.link_irq: 648 > dev.em.0.mbuf_alloc_fail: 0 > dev.em.0.cluster_alloc_fail: 0 > dev.em.0.dropped: 0 > dev.em.0.tx_dma_fail: 0 > dev.em.0.rx_overruns: 0 > dev.em.0.watchdog_timeouts: 0 > dev.em.0.device_control: 1477444168 > dev.em.0.rx_control: 100827170 > dev.em.0.fc_high_water: 11264 > dev.em.0.fc_low_water: 9764 > dev.em.0.queue0.txd_head: 975 > dev.em.0.queue0.txd_tail: 944 > dev.em.0.queue0.tx_irq: 2762762 > dev.em.0.queue0.no_desc_avail: 0 > dev.em.0.queue0.rxd_head: 960 > dev.em.0.queue0.rxd_tail: 959 > dev.em.0.queue0.rx_irq: 3895860 > dev.em.0.mac_stats.excess_coll: 0 > dev.em.0.mac_stats.single_coll: 0 > dev.em.0.mac_stats.multiple_coll: 0 > dev.em.0.mac_stats.late_coll: 0 > dev.em.0.mac_stats.collision_count: 0 > dev.em.0.mac_stats.symbol_errors: 0 > dev.em.0.mac_stats.sequence_errors: 0 > dev.em.0.mac_stats.defer_count: 0 > dev.em.0.mac_stats.missed_packets: 647 > dev.em.0.mac_stats.recv_no_buff: 0 > dev.em.0.mac_stats.recv_undersize: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_fragmented: 0 > dev.em.0.mac_stats.recv_oversize: 0 > dev.em.0.mac_stats.recv_jabber: 0 > dev.em.0.mac_stats.recv_errs: 0 > dev.em.0.mac_stats.crc_errs: 0 > dev.em.0.mac_stats.alignment_errs: 0 > dev.em.0.mac_stats.coll_ext_errs: 0 > dev.em.0.mac_stats.xon_recvd: 438789 > dev.em.0.mac_stats.xon_txd: 366 > dev.em.0.mac_stats.xoff_recvd: 438789 > dev.em.0.mac_stats.xoff_txd: 1013 > dev.em.0.mac_stats.total_pkts_recvd: 5465524 > dev.em.0.mac_stats.good_pkts_recvd: 4587299 > dev.em.0.mac_stats.bcast_pkts_recvd: 1102 > dev.em.0.mac_stats.mcast_pkts_recvd: 162 > dev.em.0.mac_stats.rx_frames_64: 325765 > dev.em.0.mac_stats.rx_frames_65_127: 1029229 > dev.em.0.mac_stats.rx_frames_128_255: 118432 > dev.em.0.mac_stats.rx_frames_256_511: 11360 > dev.em.0.mac_stats.rx_frames_512_1023: 100708 > dev.em.0.mac_stats.rx_frames_1024_1522: 3001805 > dev.em.0.mac_stats.good_octets_recvd: 4648591681 > dev.em.0.mac_stats.good_octets_txd: 2203060494 > dev.em.0.mac_stats.total_pkts_txd: 3780652 > dev.em.0.mac_stats.good_pkts_txd: 3779273 > dev.em.0.mac_stats.bcast_pkts_txd: 89 > dev.em.0.mac_stats.mcast_pkts_txd: 534 > dev.em.0.mac_stats.tx_frames_64: 1323163 > dev.em.0.mac_stats.tx_frames_65_127: 850801 > dev.em.0.mac_stats.tx_frames_128_255: 193136 > dev.em.0.mac_stats.tx_frames_256_511: 64088 > dev.em.0.mac_stats.tx_frames_512_1023: 47149 > dev.em.0.mac_stats.tx_frames_1024_1522: 1300936 > dev.em.0.mac_stats.tso_txd: 429804 > dev.em.0.mac_stats.tso_ctx_fail: 0 > dev.em.0.interrupts.asserts: 44 > dev.em.0.interrupts.rx_pkt_timer: 0 > dev.em.0.interrupts.rx_abs_timer: 0 > dev.em.0.interrupts.tx_pkt_timer: 0 > dev.em.0.interrupts.tx_abs_timer: 0 > dev.em.0.interrupts.tx_queue_empty: 0 > dev.em.0.interrupts.tx_queue_min_thresh: 0 > dev.em.0.interrupts.rx_desc_min_thresh: 0 > dev.em.0.interrupts.rx_overrun: 0 > > ifconfig em0 > em0: flags=3D8c43 metric = 0 mtu > 9000 > =A0 =A0 =A0 =A0description: LAN > > =A0options=3D219b > =A0 =A0 =A0 =A0ether 00:1b:21:ab:bf:4a > =A0 =A0 =A0 =A0inet 192.168.168.1 netmask 0xffffffc0 broadcast 192.168.16= 8.63 > =A0 =A0 =A0 =A0inet6 fe80::21b:21ff:feab:bf4a%em0 prefixlen 64 scopeid 0x= 4 > =A0 =A0 =A0 =A0inet6 2002:x:x:1::1 prefixlen 64 > =A0 =A0 =A0 =A0nd6 options=3D1 > =A0 =A0 =A0 =A0media: Ethernet autoselect (1000baseT ) > =A0 =A0 =A0 =A0status: active > > > >> >>> I've got almost the same problem with intel 82574L based nic. My platfo= rm >>> is nvidia ion running Atom 1.6 and nic is an external PCI-express adapt= er. >>> Unlike Jason's case mine is always stuck in receiving traffic, it's Ier= rs >>> increasing while Ipkts not. Thanks to Jason's script I can see those lo= cks >>> and interface flapping every several hours. My system is not a heavy lo= aded >>> server but just a home nas/router, usually routing at 100 mbps or less. >>> Nither disabling MSIX nor tuning txd rxd doesn't help me. >>> > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Oct 27 22:45:24 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F07091065674 for ; Thu, 27 Oct 2011 22:45:23 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by mx1.freebsd.org (Postfix) with ESMTP id 5A1748FC0A for ; Thu, 27 Oct 2011 22:45:23 +0000 (UTC) Received: from [192.168.1.200] (p508FD754.dip.t-dialin.net [80.143.215.84]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id 378001C0C0BD9; Fri, 28 Oct 2011 00:45:21 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1251.1) Content-Type: text/plain; charset=utf-8 From: =?iso-8859-1?Q?Michael_T=FCxen?= In-Reply-To: <1319422844876-4931035.post@n5.nabble.com> Date: Fri, 28 Oct 2011 00:45:19 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1319362410261-4929128.post@n5.nabble.com> <2E60D876-9B2C-4039-9366-40451D5914E2@lurchi.franken.de> <1319422844876-4931035.post@n5.nabble.com> To: jyl_2006 X-Mailer: Apple Mail (2.1251.1) Cc: freebsd-net@freebsd.org Subject: Re: SCTP : problems in sending ASCONF chunks X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 22:45:24 -0000 OK, please try two fixes: http://svn.freebsd.org/changeset/base/226868 This fixes a problem which resulted in the ASCONF chunks not being sent. http://svn.freebsd.org/changeset/base/226869 This fixes a problem which resulted in the path confirmation chunks being sent to the wrong destination and therefore not confirming the path. Please let me know if you still have any issues. Best regards Michael On Oct 24, 2011, at 4:20 AM, jyl_2006 wrote: > Hi,T=C3=BCxen >=20 > I will provide more detail: > The Topology is: > --------1(192.168.1.20) > computer A ---------------------------1 computer B(192.168.1.80) > --------2(192.168.1.50) > means computer has two wireless cards , I name them with A_1 and A_2, > computer B has one wireless cards, and its name is B_1. >=20 > Now, A_1 init a association with B_1. Here, I provide message getting = from > wireshark , > INIT=EF=BC=9A > Internet Protocol, Src: 192.168.1.20 (192.168.1.20), Dst: 192.168.1.80 > (192.168.1.80) > Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, > FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) > INIT=E3=80=80ACK: > Internet Protocol, Src: 192.168.1.80 (192.168.1.80), Dst: 192.168.1.20 > (192.168.1.20) > Supported Extensions parameter (Supported types: ASCONF, ASCONF_ACK, > FORWARD_TSN, PKTDROP, STREAM_RESET, AUTH) >=20 > Debug message: > SCTP_SACK process cum_ack:45452434 num_seg:0 a_rwnd:1864135 > Check for chunk output prw:1864135 tqe:1 tf=3D0 > Send called addr:0xc591c980 send length 2 > Calling ipv4 output routine from low level src addr:c0a80114 > Destination is c0a80150 > RTP route is 0xc5caaaf8 through > IP output returns 0 > m-c-o put out 1 > Ok, we have put out 1 chunks > USR Send complete qo:0 prw:1863859 unsent:18 tf:18 cooq:1 toqs:18 = err:0 > Ok laddr->ifa:0xc5baab00 is possible, asconf_queue_mgmt: inserted = asconf > ADD_IP_ADDRESS: IPv4 address: 192.168.1.50:0 > m-c-o put out 0 > Ok, we have put out 0 chunks > sctp_input() length:28 iphlen:20 > sctp_input(): Packet of length 48 received on wlan0 with csum_flags = 0x0. > Ok, Common input processing called, m:0xc72dab00 iphlen:20 offset:32 > length:48 stcb:0xcc2335dc > stcb:0xcc2335dc state:8 > sctp_process_control: iphlen=3D20, offset=3D32, length=3D48 = stcb:0xcc2335dc > sctp_process_control: processing a chunk type=3D3, len=3D16 >=20 > Actually,I also test following Topology : > --------1(192.168.1.20) =20 > --------------1(192.168.1.80) > computer A --------------------------- computer B > --------2(192.168.2.20) =20 > --------------2(192.168.2.80) > means computer has two wireless cards , I name them with A_1 and A_2, > computer B has two wireless cards, and its name is B_1 and B_2. >=20 > The result from wireshark and debug message have same results. >=20 >=20 > -- > View this message in context: = http://freebsd.1045724.n5.nabble.com/SCTP-problems-in-sending-ASCONF-chunk= s-tp4929128p4931035.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 00:29:37 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E2C061065672; Fri, 28 Oct 2011 00:29:37 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 55BCB8FC08; Fri, 28 Oct 2011 00:29:37 +0000 (UTC) Received: from lstewart1.loshell.room52.net (ppp59-167-184-191.static.internode.on.net [59.167.184.191]) by lauren.room52.net (Postfix) with ESMTPSA id 147097E820; Fri, 28 Oct 2011 11:29:35 +1100 (EST) Message-ID: <4EA9F76E.9010008@freebsd.org> Date: Fri, 28 Oct 2011 11:29:34 +1100 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111016 Thunderbird/7.0.1 MIME-Version: 1.0 To: John Baldwin References: <20111022084931.GD1697@garage.freebsd.pl> <201110240814.22368.jhb@freebsd.org> <20111026075431.GB1672@garage.freebsd.pl> <201110260753.37264.jhb@freebsd.org> In-Reply-To: <201110260753.37264.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: Kostik Belousov , Andre Oppermann , freebsd-current@freebsd.org, Pawel Jakub Dawidek , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 00:29:38 -0000 On 10/26/11 22:53, John Baldwin wrote: > On Wednesday, October 26, 2011 3:54:31 am Pawel Jakub Dawidek wrote: >> On Mon, Oct 24, 2011 at 08:14:22AM -0400, John Baldwin wrote: >>> On Sunday, October 23, 2011 11:58:28 am Pawel Jakub Dawidek wrote: >>>> On Sun, Oct 23, 2011 at 11:44:45AM +0300, Kostik Belousov wrote: >>>>> On Sun, Oct 23, 2011 at 08:10:38AM +0200, Pawel Jakub Dawidek wrote: >>>>>> My suggestion would be that if we won't be able to fix it before 9.0, >>>>>> we should turn this assertion off, as the system seems to be able to >>>>>> recover. >>>>> >>>>> Shipped kernels have all assertions turned off. >>>> >>>> Yes, I'm aware of that, but many people compile their production kernels >>>> with INVARIANTS/INVARIANT_SUPPORT to fail early instead of eg. >>>> corrupting data. I'd be fine in moving this under DIAGNOSTIC or changing >>>> it into a printf, so it will be visible. >>> >>> No, the kernel is corrupting things in other places when this is true, so >>> if you are running with INVARIANTS, we want to know about it. Specifically, >>> in several places in TCP we assume that rcv_adv>= rcv_nxt, and depend on >>> being able to do 'rcv_adv - rcv_nxt'. >>> >>> In this case, it looks like the difference is consistently less than one >>> frame. I suspect the other end of the connection is sending just beyond the >>> end of the advertised window (it probably assumes it is better to send a full >>> frame if it has that much pending data even though part of it is beyond the >>> window edge vs sending a truncated packet that just fills the window) and that >>> that frame is accepted ok in the header prediction case and it's ACK is >>> delayed, but the next packet to arrive then trips over this assumption. >>> >>> Since 'win' is guaranteed to be non-negative and we explicitly cast >>> 'rcv_adv - rcv_nxt' to (int) in the following line that the assert is checking >>> for: >>> >>> tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); >>> >>> I think we already handle this case ok and perhaps the assertion can just be >>> removed? Not sure if others feel that it warrants a comment to note that this >>> is the case being handled. >> >> I added debug to the places where rcv_adv and rcv_nxt are modified. Here >> is what happens before the panic occurs: >> >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022361548 rcv_adv 4022360100 diff -1448 >> tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022362298 rcv_adv 4022361548 diff -750 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022363746 rcv_adv 4022362298 diff -1448 >> tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022364836 rcv_adv 4022363746 diff -1090 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022366284 rcv_adv 4022364836 diff -1448 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022370628 rcv_adv 4022369690 diff -938 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022379140 rcv_adv 4022377692 diff -1448 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022387792 rcv_adv 4022386344 diff -1448 >> tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022388890 rcv_adv 4022387792 diff -1098 >> tcp_do_segment:1722 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022390338 rcv_adv 4022388890 diff -1448 >> tcp_do_segment:2847 negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 diff -221 >> panic: tcp_input negative window: tp 0xfffffe000dab1b70 rcv_nxt 4022394563 rcv_adv 4022394342 win=0 diff -221 >> >> I can send you the full log if you want, I've plenty of messages where >> rcv_adv< rcv_nxt, not all of them trigger this assertion. > > The assertion would be triggered when the next packet arrives (as I said > above). Try modifying your debugging output to also log if the ACK is > delayed. I suspect it is not delayed until the last one. (Pushing out an > ACK will reset rcv_adv to be beyond rcv_nxt in tcp_output(), so in the case > of an immediate ACK, rcv_nxt> rcv_adv is only a transient condition all > under a single lock invocation so never visible to other consumers of the > protocol control block.) If that is what you see, then that confirms what > I guessed above and I will likely just remove the assertion in tcp_input() > and patch the timewait code to handle this case. > Pawel, have you been able to confirm John's hypothesis? What I don't quite get is why we haven't had a lot more reports of this issue... Cheers, Lawrence From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 05:46:54 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0A67106564A; Fri, 28 Oct 2011 05:46:54 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (60.wheelsystems.com [83.12.187.60]) by mx1.freebsd.org (Postfix) with ESMTP id 5A09C8FC0C; Fri, 28 Oct 2011 05:46:54 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id 2CF22A39; Fri, 28 Oct 2011 07:46:52 +0200 (CEST) Date: Fri, 28 Oct 2011 07:46:07 +0200 From: Pawel Jakub Dawidek To: Lawrence Stewart Message-ID: <20111028054605.GF1667@garage.freebsd.pl> References: <20111022084931.GD1697@garage.freebsd.pl> <201110240814.22368.jhb@freebsd.org> <20111026075431.GB1672@garage.freebsd.pl> <201110260753.37264.jhb@freebsd.org> <4EA9F76E.9010008@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RE3pQJLXZi4fr8Xo" Content-Disposition: inline In-Reply-To: <4EA9F76E.9010008@freebsd.org> X-OS: FreeBSD 9.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Kostik Belousov , freebsd-net@freebsd.org, freebsd-current@freebsd.org, Andre Oppermann , John Baldwin Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 05:46:54 -0000 --RE3pQJLXZi4fr8Xo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 28, 2011 at 11:29:34AM +1100, Lawrence Stewart wrote: > On 10/26/11 22:53, John Baldwin wrote: > > The assertion would be triggered when the next packet arrives (as I said > > above). Try modifying your debugging output to also log if the ACK is > > delayed. I suspect it is not delayed until the last one. (Pushing out= an > > ACK will reset rcv_adv to be beyond rcv_nxt in tcp_output(), so in the = case > > of an immediate ACK, rcv_nxt> rcv_adv is only a transient condition all > > under a single lock invocation so never visible to other consumers of t= he > > protocol control block.) If that is what you see, then that confirms w= hat > > I guessed above and I will likely just remove the assertion in tcp_inpu= t() > > and patch the timewait code to handle this case. > > >=20 > Pawel, have you been able to confirm John's hypothesis? [...] Yeah, sorry. I moved the debug to the points where we drop the t_inpcb lock and I still see rcv_nxt being greater than rcv_adv: tcp_do_segment:2970 negative window: tp 0xfffffe00685ee3d0 rcv_nxt 1312878= 324 rcv_adv 1312878187 This is just before the INP_WUNLOCK(tp->t_inpcb) under 'check_delack' label. I see this a lot (it was logged 545 times for 11 different tp pointers during 24h period). tcp_do_segment:3009 negative window: tp 0xfffffe005cfc6000 rcv_nxt 1442546= 453 rcv_adv 1442545722 This is just before calling tcp_output(). This one was logged 65 times for 3 different tp pointers. I placed a debug also after tcp_output() call, but it is not logged, so once we return from tcp_output() everything is fine. The panic would be triggered 115 times for 5 different tp pointers during that time. I write 'tp pointers' as I'm not 100% sure if the same pointer always represents the same connection or if it is reused. > [...] What I don't=20 > quite get is why we haven't had a lot more reports of this issue... Maybe because my TCP/IP stack is heavly modified? ...not:) No idea to be honest. Ask Ken to turn on INVARIANTS in 9.0-RC2 and we will see:) --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com --RE3pQJLXZi4fr8Xo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAk6qQZ0ACgkQForvXbEpPzTIcwCcC6C06i2hgJshb29NsE5iZ5NJ l/EAoO/qBU7/4+8tJOElQQUArjNWpq4t =CGv+ -----END PGP SIGNATURE----- --RE3pQJLXZi4fr8Xo-- From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 08:38:23 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E69BB106566C for ; Fri, 28 Oct 2011 08:38:23 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id 8FA078FC0C for ; Fri, 28 Oct 2011 08:38:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id 339711C0896; Fri, 28 Oct 2011 12:38:19 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.51]) by mailgate.pik.ru (Postfix) with ESMTP id 1BB841C088C; Fri, 28 Oct 2011 12:38:19 +0400 (MSD) Received: from EX21PIK.PICompany.ru ([192.168.156.131]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Fri, 28 Oct 2011 12:34:10 +0400 Received: from [192.168.148.9] (192.168.148.9) by EX21PIK.PICompany.ru (192.168.156.131) with Microsoft SMTP Server id 14.1.218.12; Fri, 28 Oct 2011 12:34:10 +0400 Message-ID: <4EAA68EF.1080700@hotplug.ru> Date: Fri, 28 Oct 2011 12:33:51 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: Arnaud Lacombe References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 Oct 2011 08:34:10.0664 (UTC) FILETIME=[5D8F7E80:01CC954C] Cc: Hooman, freebsd-net@freebsd.org, Fazaeli , Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 08:38:24 -0000 >>> Hi, >>> >>> Can yan you pls post the output of these command _when_ the problem >>> happens? >>> >>> uname -a >>> sysctl dev.em >>> netstat -ind >>> ifconfig >>> >> Hi Hooman >> >> Here is what I've got when the script triggered just in time when the >> interface was locked >> >> >> 11.10.26-23:39:10 ... interface em0 is down... >> >> FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 20:20:25 >> > Please upgrade to 8-STABLE, similar issues have been fixed there. > > Thanks, > - Arnaud > Hello, Arnaud My system's age is about a week old, updated to tag=RELENG_8. I've csup'ed yesterday, but I didn't noticed any changes related, did I miss something important? Anyway I'll do the upgrade this weekend maybe. Is there any point to move to RELENG_9? I don't want to mess with the 9'th changes for a little bit while. From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 11:56:33 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9FBE4106564A; Fri, 28 Oct 2011 11:56:33 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 74C618FC1A; Fri, 28 Oct 2011 11:56:33 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 5BFAD46B09; Fri, 28 Oct 2011 07:56:27 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id F011C8A02E; Fri, 28 Oct 2011 07:56:26 -0400 (EDT) From: John Baldwin To: Pawel Jakub Dawidek Date: Fri, 28 Oct 2011 07:56:23 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <20111022084931.GD1697@garage.freebsd.pl> <4EA9F76E.9010008@freebsd.org> <20111028054605.GF1667@garage.freebsd.pl> In-Reply-To: <20111028054605.GF1667@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201110280756.23317.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 28 Oct 2011 07:56:27 -0400 (EDT) Cc: Kostik Belousov , Lawrence Stewart , freebsd-current@freebsd.org, Andre Oppermann , freebsd-net@freebsd.org Subject: Re: 9.0-RC1 panic in tcp_input: negative winow. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 11:56:33 -0000 On Friday, October 28, 2011 1:46:07 am Pawel Jakub Dawidek wrote: > On Fri, Oct 28, 2011 at 11:29:34AM +1100, Lawrence Stewart wrote: > > On 10/26/11 22:53, John Baldwin wrote: > > > The assertion would be triggered when the next packet arrives (as I said > > > above). Try modifying your debugging output to also log if the ACK is > > > delayed. I suspect it is not delayed until the last one. (Pushing out an > > > ACK will reset rcv_adv to be beyond rcv_nxt in tcp_output(), so in the case > > > of an immediate ACK, rcv_nxt> rcv_adv is only a transient condition all > > > under a single lock invocation so never visible to other consumers of the > > > protocol control block.) If that is what you see, then that confirms what > > > I guessed above and I will likely just remove the assertion in tcp_input() > > > and patch the timewait code to handle this case. > > > > > > > Pawel, have you been able to confirm John's hypothesis? [...] > > Yeah, sorry. I moved the debug to the points where we drop the t_inpcb > lock and I still see rcv_nxt being greater than rcv_adv: > > tcp_do_segment:2970 negative window: tp 0xfffffe00685ee3d0 rcv_nxt 1312878324 rcv_adv 1312878187 Yes, I still expect this. What I want to see is if 'delack' is always true in this case. > This is just before the INP_WUNLOCK(tp->t_inpcb) under 'check_delack' > label. I see this a lot (it was logged 545 times for 11 different tp > pointers during 24h period). > > tcp_do_segment:3009 negative window: tp 0xfffffe005cfc6000 rcv_nxt 1442546453 rcv_adv 1442545722 > > This is just before calling tcp_output(). This one was logged 65 times > for 3 different tp pointers. > I placed a debug also after tcp_output() call, but it is not logged, so > once we return from tcp_output() everything is fine. That is consistent with what I expect then, since in the delack case, tcp_output() isn't called. -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 14:51:30 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFB73106566B for ; Fri, 28 Oct 2011 14:51:30 +0000 (UTC) (envelope-from nyoman.bogi@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 75EB18FC0A for ; Fri, 28 Oct 2011 14:51:30 +0000 (UTC) Received: by gyd8 with SMTP id 8so4953367gyd.13 for ; Fri, 28 Oct 2011 07:51:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=y6M89z/uoIVYuD3K7Dm4h9HnWEyOgE3zQjSK5/pWh+0=; b=a6ltoQwa6six83uCcQp0v3FvOtUfGYt+47nr/Gt+irEcfc6uIepFjJWQmZDzj6q9us epMo2WNrNboQF2zguuHAlIqd7q6QElHjClfKlBTAXno7fgKB6TdOVQK/ubt/dqrrkMhV thVp1NBAJrsVvIxMV6oQDQ8MeEbf5pCMm/aUA= MIME-Version: 1.0 Received: by 10.236.200.201 with SMTP id z49mr4172757yhn.20.1319811745824; Fri, 28 Oct 2011 07:22:25 -0700 (PDT) Received: by 10.236.157.73 with HTTP; Fri, 28 Oct 2011 07:22:25 -0700 (PDT) Date: Fri, 28 Oct 2011 21:22:25 +0700 Message-ID: From: "nyoman.bogi@gmail.com" To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: multiple ISP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 14:51:30 -0000 dear all, I need to set up a router (using FreeBSD) that connect to the Internet to accomodate multiple ISP, so users can be load balanced through those several ISP lines. how can I do that? thanks in advance ------------------------------- Bogi Aditya Sisfo - IMTelkom http://bogi.blog.imtelkom.ac.id From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 15:10:36 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F3DD3106568B for ; Fri, 28 Oct 2011 15:10:35 +0000 (UTC) (envelope-from gpm@hotplug.ru) Received: from gate.pikinvest.ru (gate.pikinvest.ru [87.245.155.170]) by mx1.freebsd.org (Postfix) with ESMTP id A0B568FC22 for ; Fri, 28 Oct 2011 15:10:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mailgate.pik.ru (Postfix) with ESMTP id D3D451C0831; Fri, 28 Oct 2011 19:10:32 +0400 (MSD) Received: from EX03PIK.PICompany.ru (unknown [192.168.156.51]) by mailgate.pik.ru (Postfix) with ESMTP id D1B711C0822; Fri, 28 Oct 2011 19:10:32 +0400 (MSD) Received: from EX21PIK.PICompany.ru ([192.168.156.131]) by EX03PIK.PICompany.ru with Microsoft SMTPSVC(6.0.3790.4675); Fri, 28 Oct 2011 19:10:17 +0400 Received: from [192.168.148.9] (192.168.148.9) by EX21PIK.PICompany.ru (192.168.156.131) with Microsoft SMTP Server id 14.1.218.12; Fri, 28 Oct 2011 19:10:16 +0400 Message-ID: <4EAAC5C5.6090803@hotplug.ru> Date: Fri, 28 Oct 2011 19:09:57 +0400 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.23) Gecko/20110922 Thunderbird/3.1.15 MIME-Version: 1.0 To: , Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 28 Oct 2011 15:10:17.0059 (UTC) FILETIME=[B36F9330:01CC9583] Cc: Subject: ipfw reass brakes ipv6 operation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 15:10:36 -0000 Hi all I've got into some strange behavior with ipv6. Somehow ipfw reassembly totally brakes it's operation. As soon as I add a rule "ipfw add 100 reass all from any to any in" all ipv6 operation is not available any more, I can only ping6 localhost. Outgoing ipv6 packets are OK, I can see them via tcpdump on an interface stf0 and after that leaving encapsulated in ip4 through another interface. But all incoming ipv6 packets are blackholed. I can see them arriving as an encapsulated payload in ip4 and after that they disappear. I don't know if this a bug or a feature, using "ipfw add reass ip4 from any to any in" works as a workaround. Shouldn't reass just pass ipv6 packets intact? Or if it is a feature than maybe there should be a note in IPFW(8) man page to not to use reass for anything except ip4? Thanks. From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 17:02:43 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F1C851065672; Fri, 28 Oct 2011 17:02:43 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from mail.ciam.ru (mail.ciam.ru [91.209.218.18]) by mx1.freebsd.org (Postfix) with ESMTP id A78448FC15; Fri, 28 Oct 2011 17:02:40 +0000 (UTC) Received: from [46.242.19.18] (helo=[172.16.100.20]) by mail.ciam.ru with esmtpa (Exim 4.x) id 1RJpWo-0005TO-Tw; Fri, 28 Oct 2011 20:43:35 +0400 Message-ID: <4EAADBB6.5090901@FreeBSD.org> Date: Fri, 28 Oct 2011 20:43:34 +0400 From: Sergey Matveychuk User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 MIME-Version: 1.0 To: Emil Muratov References: <4EAAC5C5.6090803@hotplug.ru> In-Reply-To: <4EAAC5C5.6090803@hotplug.ru> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-ipfw@freebsd.org, freebsd-net@freebsd.org Subject: Re: ipfw reass brakes ipv6 operation X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 17:02:44 -0000 28.10.2011 19:09, Emil Muratov wrote: > > Hi all > > I've got into some strange behavior with ipv6. Somehow ipfw reassembly > totally brakes it's operation. > As soon as I add a rule "ipfw add 100 reass all from any to any in" all > ipv6 operation is not available any more, > I can only ping6 localhost. Outgoing ipv6 packets are OK, I can see them > via tcpdump on an interface stf0 and after that leaving encapsulated in > ip4 through another interface. But all incoming ipv6 packets are > blackholed. I can see them arriving as an encapsulated payload in ip4 > and after that they disappear. I don't know if this a bug or a feature, > using "ipfw add reass ip4 from any to any in" works as a workaround. > Shouldn't reass just pass ipv6 packets intact? Or if it is a feature > than maybe there should be a note in IPFW(8) man page to not to use > reass for anything except ip4? Yes, reass implemented only for ipv4 and breaks ipv6 packets. It should be fixed, not documented. From owner-freebsd-net@FreeBSD.ORG Fri Oct 28 23:43:15 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40B25106566C for ; Fri, 28 Oct 2011 23:43:15 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 1403C8FC1C for ; Fri, 28 Oct 2011 23:43:14 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p9SNhC4N073241 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 28 Oct 2011 16:43:13 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4EAB3E0C.3020203@freebsd.org> Date: Fri, 28 Oct 2011 16:43:08 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.23) Gecko/20110920 Thunderbird/3.1.15 MIME-Version: 1.0 To: "nyoman.bogi@gmail.com" References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: multiple ISP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Oct 2011 23:43:15 -0000 On 10/28/11 7:22 AM, nyoman.bogi@gmail.com wrote: > dear all, > > I need to set up a router (using FreeBSD) > that connect to the Internet > to accomodate multiple ISP, > so users can be load balanced through > those several ISP lines. > > how can I do that? Is it ok for you to NAT the connecitons? You pretty much have to unless you can convince your ISPs to call you a peer (unlikely). Basically NAT both outgoing interfaces (see descriptions elsewhere on how to run natd on two interfaces), and then use the setfib rule in ipfw to select which routing table to use for each session and then set up the tables to use different outgoing interfaces. you could also use the 'fwd' ipfw rule instead of the setfib rule. > thanks in advance > > ------------------------------- > Bogi Aditya > Sisfo - IMTelkom > http://bogi.blog.imtelkom.ac.id > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 00:11:59 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5E4B1065687 for ; Sat, 29 Oct 2011 00:11:59 +0000 (UTC) (envelope-from lacombar@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 63B738FC1D for ; Sat, 29 Oct 2011 00:11:59 +0000 (UTC) Received: by wyh11 with SMTP id 11so35215wyh.13 for ; Fri, 28 Oct 2011 17:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=aBN07AVCKOvfwBzeTV1AIFiuyAPJ5xq/Y0+pIw4FUVQ=; b=WkSr1QkL85xVWxIZt8qsHOA10Fl4lyOLob3FQ8e3GsetFT3GUGofz/o93ep4Y/A2+e 37klS6NzMJdx7EfXZDBo0Gx6l9hQvj5hxzTNDZ/P0Jx7NIZrdY4ByxtCB2hMcmZS4Vxa 6GuWw44znaxCTBoqVUvihL3ygenrTu9IGtAxw= MIME-Version: 1.0 Received: by 10.216.221.11 with SMTP id q11mr170752wep.86.1319847118332; Fri, 28 Oct 2011 17:11:58 -0700 (PDT) Received: by 10.180.105.162 with HTTP; Fri, 28 Oct 2011 17:11:57 -0700 (PDT) In-Reply-To: <4EAA68EF.1080700@hotplug.ru> References: <4E8F157A.40702@sentex.net> <4E8F51D4.1060509@sentex.net> <4EA7E203.3020306@sepehrs.com> <4EA80818.3030504@sentex.net> <4EA80F88.4000400@hotplug.ru> <4EA82715.2000404@gmail.com> <4EA8FA40.7010504@hotplug.ru> <4EAA68EF.1080700@hotplug.ru> Date: Fri, 28 Oct 2011 20:11:57 -0400 Message-ID: From: Arnaud Lacombe To: Emil Muratov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org, Hooman Fazaeli , Jason Wolfe Subject: Re: Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 00:11:59 -0000 Hi, On Fri, Oct 28, 2011 at 4:33 AM, Emil Muratov wrote: > >>>> Hi, >>>> >>>> Can yan you pls post the output of these command _when_ the problem >>>> happens? >>>> >>>> uname -a >>>> sysctl dev.em >>>> netstat -ind >>>> ifconfig >>>> >>> Hi Hooman >>> >>> Here is what I've got when the script triggered just in time when the >>> interface was locked >>> >>> >>> 11.10.26-23:39:10 ... interface em0 is down... >>> >>> FreeBSD ion.hotplug.ru 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Oct 20 >>> 20:20:25 >>> >> Please upgrade to 8-STABLE, similar issues have been fixed there. >> >> Thanks, >> =A0- Arnaud >> > Hello, Arnaud > My system's age is about a week old, updated to tag=3DRELENG_8. > I've csup'ed yesterday, but I didn't noticed any changes related, did I m= iss > something important? Anyway I'll do the upgrade this weekend maybe. Is > =A0there any point to move to RELENG_9? I don't want to mess with the 9't= h > changes for a little bit while. > Sorry, I stopped reading at 8.2 :/ Actually, your problem do not seem like the one fixed around April. my bad, - Arnaud From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 05:57:34 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D85D5106566C for ; Sat, 29 Oct 2011 05:57:34 +0000 (UTC) (envelope-from samspeed@mail.ru) Received: from f193.mail.ru (f193.mail.ru [217.69.129.202]) by mx1.freebsd.org (Postfix) with ESMTP id 04CBC8FC12 for ; Sat, 29 Oct 2011 05:57:33 +0000 (UTC) Received: from mail by f193.mail.ru with local id 1RK1v8-0006v8-00; Sat, 29 Oct 2011 09:57:30 +0400 Received: from [77.45.236.224] by e.mail.ru with HTTP; Sat, 29 Oct 2011 09:57:30 +0400 From: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= To: pyunyh@gmail.com Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [77.45.236.224] Date: Sat, 29 Oct 2011 09:57:30 +0400 References: <20111026164800.GA13234@michelle.cdnetworks.com> In-Reply-To: <20111026164800.GA13234@michelle.cdnetworks.com> X-Priority: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Message-Id: X-Spam: Not detected X-Mras: Ok X-Mailman-Approved-At: Sat, 29 Oct 2011 11:08:09 +0000 Cc: freebsd-net@freebsd.org Subject: Re: PCI-E VT6130 NIC (if_vge) hang system with gigabit link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?UTF-8?B?QW5kcmV5IFNtYWdpbg==?= List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 05:57:34 -0000 Ck9rLiBXaXRoIGF1dG9uZWdvdGlhdGlvbiBpZmNvbmZpZyBzaG93IHNwZWVkIDEwME1CaXQuCndp dGggbWFudWFsIDEwMDBiYXNlVCBmdWxsLWR1cGxleCBzZXR0aW5ncyBpbiBkbWVzZzoKdmdlMDog ZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdl MDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBsaW5rIHN0YXRlIGNoYW5nZWQg dG8gVVAKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkg YXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBN SUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFy dCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBz dGFydCBNSUkgYXV0b3BvbGwKdmdlMDogbGluayBzdGF0ZSBjaGFuZ2VkIHRvIERPV04KdmdlMDog TUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdl MDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwK dmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3Bv bGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0 b3BvbGwKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0 aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xl YXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlC IGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6 IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2 Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91 dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1l ZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAg dGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBk dW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50 ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBj b3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBN SUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdl MDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQh CnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQg b3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRp bWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVh ciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIg Y2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDog TUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZn ZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0 IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVk IG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0 aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1 bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRl ciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogZmxhZ3M9 ODk0MzxVUCxCUk9BRENBU1QsUlVOTklORyxQUk9NSVNDLFNJTVBMRVgsTVVMVElDQVNUPiBtZXRy aWMgMCBtdHUgMTUwMAoJb3B0aW9ucz0zODliPFJYQ1NVTSxUWENTVU0sVkxBTl9NVFUsVkxBTl9I V1RBR0dJTkcsVkxBTl9IV0NTVU0sV09MX1VDQVNULFdPTF9NQ0FTVCxXT0xfTUFHSUM+CglldGhl ciAwMDphMTpiMDowMDowMTozZQoJaW5ldCAxOTIuMTY4LjQuMSBuZXRtYXNrIDB4ZmZmZmZmZmMg YnJvYWRjYXN0IDE5Mi4xNjguNC4zCgluZDYgb3B0aW9ucz0yOTxQRVJGT1JNTlVELElGRElTQUJM RUQsQVVUT19MSU5LTE9DQUw+CgltZWRpYTogRXRoZXJuZXQgMTAwMGJhc2VUIDxmdWxsLWR1cGxl eD4gKDEwYmFzZVQvVVRQIDxoYWxmLWR1cGxleD4pCglzdGF0dXM6IG5vIGNhcnJpZXIKCndpdGgg bWFudWFsIDEwMDBiYXNlVCBmdWxsLWR1cGxleCxtYXN0ZXIgc2V0dGluZ3MgaW4gZG1lc2c6CnZn ZTA6IHByb21pc2N1b3VzIG1vZGUgZW5hYmxlZAp2Z2UwOiBsaW5rIHN0YXRlIGNoYW5nZWQgdG8g RE9XTgpuZmUwOiBsaW5rIHN0YXRlIGNoYW5nZWQgdG8gRE9XTgp2Z2UwOiBsaW5rIHN0YXRlIGNo YW5nZWQgdG8gVVAKdmdlMDogbGluayBzdGF0ZSBjaGFuZ2VkIHRvIERPV04KdmdlMDogZmFpbGVk IHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFp bGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBsaW5rIHN0YXRlIGNoYW5nZWQgdG8gVVAK dmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3Bv bGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0 b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkg YXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBN SUkgYXV0b3BvbGwKdmdlMDogbGluayBzdGF0ZSBjaGFuZ2VkIHRvIERPV04KdmdlMDogTUlJIHJl YWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJ IHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDog TUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdl MDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwK dmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBv dXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGlt ZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFy IHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBj bGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBN SUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdl MDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQh CnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQg b3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRp bWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVt cCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVy IGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291 bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlC IGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6 IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2 Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91 dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1l ZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIg dGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNs ZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1J QiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2Uw OiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEK dmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBv dXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGlt ZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1w IHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIg ZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3Vu dGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIg Y291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDog TUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZn ZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0 IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVk IG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0 aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xl YXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlC IGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6 IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2 Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91 dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1l ZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAg dGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBk dW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50 ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBj b3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBN SUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdl MDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQh CnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQg b3V0IQp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBh dXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1J SSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0 IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0 YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRv IHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVk IHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFp bGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDog ZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVk IG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0 CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQg b3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGlt ZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQg dGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlCIGNv dW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1J QiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2Uw OiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEK dmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBv dXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGlt ZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFy IHRpbWVkIG91dCEKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFy dCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBz dGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0 byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxl ZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQh CnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JSSByZWFkIHRpbWVkIG91dAp2Z2Uw OiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSByZWFkIHRpbWVkIG91dAp2 Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSByZWFkIHRpbWVkIG91 dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSByZWFkIHRpbWVk IG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSByZWFkIHRp bWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSByZWFk IHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSBy ZWFkIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1J SSByZWFkIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6 IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2 Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91 dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1l ZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIg dGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNs ZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1J QiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2Uw OiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY291bnRlciBkdW1wIHRpbWVkIG91dCEK dmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6 IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZn ZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0 CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQg b3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlCIGNvdW50ZXIg ZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JSSByZWFk IHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSBy ZWFkIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1J SSByZWFkIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6 IE1JSSByZWFkIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZn ZTA6IE1JSSB3cml0ZSB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9s bAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRv cG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBh dXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1J SSBhdXRvcG9sbAp2Z2UwOiBNSUkgd3JpdGUgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFy dCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHdyaXRlIHRpbWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8g c3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSB3cml0ZSB0aW1lZCBvdXQKdmdlMDogZmFpbGVk IHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDogZmFp bGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdlMDog ZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQKdmdl MDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkgcmVhZCB0aW1lZCBvdXQK dmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUIgY291bnRlciBkdW1w IHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDogTUlCIGNvdW50ZXIg ZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjb3Vu dGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUIg Y291bnRlciBkdW1wIHRpbWVkIG91dCEKdmdlMDogTUlCIGNsZWFyIHRpbWVkIG91dCEKdmdlMDog TUlCIGNvdW50ZXIgZHVtcCB0aW1lZCBvdXQhCnZnZTA6IE1JQiBjbGVhciB0aW1lZCBvdXQhCnZn ZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0 IQp2Z2UwOiB3YXRjaGRvZyB0aW1lb3V0CnZnZTA6IE1JQiBjb3VudGVyIGR1bXAgdGltZWQgb3V0 IQp2Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBzb2Z0IHJlc2V0IHRpbWVkIG91dAp2 Z2UwOiBNSUIgY2xlYXIgdGltZWQgb3V0IQp2Z2UwOiBNSUkgd3JpdGUgdGltZWQgb3V0CnZnZTA6 IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZn ZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0 CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHJlYWQgdGltZWQg b3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDogTUlJIHdyaXRlIHRp bWVkIG91dAp2Z2UwOiBmYWlsZWQgdG8gc3RhcnQgTUlJIGF1dG9wb2xsCnZnZTA6IE1JSSB3cml0 ZSB0aW1lZCBvdXQKdmdlMDogZmFpbGVkIHRvIHN0YXJ0IE1JSSBhdXRvcG9sbAp2Z2UwOiBNSUkg d3JpdGUgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdlMDog TUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwKdmdl MDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3BvbGwK dmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0b3Bv bGwKdmdlMDogTUlJIHJlYWQgdGltZWQgb3V0CnZnZTA6IGZhaWxlZCB0byBzdGFydCBNSUkgYXV0 b3BvbGwKdmdlMDogZmxhZ3M9ODk0MzxVUCxCUk9BRENBU1QsUlVOTklORyxQUk9NSVNDLFNJTVBM RVgsTVVMVElDQVNUPiBtZXRyaWMgMCBtdHUgMTUwMAoJb3B0aW9ucz0zODliPFJYQ1NVTSxUWENT VU0sVkxBTl9NVFUsVkxBTl9IV1RBR0dJTkcsVkxBTl9IV0NTVU0sV09MX1VDQVNULFdPTF9NQ0FT VCxXT0xfTUFHSUM+CglldGhlciAwMDphMTpiMDowMDowMTozZQoJaW5ldCAxOTIuMTY4LjQuMSBu ZXRtYXNrIDB4ZmZmZmZmZmMgYnJvYWRjYXN0IDE5Mi4xNjguNC4zCgluZDYgb3B0aW9ucz0yOTxQ RVJGT1JNTlVELElGRElTQUJMRUQsQVVUT19MSU5LTE9DQUw+CgltZWRpYTogRXRoZXJuZXQgMTAw MGJhc2VUIDxmdWxsLWR1cGxleCxtYXN0ZXI+ICgxMGJhc2VUL1VUUCA8aGFsZi1kdXBsZXg+KQoJ c3RhdHVzOiBubyBjYXJyaWVyCgoKCgoKPiA+Cj4gPiAyNiDQvtC60YLRj9Cx0YDRjyAyMDExLCAw MDoxMSDQvtGCIFlvbmdIeWVvbiBQWVVOIDxweXVueWhAZ21haWwuY29tPjoKPiA+ID4gT24gVHVl LCBPY3QgMjUsIDIwMTEgYXQgMTA6NDQ6NDhBTSArMDQwMCwgQW5kcmV5IFNtYWdpbiB3cm90ZToK PiA+ID4gPiBIaSBBTEwgISAgSWYgSSBjb25uZWN0IGdpZ2FiaXQgc3dpdGNoIHRvIG15IGNhcmQg LSBzeXN0ZW0gaGFuZyB1bnRpbCBJIHVucGx1ZyBwYXRjaGNvcmQgZnJvbSBkZXZpY2UuCj4gPiA+ ID4gV2l0aCAxMDBNYml0IHN3aXRjaCBjYXJkIHdvcmsgZ29vZC4KPiA+ID4KPiA+ID4gU2hvdyBt ZSB0aGUgb3V0cHV0IG9mIGRtZXNnIGFuZCAncGNpY29uZiAtbGNidicuCj4gPiA+Cj4gPiA+ID4g U3lzdGVtOiAgQ3VycmVudCAtIHIyMjYxNjMKPiA+ID4gPgo+ID4gPgo+ID4KPiA= From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 12:05:30 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 098FC106566C; Sat, 29 Oct 2011 12:05:30 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D57FC8FC0C; Sat, 29 Oct 2011 12:05:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9TC5TM2064654; Sat, 29 Oct 2011 12:05:29 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9TC5TXT064650; Sat, 29 Oct 2011 12:05:29 GMT (envelope-from linimon) Date: Sat, 29 Oct 2011 12:05:29 GMT Message-Id: <201110291205.p9TC5TXT064650@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/162110: [igb] [panic] RELENG_9 panics on boot in IGB driver - [regression] from 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 12:05:30 -0000 Old Synopsis: Releng_9 panics on boot in IGB driver - regression from 8.2 New Synopsis: [igb] [panic] RELENG_9 panics on boot in IGB driver - [regression] from 8.2 Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Sat Oct 29 12:04:50 UTC 2011 Responsible-Changed-Why: reclassify. http://www.freebsd.org/cgi/query-pr.cgi?pr=162110 From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 12:11:41 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A61981065672; Sat, 29 Oct 2011 12:11:41 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7E2318FC0C; Sat, 29 Oct 2011 12:11:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9TCBfkV072967; Sat, 29 Oct 2011 12:11:41 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9TCBf4e072963; Sat, 29 Oct 2011 12:11:41 GMT (envelope-from linimon) Date: Sat, 29 Oct 2011 12:11:41 GMT Message-Id: <201110291211.p9TCBf4e072963@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/162068: [msk] Marvell Yukon GE onboard card does not work in 1000baseTX mode X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 12:11:41 -0000 Old Synopsis: msk Marvell Yukon GE onboard card does not work in 1000baseTX mode New Synopsis: [msk] Marvell Yukon GE onboard card does not work in 1000baseTX mode Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Sat Oct 29 12:11:29 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=162068 From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 12:13:38 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D79111065768; Sat, 29 Oct 2011 12:13:38 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AFBBA8FC14; Sat, 29 Oct 2011 12:13:38 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9TCDc5R073639; Sat, 29 Oct 2011 12:13:38 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9TCDcSk073633; Sat, 29 Oct 2011 12:13:38 GMT (envelope-from linimon) Date: Sat, 29 Oct 2011 12:13:38 GMT Message-Id: <201110291213.p9TCDcSk073633@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/162028: [ixgbe] [patch] misplaced #endif in ixgbe.c X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 12:13:39 -0000 Old Synopsis: misplaced #endif in ixgbe.c New Synopsis: [ixgbe] [patch] misplaced #endif in ixgbe.c Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Sat Oct 29 12:13:14 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=162028 From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 12:27:21 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D18E8106564A for ; Sat, 29 Oct 2011 12:27:21 +0000 (UTC) (envelope-from nyoman.bogi@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9137D8FC12 for ; Sat, 29 Oct 2011 12:27:21 +0000 (UTC) Received: by gyg4 with SMTP id 4so521958gyg.13 for ; Sat, 29 Oct 2011 05:27:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=b9lSD+RudV/KrSrEUGKP1NpKUqgrsrha826dxMBX7JQ=; b=WgiF6Xbfhz7FgnA1OEGLbxwsFswtu6R9bt8D+t6PDaGrNhmQZVmjr6z0RlIXNj1Zo2 6n0pFdBAGJSCoPwqxHk4y1APLI+46ULYHtpoPEmw5Vm59UtUU9IMWHOB52PmoSWeR7bI t7bgAb0ZfWH3BoIS4Ueq4FP318UZ/vmXuhaFM= MIME-Version: 1.0 Received: by 10.236.181.131 with SMTP id l3mr8308998yhm.105.1319891240936; Sat, 29 Oct 2011 05:27:20 -0700 (PDT) Received: by 10.236.157.73 with HTTP; Sat, 29 Oct 2011 05:27:20 -0700 (PDT) In-Reply-To: <4EAB3E0C.3020203@freebsd.org> References: <4EAB3E0C.3020203@freebsd.org> Date: Sat, 29 Oct 2011 19:27:20 +0700 Message-ID: From: "nyoman.bogi@gmail.com" To: Julian Elischer Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: multiple ISP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 12:27:21 -0000 thanks Julian, buat I think fwd in ipfw will use round robin mechanism instead of load balancing, which I prefer. or perhaps my knowledge about fwd is a little bit old maybe fwd can do the load balancing? On Sat, Oct 29, 2011 at 6:43 AM, Julian Elischer wrote: > On 10/28/11 7:22 AM, nyoman.bogi@gmail.com wrote: > >> dear all, >> >> I need to set up a router (using FreeBSD) >> that connect to the Internet >> to accomodate multiple ISP, >> so users can be load balanced through >> those several ISP lines. >> >> how can I do that? >> > > Is it ok for you to NAT the connecitons? You pretty much have to unless > you can convince your ISPs > to call you a peer (unlikely). > > Basically NAT both outgoing interfaces (see descriptions elsewhere > on how to run natd on two interfaces), and then use the setfib rule in > ipfw to select which routing table to use for each session > and then set up the tables to use different outgoing interfaces. > > you could also use the 'fwd' ipfw rule instead of the setfib rule. > > > thanks in advance >> >> ------------------------------**- >> Bogi Aditya >> Sisfo - IMTelkom >> http://bogi.blog.imtelkom.ac.**id >> ______________________________**_________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@**freebsd.org >> " >> >> > -- ------------------------------- Bogi Aditya Sisfo - IMTelkom http://bogi.blog.imtelkom.ac.id From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 13:30:19 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 86A57106566B for ; Sat, 29 Oct 2011 13:30:19 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5CEB78FC0C for ; Sat, 29 Oct 2011 13:30:19 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9TDUJcZ040842 for ; Sat, 29 Oct 2011 13:30:19 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9TDUJXP040832; Sat, 29 Oct 2011 13:30:19 GMT (envelope-from gnats) Date: Sat, 29 Oct 2011 13:30:19 GMT Message-Id: <201110291330.p9TDUJXP040832@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Sergey Kandaurov Cc: Subject: Re: kern/162028: [ixgbe] [patch] misplaced #endif in ixgbe.c X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Sergey Kandaurov List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 13:30:19 -0000 The following reply was made to PR kern/162028; it has been noted by GNATS. From: Sergey Kandaurov To: bug-followup@FreeBSD.org, hoomanfazaeli@gmail.com Cc: Subject: Re: kern/162028: [ixgbe] [patch] misplaced #endif in ixgbe.c Date: Sat, 29 Oct 2011 16:58:09 +0400 I have a more complete patch. Can you test it please? Index: sys/dev/ixgbe/ixgbe.c =================================================================== --- sys/dev/ixgbe/ixgbe.c (revision 226068) +++ sys/dev/ixgbe/ixgbe.c (working copy) @@ -867,16 +867,15 @@ static int ixgbe_ioctl(struct ifnet * ifp, u_long command, caddr_t data) { struct adapter *adapter = ifp->if_softc; - struct ifreq *ifr = (struct ifreq *) data; + struct ifreq *ifr = (struct ifreq *)data; #if defined(INET) || defined(INET6) - struct ifaddr *ifa = (struct ifaddr *)data; - bool avoid_reset = FALSE; + struct ifaddr *ifa = (struct ifaddr *)data; #endif - int error = 0; + bool avoid_reset = FALSE; + int error = 0; switch (command) { - - case SIOCSIFADDR: + case SIOCSIFADDR: #ifdef INET if (ifa->ifa_addr->sa_family == AF_INET) avoid_reset = TRUE; @@ -885,7 +884,6 @@ ixgbe_ioctl(struct ifnet * ifp, u_long command, ca if (ifa->ifa_addr->sa_family == AF_INET6) avoid_reset = TRUE; #endif -#if defined(INET) || defined(INET6) /* ** Calling init results in link renegotiation, ** so we avoid doing it when possible. @@ -894,12 +892,13 @@ ixgbe_ioctl(struct ifnet * ifp, u_long command, ca ifp->if_flags |= IFF_UP; if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) ixgbe_init(adapter); +#ifdef INET if (!(ifp->if_flags & IFF_NOARP)) arp_ifinit(ifp, ifa); +#endif } else error = ether_ioctl(ifp, command, data); break; -#endif case SIOCSIFMTU: IOCTL_DEBUGOUT("ioctl: SIOCSIFMTU (Set Interface MTU)"); if (ifr->ifr_mtu > IXGBE_MAX_FRAME_SIZE - ETHER_HDR_LEN) { -- wbr, pluknet From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 18:19:30 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA34B1065670 for ; Sat, 29 Oct 2011 18:19:30 +0000 (UTC) (envelope-from adrian.minta@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7C1848FC13 for ; Sat, 29 Oct 2011 18:19:30 +0000 (UTC) Received: by faar19 with SMTP id r19so6427240faa.13 for ; Sat, 29 Oct 2011 11:19:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=4bfxXPovbQBwuAgfGInHu6opPBn9Bq8GO03OiSVBBFE=; b=qTiWBOvWpiFuRMvjnJdOgZ+Bn6NJa6ftWksp1dqwVVfZaka/SqTuEWqsMMboHWF5g6 5RDEpAf412BoArsdxI0LS0GmIv7ch9cLxA3WlCTcMwiy92j5HObrq8T5QzpLi9hs1IQz d/RVManGauIK0ZYWs08SGbE4/m8Y5iSBbLNDw= Received: by 10.223.91.73 with SMTP id l9mr10957742fam.22.1319910484680; Sat, 29 Oct 2011 10:48:04 -0700 (PDT) Received: from [192.168.10.10] ([188.26.159.2]) by mx.google.com with ESMTPS id d21sm22706071fac.4.2011.10.29.10.48.01 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 29 Oct 2011 10:48:02 -0700 (PDT) Message-ID: <4EAC3C4F.40907@gmail.com> Date: Sat, 29 Oct 2011 20:47:59 +0300 From: Adrian Minta User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20111010 Icedove/3.1.15 MIME-Version: 1.0 To: freebsd-net@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: multiple ISP X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 18:19:31 -0000 On 10/28/11 17:22, nyoman.bogi@gmail.com wrote: > dear all, > > I need to set up a router (using FreeBSD) > that connect to the Internet > to accomodate multiple ISP, > so users can be load balanced through > those several ISP lines. > > how can I do that? > > thanks in advance > Take a look at: http://www.pfsense.org/ ... and ... http://doc.pfsense.org/index.php/Multi-WAN_2.0 -- Best regards, Adrian Minta MA3173-RIPE, www.minta.ro From owner-freebsd-net@FreeBSD.ORG Sat Oct 29 19:53:05 2011 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB782106564A; Sat, 29 Oct 2011 19:53:05 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B3FCB8FC08; Sat, 29 Oct 2011 19:53:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9TJr5KY097414; Sat, 29 Oct 2011 19:53:05 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9TJr5Nc097410; Sat, 29 Oct 2011 19:53:05 GMT (envelope-from linimon) Date: Sat, 29 Oct 2011 19:53:05 GMT Message-Id: <201110291953.p9TJr5Nc097410@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-amd64@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/162153: [em] intel em driver 7.2.4 don't compile X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Oct 2011 19:53:05 -0000 Old Synopsis: intel em driver 7.2.4 don't compile New Synopsis: [em] intel em driver 7.2.4 don't compile Responsible-Changed-From-To: freebsd-amd64->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Sat Oct 29 19:52:51 UTC 2011 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=162153